Efficient Bottleneck Detection in Stream Process System Using Fuzzy Logic Model

Big Data has shown lots of potential in numerous domain and becomes one of the emerging technologies that are bringing revolution in some real world industry. It has the power to provide insights into the unseen aspects of immense volume of data. Some applications are processing the data using a store-Then-process paradigm, whereas other applications, like telecommunications and large-scale sensor networks, have to analyze continuous data flow online. Stream Processing Engines(SPEs) are designed to support applications which require timely analysis of high volume data streams. The dynamic nature of data stream requires SPEs to have high scalability. However, current SPEs mostly adopt a static configuration and can not scale out/in flexibly along with the changing of the data stream. In this paper, we proposed a fuzzy logic based runtime bottleneck operator detection approach to improve the scalability of SPEs by providing resources in the cloud environment. Our experimental results show that the fuzzy logic component developed in this work could detect bottleneck operators efficiently. Compared with other bottleneck detection methods, the decision results generated by our approach is more flexible and will not scale out/in the system when the workload change instantly.