Siddhi: a comprehensive architecture for stream processing

Abstract

Complex Event Processing (CEP) is one of the most rapidly emerging fields in data processing. Processing of high volume of events to derive higher level events is a vital part of several applications. The use-cases found in Business applications, financial trading applications, operational analytics applications and business activity monitoring applications are directly related to complex event processing. This paper discusses different design decisions associated with CEP Engines, and proposes to improve CEP performance by using a stream-processing-style pipelines. Furthermore, the paper discusses Siddhi, a CEP Engine that implements pipelines for stream processing and presents a performance study that exhibits that Siddhi CEP Engine has significantly improved performance. Primary contributions of this paper are, a critical analysis of the CEP and Event Stream Processing engine architecture and identifying areas for improvements, implementing those improvements through Siddhi, and demonstrating the soundness of those suggestions through empirical evidence.