Event Processing over a Distributed JSON Store: Design and Performance

  • Miki Enoki
  • Jérôme Siméon
  • Hiroshi Horii
  • Martin Hirzel
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8787)


Web applications are increasingly built to target both desktop and mobile users. As a result, modern Web development infrastructure must be able to process large numbers of events (e.g., for location-based features) and support analytics over those events, with applications ranging from banking (e.g., fraud detection) to retail (e.g., just-in-time personalized promotions). We describe a system specifically designed for those applications, allowing high-throughput event processing along with analytics. Our main contribution is the design and implementation of an in-memory JSON store that can handle both events and analytics workloads. The store relies on the JSON model in order to serve data through a common Web API. Thanks to the flexibility of the JSON model, the store can integrate data from systems of record (e.g., customer profiles) with data transmitted between the server and a large number of clients (e.g., location-based events or transactions). The proposed store is built over a distributed, transactional, in-memory object cache for performance. Our experiments show that our implementation handles high throughput and low latency without sacrificing scalability.


Events Processing Analytics Rules JSON In-Memory Database 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Abadi, D.J., Ahmad, Y., et al.: The design of the Borealis stream processing engine. In: Conference on Innovative Data Systems Research (CIDR), pp. 277–289 (2005)Google Scholar
  2. 2.
    Alvaro, P., Condie, T., Conway, N., Elmeleegy, K., Hellerstein, J.M., Sears, R.: BOOM analytics: Exploring data-centric, declarative programming for the cloud. In: European Conference on Computer Systems (EuroSys), pp. 223–236 (2010)Google Scholar
  3. 3.
    Anderson, J.C., Lehnardt, J., Slater, N.: CouchDB: The definitive guide. O’Reilly (2010)Google Scholar
  4. 4.
    Bonner, A.J.: Workflow, transactions and Datalog. In: Symposium on Principles of Database Systems (PODS), pp. 294–305 (1999)Google Scholar
  5. 5.
    Brenna, L., Gehrke, J., Johansen, D., Hong, M.: Distributed event stream processing with non-deterministic finite automata. In: Conference on Distributed Event-Based Systems (DEBS) (2009)Google Scholar
  6. 6.
    Ceri, S., Widom, J.: Production rules in parallel and distributed database environments. In: Conference on Very Large Data Bases (VLDB), pp. 339–351 (1992)Google Scholar
  7. 7.
    Chang, F., Dean, J., Ghemawat, S., Hsieh, W.C., Wallach, D.A., Burrows, M., Chandra, T., Fikes, A., Gruber, R.E.: Bigtable: a distributed storage system for structured data. In: Operating Systems Design and Implementation (OSDI), pp. 205–218 (2006)Google Scholar
  8. 8.
    Cooper, B.F., Silberstein, A., et al.: Benchmarking cloud serving systems with YCSB. In: Symposium on Cloud Computing (SoCC), pp. 143–154 (2010)Google Scholar
  9. 9.
    Florescu, D., Fourny, G.: JSONiq: The history of a query language. IEEE Internet Computing 17(5), 86–90 (2013)CrossRefGoogle Scholar
  10. 10.
    Forgy, C.L.: OPS5 user’s manual. Technical Report 2397, Carnegie Mellon University (CMU) (1981)Google Scholar
  11. 11.
    Hirzel, M.: Partition and compose: Parallel complex event processing. In: Conference on Distributed Event-Based Systems (DEBS), pp. 191–200 (2012)Google Scholar
  12. 12.
    Hirzel, M., Andrade, H., et al.: IBM Streams Processing Language: Analyzing big data in motion. IBM Journal of Research and Development (IBMRD) 57(3/4), 7:1–7:11 (2013)Google Scholar
  13. 13.
    Kantere, V., Kiringa, I., Zhou, Q., Mylopoulos, J., McArthur, G.: Distributed triggers for peer data management. In: Meersman, R., Tari, Z. (eds.) OTM 2006. LNCS, vol. 4275, pp. 17–35. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  14. 14.
    MongoDB NoSQL database, (retrieved December 2013)
  15. 15.
    Node.js v0.10.24 manual & documentation (2013),
  16. 16.
    Pascalau, E., Giurca, A.: JSON rules: The JavaScript rule engine. In: Knowledge Engineering and Software Engineering (KESE) (2008)Google Scholar
  17. 17.
    Peng, D., Dabek, F.: Large-scale incremental processing using distributed transactions and notifications. In: Operating Systems Design and Implementation (OSDI), pp. 251–264 (2010)Google Scholar
  18. 18.
    Rivera, J., van der Meulen, R.: Gartner says in-memory computing is racing towards mainstream adoption. Press Release (April 2013),
  19. 19.
    Stolfo, S.J., Prodromidis, A.L., Tselepis, S., Lee, W., Fan, D.W., Chan, P.K.: JAM: Java agents for meta-learning over distributed databases. In: Conference on Knowledge Discovery and Data Mining (KDD), pp. 74–81 (1997)Google Scholar
  20. 20.
    Streambase, (retrieved December 2013)
  21. 21.
    WODM: IBM Operational Decision Manager, (retrieved December 2013)
  22. 22.
    WXS: IBM WebSphere eXtreme Scale (2013), (retrieved November)

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Miki Enoki
    • 1
  • Jérôme Siméon
    • 2
  • Hiroshi Horii
    • 1
  • Martin Hirzel
    • 2
  1. 1.IBM ResearchTokyoJapan
  2. 2.IBM ResearchNew YorkUSA

Personalised recommendations