Skip to main content
SpringerLink
Log in

Quality-of-Service in Data Center Stream Processing for Smart City Applications

  • Chapter
  • First Online:
Handbook on Data Centers

  • 4075 Accesses

Abstract

In this chapter we analyze the state-of-the-art of distributed stream processing systems, with a strong focus on the characteristics that make them more or less suitable to serve the novel processing needs of Smart City scenarios. In particular, we concentrate on the ability to offer differentiated Quality of Service (QoS). A growing number of Smart City applications, in fact, including those in the security, healthcare, or financial areas, require configurable and predictable behavior. For this reason, a key factor for the success of new and original stream processing supports will be their ability to efficiently meet those needs, while still being able to scale to fast growing workloads.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Note that, while in IBM InfoSphere Streams (Sect. 3.1) the concept of PE belongs to the execution model, in S4, it represents an abstract model concept.

References

  1. Abadi, D.J., Carney, D., Çetintemel, U., Cherniack, M., Convey, C., Lee, S., Stonebraker, M., Tatbul, N., Zdonik, S.: Aurora: a new model and architecture for data stream management. The VLDB Journal, 12, 2, pp. 120–139 (2003).

    Google Scholar 

  2. Abadi, D.J., Ahmad, Y., Balazinska, M., Çetintemel, U., Cherniack, M., Hwang, J.-H., Lindner, W., Maskey, A.S., Rasin, A., Ryvkina, E., Tatbul, N., Xing, Y., Zdonik, S.: The Design of the Borealis Stream Processing Engine.Proceedings of the 2nd Biennial Conference on Innovative Data Systems Research (CIDR). IEEE, Asilomar, CA (2005).

    Google Scholar 

  3. Agha, G. A.: Actors: a model of concurrent computation in distributed systems, Ph.D. dissertation, Artificial Intelligence Laboratory, Cambridge MA, USA (1985).

    Google Scholar 

  4. Amini, L., Andrade, H., Bhagwan, R., Frank Eskesen and Richard King and Yoonho Park and Chitra Venkatramani: SPC: A distributed, scalable platform for data mining. Proceedings of the Workshop on Data Mining Standards, Services and Platforms (DM-SS 2006). pp. 27–37. ACM, Philadelphia, PA (2006).

    Google Scholar 

  5. Apache S4 Project Web Site. Available, http://incubator.apache.org/s4. Last visited in September 2013.

  6. Arasu, A., Babcock, B., Babu, S., Cieslewicz, J., Ito, K., Motwani, R., Srivastava, U., and Widom, J.: STREAM: The Stanford Data Stream Management System, Technical report, Stanford InfoLab (2004).

    Google Scholar 

  7. Arasu, A., Babu, S., Widom, J.: The CQL continuous query language: semantic foundations and query execution. The VLDB Journal. 15, 2, pp. 121–142 (2005).

    Google Scholar 

  8. Avnur, R., Hellerstein, and J.M.: Eddies: continuosly adaptive query processing. Proceedings of the ACM SIGMOD international conference on Management of data (SIDMOD 2000). pp. 261–272. ACM, Dallas, TX, USA (2000).

    Google Scholar 

  9. Balazinska, M., Balakrishnan, H., Madden, S.R., Stonebraker, M.: Fault-tolerance in the borealis distributed stream processing system. ACM Trans. Database Syst. 33, 1, Article 3, 44 pages (2008).

    Google Scholar 

  10. Bellavista, P., Corradi, A., Reale, A.: The QUASIT Model and Framework for Scalable Data Stream Processing with Quality of Service. Proceedings of the 5th International Conference on Mobile Wireless Middleware, Operating Systems, and Applications (MOBILWARE 2012). Springer Berlin-Heidelberg, Berlin, Germany (2012).

    Google Scholar 

  11. Bellavista. P., Corradi, A., Reale, A.: Design and Implementation of a Scalable and QoS-aware Stream Processing Framework: the Quasit Prototype. Proceedings of the IEEE International Conference on Cyber, Physical and Social Computing (CPSCOM 2012). IEEE, Besançon, France (2012).

    Google Scholar 

  12. Bellavista. P., Corradi, A., Kotoulas, S., Reale, A.: Dynamic datacenter resource provisioning for high-performacne distributed stream processing with adaptive fault-tolerance. Proceedings of the 14th ACM/IFIP/USENIX International Middleware Conference—Demo & Poster Track, ACM, Beijing, China (2013).

    Google Scholar 

  13. Bellavista. P., Corradi, A., Kotoulas, S., Reale, A.: Adaptive fault-tolerance for dynamic resouce provisioning in distributed stream processing systems. Proceedings of the 17th International Conference on Extending Database Technology (EDBT 2014), ACM, Athens, Greece (2014). To appear.

    Google Scholar 

  14. Brito, A., Fetzerm C., Felber, P.: Multithreading-enabled active replication for event stream processing operators. In: 28th Symposium on Reliable Distributed Systems, pp. 22–31, IEEE, Niagara Falls, NY, USA (2009).

    Google Scholar 

  15. Cai, Z., Kumar, V., Cooper, B.F., Eisenhauer, G., Schwan, K., Strom, R.E.: Utility-driven proactive management of availability in enterprise-scale information flows. In: ACM/IFIP/USENIX 7h International Middleware Conference, Springer, Melbourne, Australia (2006).

    Google Scholar 

  16. Carney, D., Çetintemel, U., Cherniack, M., Convey, C., Lee, S., Seidman, G., Stonebraker, M., Tatbul, N., Zdonik, S.: Monitoring streams: a new class of data management applications. Proceedings of the 28th international conference on Very Large Data Bases (VLDB 2002). The VLDB Endowment, Hong Kong, PRC (2002).

    Google Scholar 

  17. Chandrasekaran, S., Shah, M.A., Cooper, O., Deshpande, A., Franklin, M.J., Hellerstein, J.M., Hong, W., Krishnamurthy, S., Madden, S.R., Reiss, F.: TelegraphCQ. Proceedings of the ACM SIGMOD international conference on on Management of data (SIGMOD 2003). pp. 668. ACM, San Diego, CA, USA (2003).

    Google Scholar 

  18. Condie, T., Conway, N., Alvaro, P., Hellerstein, J.M., Elmeleegy, K., Sears, R.: MapReduce Online. Proceedings of the 7th USENIX conference on Networked systems design and implementation (NSDI 2010). USENIX Association, San Jose, CA, USA (2010).

    Google Scholar 

  19. Cugola, G., Margara, A.: Processing flows of information: From Data Stream to Complex Event Processing. ACM Comput. Surv.. 44, 3, pp. 1–62 (2012).

    Google Scholar 

  20. Dean, J., Ghemawat, S.: MapReduce: Simplified Data Processing on Large Clusters. Commun. ACM, vol. 51, no. 1, pp. 107–113 (2008).

    Google Scholar 

  21. Digital Cities Project Web Site. Available, http://www.digital-cities.eu. Last visited in September 2013.

  22. Djahel, S., Salehie, M., Tal, I., Jamshidi, P.: Adaptive Traffic Management for Secure and Efficient Emergency Services in Smart Cities. Proceedings of the IEEE International Conference on Pervasive Computing and Communicatino (PerCom 2013)—WiP Session. pp. 340–343, IEEE, San Diego, CA, USA (2013).

    Google Scholar 

  23. EUROCITIES Web Site. Available, http://www.eurocities.eu/. Last visited in September 2013.

  24. Gedik, B., Andrade, H.: A model-based framework for building extensible, high performance stream processing middleware and programming language for IBM InfoSphere Streams. Softw. Pract. Exper. 42, 11, 1363–1391 (2012).

    Google Scholar 

  25. Gedik, B., Andrade, H., Wu, K.-L.: A code generation approach to optimizing high-performance distributed data stream processing. Proceeding of the 18th ACM conference on Information and knowledge management (CIKM 2009). p. 847, ACM, Hong Kong, PRC (2009).

    Google Scholar 

  26. Giffinger, R., Fertner, C., Kramar, H., Kalasek, R., Pichler-Milanovic, P., Meijers, M.: Smart cities – Ranking of European medium-sized cities. Vienna UT, Centre of Regional Science (2007) Available, http://www.smart-cities.eu/download/smart_cities_final_report.pdf. Last visited in September 2013.

  27. Haller, P. and Odersky, M.: Scala Actors: Unifying thread-based and event-based programming. Theoretical Computer Science, vol. 410, no. 2–3, pp. 202–220 (2009).

    Google Scholar 

  28. Hwang, J.-H., Balazinska, M., Rasin, A., Çetintemel, U., Stonebraker, M., Zdonik, S.: High-availability algorithms for distributed stream processing. In: 21st International Conference on Data Engineering, pp. 779–790, IEEE, Tokyo, Japan (2005).

    Google Scholar 

  29. IBM Smarter Cities Project Web Site. Available, http://www.ibm.com/smarterplanet/us/en/smarter_cities/. Last visited in September 2013.

  30. Intel Collaborative Research Institute for Sustainable Connected Cities. Available, http://www.intel-university-collaboration.net/?page_id=1420. Last visited in September 2013.

  31. Isard, M., Budiu, M., Yu, Y., Birrell, A., and Fetterly, D.: Dryad: distributed data-parallel programs from sequential building blocks. In: 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems, vol. 41, no. 3, p. 59–72, ACM New York, NY, USA (2007).

    Google Scholar 

  32. Jacques-Silva, G., Gedik, B., Andreade, H., Wu, K.-L.: Language level checkpointing support for stream processing applications. In: 2009 International Conference on Dependable Systems & Networks, pp. 145–154, IEEE, Estoril, Portugal (2009)

    Google Scholar 

  33. Khandekar, R., Hildrum, K., Parekh, S., Rajan, D., Wolf, J.: COLA: Optimizing Stream Processing Applications Via Graph Partitioning. Proceedings of the ACM/IFIP/USENIX 10th International Middeware Conference. pp. 308–327, Springer Berlin Heidelberg, Urbana Champagin, IL, USA (2009).

    Google Scholar 

  34. Martinez, F., Toh, C.-K., Cano, J., Calafate, C., Manzoni, P.: Emergency Services in Future Intelligent Transportation Systems Based on Vehicular Communication Networks. IEEE Intelligent Transportation Systems Magazine 2,2, pp. 6–20 (2010).

    Google Scholar 

  35. Neumeyer, L., Robbins, B., Nair, A., Kesari, A.: S4: Distributed Stream Computing Platform. In: 2010 IEEE International Conference on Data Mining Workshops (ICDMW '10), pp. 170–177, IEEE Los Alamitos, USA (2010).

    Google Scholar 

  36. OMG: Data Distribution Service for Real-time Systems – Version 1.2, Specification. Object Management Group (2007).

    Google Scholar 

  37. Pardo-Castellote, G.: OMG data-distribution service: Architectural overview. Proceedings of the 23rd International Conference on Distributed Computing Systems Workshops, pp. 200–206. IEEE, Providence, RI, USA (2003).

    Google Scholar 

  38. Wolf, J., Bansal, N., Hildrum, K., Parekh, S., Rajan, D.: SODA: An Optimizing Scheduler for Large-Scale Stream-Based Distributed Computer Systems. Proceedings of the ACM/IFIP/USENIX 9th International Middleware Conference. pp. 306–325. Springer Berlin Heidelberg, Leuven, Belgium (2008).

    Google Scholar 

  39. Xing, Y., Zdonik, S., Hwang, J.-H.: Dynamic Load Distribution in the Borealis Stream Processor. Proceedings of the 21st International Conference on Data Engineering (ICDE 2005). pp. 791–802. IEEE, Tokyo, Japan (2005).

    Google Scholar 

  40. Xing, Y., Hwang, J.-H., Zdonik, S.: Providing Resiliency to Load Variations in Distributed Stream Processing. Proceedings of the 32nd international conference on Very large data bases (VLDB 2006). pp. 775–786. The VLDB Endowment, Seoul, Korea (2006).

    Google Scholar 

  41. Safe City Project Web Site. Available, http://www.safecity-project.eu/. Last visited in September 2013.

  42. Smart Cities Stakeholder Platform. Available, http://www.eu-smartcities.eu/. Last visited in September 2013.

  43. Shah, M., Hellerstein, J., Brewer, E.: Highly available, fault-tolerant parallel dataflows. In: ACM International Conference on Management of Data, pp. 827–838, ACM, Paris, France (2004).

    Google Scholar 

  44. The Storm Project Web Site. Available, http://storm-project.net/. Last visited in September 2013.

  45. Tang, P., Venables, T.: “Smart” homes and telecare for independent living. J. Telemed. Telecare. 6, 1, pp. 8–14 (2000).

    Google Scholar 

  46. Yang, H.-c., Dasdan, A., Hsiao, R., Parker, D.: Map-reduce-merge: simplified relational data processing on large clusters. In: Proceedings of the ACM SIGMOD international conference on Management of data (SIGMOD 2007). pp. 1029–1040, Beijing, PRC (2007).

    Google Scholar 

  47. Zhang, Z., Gu, Y., Ye, F., Yang, H., Kim, M., Lei, H., Liu, Z.: A hybrid approach to high availability in stream processing systems. In: 30th IEEE International Conference on Distributed Computing Systems, pp. 138–148, Genoa, Italy (2010).

    Google Scholar 

Download references

Acknowledgements

We would like to thank the IBM Research Dublin Lab, and in particular Spyros Kotoulas, for his valuable work and feedback on LAAR (overviewed in Sect. 6), designed and implemented within a joint research collaboration.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering (DISI), Università di Bologna, Bologna, Italy

    Paolo Bellavista, Antonio Corradi & Andrea Reale

Authors
  1. Paolo Bellavista
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Antonio Corradi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andrea Reale
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Paolo Bellavista .

Editor information

Editors and Affiliations

  1. Department of Electrical and Computer Engineering, North Dakota State University, Fargo, North Dakota, USA

    Samee U. Khan

  2. School of Information Technologies, The University of Sydney, Sydney, New South Wales, Australia

    Albert Y. Zomaya

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer Science+Business Media New York

About this chapter

Cite this chapter

Bellavista, P., Corradi, A., Reale, A. (2015). Quality-of-Service in Data Center Stream Processing for Smart City Applications. In: Khan, S., Zomaya, A. (eds) Handbook on Data Centers. Springer, New York, NY. https://doi.org/10.1007/978-1-4939-2092-1_35

Download citation

  • DOI: https://doi.org/10.1007/978-1-4939-2092-1_35

  • Published:

  • Publisher Name: Springer, New York, NY

  • Print ISBN: 978-1-4939-2091-4

  • Online ISBN: 978-1-4939-2092-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation