Towards a Big Data Analytics Framework for IoT and Smart City Applications

  • Martin StrohbachEmail author
  • Holger Ziekow
  • Vangelis Gazis
  • Navot Akiva
Part of the Modeling and Optimization in Science and Technologies book series (MOST, volume 4)


An increasing amount of valuable data sources, advances in Internet of Things and Big Data technologies as well as the availability of a wide range of machine learning algorithms offers new potential to deliver analytical services to citizens and urban decision makers. However, there is still a gap in combining the current state of the art in an integrated framework that would help reducing development costs and enable new kind of services. In this chapter, we show how such an integrated Big Data analytical framework for Internet of Things and Smart City application could look like. The contributions of this chapter are threefold: (1) we provide an overview of Big Data and Internet of Things technologies including a summary of their relationships, (2) we present a case study in the smart grid domain that illustrates the high-level requirements towards such an analytical Big Data framework, and (3) we present an initial version of such a framework mainly addressing the volume and velocity challenge. The findings presented in this chapter are extended results from the EU funded project BIG and the German funded project PEC.


Smart City Hadoop Distribute File System MapReduce Framework Complex Event Processing Batch Layer 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Abadi, D.J., et al.: The Design of the Borealis Stream Processing Engine. In: CIDR, vol. 5, pp. 277–289 (2005)Google Scholar
  2. 2.
    Atzori, L., Iera, A., Morabito, G.: The internet of things: A survey. Computer Networks 54(15), 2787–2805 (2010)CrossRefzbMATHGoogle Scholar
  3. 3.
    Barga, R.S., et al.: Consistent streaming through time: A vision for event stream processing. arXiv preprint cs/0612115 (2006)Google Scholar
  4. 4.
    Batty, M.: Smart Cities and Big Data,
  5. 5.
    Bauer, M., Bui, N., Giacomin, P., Gruschka, N., Haller, S., Ho, E., Kernchen, R., Lischka, M., Loof, J.D., Magerkurth, C., Meissner, S., Meyer, S., Nettsträter, A., Lacalle, F.O., Segura, A.S., Serbanati, A., Strohbach, M., Toubiana, V., Walewski, J.W.: IoT-A Project Deliverable D1.2 – Initial Architectural Reference Model for IoT (2011), (last accessed September 18, 2013)
  6. 6.
    Bifet, A., Frank, E.: Sentiment knowledge discovery in twitter streaming data. In: Pfahringer, B., Holmes, G., Hoffmann, A. (eds.) DS 2010. LNCS (LNAI), vol. 6332, pp. 1–15. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  7. 7.
    BIG Project Website, (last accessed September 19, 2013)
  8. 8.
    Chandrasekaran, S., et al.: TelegraphCQ: continuous dataflow processing. In: ACM SIGMOD International Conference on Management of Data, pp. 668–668. ACM (2003)Google Scholar
  9. 9.
    Chu, C.-T., Kim, S.K., Lin, Y.A., Yu, Y.Y., Bradski, G., Ng, A.Y., Olukotun, K.: Map-Reduce for Machine Learning on Multicore. In: Schölkopf, B., Platt, J.C., Hoffman, T. (eds.) Advances in Neural Information Processing Systems 19 (NIPS 2006), pp. 281–288. MIT Press, Cambridge (2007)Google Scholar
  10. 10.
    Correia, Z.P.: Toward a Stakeholder Model for the Co-Production of the Public Sector Information System. Information Research 10(3), paper 228 (2005), (last accessed February 27, 2013)
  11. 11.
    DataMarket, (last accessed September 21, 2013)
  12. 12., (last accessed September 21, 2013)
  13. 13.
    Davis, J.R., Clodoveu, A., et al.: Inferring the Location of Twitter Messages based on User Relationships. Transactions in GIS 15(6), 735–751 (2011)CrossRefGoogle Scholar
  14. 14.
    Dean, J., Ghemawat, S.: MapReduce: Simplified Data Processing on Large Clusters. Communications of the ACM 51(1), 1–13 (2008), doi:10.1145/1327452.1327492CrossRefGoogle Scholar
  15. 15.
    Directive 2003/98/EC of the European Parliament and of the Council of 17 November 2003 on the re-use of public sector information, (last accessed January 13, 2014)
  16. 16.
    Dohler, M.: Machine-to-Machine Technologies, Applications & Markets. In: 27th IEEE International Conference on Advanced Information Networking and Applications (AINA) (2013)Google Scholar
  17. 17.
    Dredze, M., Paul, M.J., Bergsma, S., Tran, H.: Carmen: A Twitter Geolocation System with Applications to Public Health (2013)Google Scholar
  18. 18.
    The Economist, Running out of road (November 2006)Google Scholar
  19. 19.
    EsperTech, (last accessed September 22, 2013)
  20. 20.
    Etzion, O.: On Off-Line Event Processing. Event Processing Thinking Online Blog (2009), (last accessed September 17, 2013)
  21. 21.
    European Open Data Portal, (last accessed September 21, 2013)
  22. 22.
    Farroukh, A., Sadoghi, M., Jacobsen, H.-A.: Towards vulnerability-based intrusion detection with event processing. In: 5th ACM International Conference on Distributed Event-based System, pp. 171–182. ACM (2011)Google Scholar
  23. 23.
    Gazis, V., Strohbach, M., Akiva, N., Walther, M.: A Unified View on Data Path Aspects for Sensing Applications at a Smart City Scale. In: IEEE 27th International Conference onAdvanced Information Networking and Applications Workshops (WAINA 2013), pp. 1283–1288. IEEE Computer Society, Barcelona (2013), doi:10.1109/WAINA.2013.66CrossRefGoogle Scholar
  24. 24.
    Gedik, B., Andrade, H., Wu, K.L., Yu, P.S., Doo, M.: SPADE: the system s declarative stream processing engine. In: ACM SIGMOD International Conference on Management of Data, pp. 1123–1134. ACM (2008)Google Scholar
  25. 25.
    Giraph Project, (last accessed September 21, 2013)
  26. 26.
    Gubbi, J., Buyya, R., Marusic, S., Palaniswami, M.: Internet of things (IoT): A vision, architectural elements, and future directions. Future Generation Computer Systems (2013)Google Scholar
  27. 27.
    Hinze, A., Sachs, K., Buchmann, A.: Event-based applications and enabling technologies. In: Third ACM International Conference on Distributed Event-Based Systems (2009)Google Scholar
  28. 28.
    INFSO D.4 Networked Enterprise & RFID INFSO G.2 Micro & Nanosystems, Internet of Things in 2020 – A roadmap for the Future (September 2008), report available at
  29. 29.
    ITU, The Internet of Things (2005)Google Scholar
  30. 30.
    Fidler, E., Jacobsen, H.A., Li, G., Mankovski, S.: The PADRES Distributed Publish/Subscribe System. In: FIW, pp. 12–30 (2005)Google Scholar
  31. 31.
    van Kasteren, T., Ravkin, H., Strohbach, M., Lischka, M., Tinte, M., Pariente, T., Becker, T., Ngonga, A., Lyko, K., Hellmann, S., Morsey, M., Frischmuth, P., Ermilov, I., Martin, M., Zaveri, A., Capadisli, S., Curry, E., Freitas, A., Rakhmawati, N.A., Ul Hassan, U., Iqbal, A.: BIG Project Deliverable D2.2.1 – First Draft of Technical White Papers (2013), (last accessed September 19, 2013)
  32. 32.
    Laney, D. 3D Data Management: Controlling Data Volume, Velocity and Variety. Meta Group Research Report (2001), (last accessed September 21, 2013)
  33. 33.
    Leeds, D.J.: THE SOFT GRID 2013-2020:Big Data & Utility Analytics for Smart Grid. GTM Research Report (2012), (last accessed September 21, 2013)
  34. 34.
    Liu, B.: Sentiment analysis and opinion mining. Synthesis Lectures on Human Language Technologies 5(1), 1–167 (2012)CrossRefGoogle Scholar
  35. 35.
    The London Dahsboard, (last accessed September 21, 2013)
  36. 36.
    Madden, S.R., Franklin, M.J., Hellerstein, J.M., Hong, W.: TinyDB: An acquisitional query processing system for sensor networks. ACM Transactions on Database Systems 30, 122–173 (2005)CrossRefGoogle Scholar
  37. 37.
    Manyika, J., Chui, M., Brown, B., Bughin, J., Dobbs, R., Roxburgh, C., Hung Byers, A.: Big data: The next frontier for innovation, competition, and productivity. McKinsey Global Institute (2013), (last accessed August 20, 2013)
  38. 38.
    Marz, N., Warren, J.: A new paradigm for Big Data. In: Big Data – Principles and Best Practices of Scalable Real-time Data Systems, ch. 1, Manning Publications Co. (to appear), (last accessed August 16, 2013), ISBN 9781617290343
  39. 39.
    Miorandi, S., Sicari, F., Pellegrini, D., Chlamtac, I.: Internet of things: Vision, applications and research challenges. Ad Hoc Networks 10(7), 1497–1516 (2012)CrossRefGoogle Scholar
  40. 40.
    Microsoft BI Team, Big Data, Hadoop and StreamInsightTM, (last accessed September 09, 2013)
  41. 41.
    Rajeev, M., Widom, J., Arasu, A., Babcock, B., Babu, S., Datar, M., Manku, G., Olston, C., Rosenstein, J., Varma, R.: Query processing, approximation, and resource management in a data stream management system. In: CIDR Conference, pp. 1–16 (2002)Google Scholar
  42. 42.
    Owen, S., Anil, R., Dunning, T., Friedman, E.: Mahout in Action. Manning Publications Co. (2011) ISBN 9781935182689Google Scholar
  43. 43.
    Neubauer, P.: Neo4j and some graph problems, (last accessed September 21, 2013)
  44. 44.
    Palensky, P., Dietrich, D.: Demand side management: Demand response, intelligent energy systems, and smart loads. IEEE Transactions on Industrial Informatics 7(3), 381–388 (2011)CrossRefGoogle Scholar
  45. 45.
    Pang, B., Lee, L.: Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval 2(1-2), 1–135 (2008)CrossRefGoogle Scholar
  46. 46.
    Peer Energy Cloud project website, (last accessed September 19, 2013)
  47. 47.
    PredPol, (September 21, 2013)
  48. 48.
    Radoop, (last accessed December 16, 2013)
  49. 49.
    Redis Project, (last accessed September 17, 2013)
  50. 50.
    ruleCore, (last accessed September 22, 2013)
  51. 51.
    S4 Project, (last accessed August 16, 2013)
  52. 52.
    Salton, G., Buckley, C.: Term-Weighting Approaches in Automatic Text Retrieval. Information Processing and Management 24(5), 513–523 (1988)CrossRefGoogle Scholar
  53. 53.
    Seo, S., Yoon, E.J., Kim, J., Jin, S., Kim, J.-S., Maeng, S.: HAMA: An Efficient Matrix Computation with the MapReduce Framework. In: 2nd IEEE International Conference on Cloud Computing Technology and Science (CloudCom 2010), pp. 721–726. IEEE Computer Society (2013)Google Scholar
  54. 54.
    Shigeru, O.: M2M and Big Data to Realize the Smart City. NEC Technical Journal 7(2) (2012)Google Scholar
  55. 55.
    Siddhi CEP - The Complex Event Processing Engine, (last accessed September 22, 2013)
  56. 56.
    Storm Project, (last accessed August 16, 2013)
  57. 57.
    Stonebraker, M.: What Does ‘Big Data’ Mean? (Part 3). BLOG@ACM (2012), (last accessed August 16, 2013)
  58. 58.
    United Nations, World Urbanization Prospects 2011 Revision (2011)Google Scholar
  59. 59.
    US National Intelligence Council.: Disruptive Civil Technologies: Six Technologies with Potential Impacts on US Interests out to 2025, (last accessed December 20, 2013)
  60. 60.
    Wang, F.-s., Liu, S., Liu, P., Bai, Y.: Bridging physical and virtual worlds: complex event processing for RFID data streams. In: Ioannidis, Y., et al. (eds.) EDBT 2006. LNCS, vol. 3896, pp. 588–607. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  61. 61.
    Weidlich, M., Ziekow, H., Mendling, J., Günther, O., Weske, M., Desai, N.: Event-based monitoring of process execution violations. In: Rinderle-Ma, S., Toumani, F., Wolf, K. (eds.) BPM 2011. LNCS, vol. 6896, pp. 182–198. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  62. 62.
    Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The WEKA Data Mining Software: An Update. SIGKDD Explorations 11, 10–18 (2009)CrossRefGoogle Scholar
  63. 63.
    White, T.: Hadoop: The Definitive Guide. O’Reilly (2012)Google Scholar
  64. 64.
    Xively, (last accessed September 21, 2013)
  65. 65.
    Ziekow, H., Doblander, C., Goebel, C., Jacobsen, H.-A.: Forecasting Household Electricity Demand with Complex Event Processing: Insights from a Prototypical Solution. In: Middleware Conference, Beijing, China (2013)Google Scholar
  66. 66.
    Ziekow, H., Goebel, C., Strüker, J., Jacobsen, H.-A.: The Potential of Smart Home Sensors in Forecasting Household Electricity Demand. In: IEEE International Conference on Smart Grid Communications (SmartGridComm 2013), Vancouver, Canada (2013)Google Scholar
  67. 67.
    Zillner, S., Rusitschka, S., Munné, R., Lippell, H., Lobillo Vilela, F., Hussain, K., Becker, T., Jung, R., Paradowski, D., Huang, Y.: BIG Project Deliverable D2.3.1 – First Draft of Sector’s Requisites (2013), (last accessed September 19, 2013)
  68. 68.
    European Smart Cities project,
  69. 69.
    Smart City Week 2012 international conference and exhibition report, October 29-November 2 (2012),

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Martin Strohbach
    • 1
    Email author
  • Holger Ziekow
    • 1
  • Vangelis Gazis
    • 1
  • Navot Akiva
    • 1
  1. 1.AGT InternationalDarmstadtGermany

Personalised recommendations