Big Data and the Internet of Things

  • Mohak ShahEmail author
Part of the Studies in Big Data book series (SBD, volume 16)


Advances in sensing and computing capabilities are making it possible to embed increasing computing power in small devices. This has enabled the sensing devices not just to passively capture data at very high resolution but also to take sophisticated actions in response. Combined with advances in communication, this results in an ecosystem of highly interconnected devices referred to as the Internet of Things—IoT. In conjunction, the advances in machine learning have allowed building models on this ever increasing amount of data. Consequently, devices all the way from heavy assets such as aircraft engines to wearables such as health monitors can all now not only generate massive amounts of data but can draw back on aggregate analytics to “improve” their performance over time. Big data analytics has been identified as a key enabler for the IoT. In this chapter, we discuss various avenues of the IoT where big data analytics either is already making a significant impact or is on the cusp of doing so. We also discuss social implications and areas of concern.


Internet of things IoT IoTS Big data Industrial analytics Industrial internet 


  1. 1.
    Aggarwal, C.C., Abdelzaher, T.: Integrating sensors and social networks. In: Aggarwal, C.C. (ed.) Social Network Data Analytics, pp. 379–412. Springer, US (2011). doi: 10.1007/978-1-4419-8462-3_14; ISBN:978-1-4419-8461-6; Google Scholar
  2. 2.
    Aggarwal, C.C., Ashish, N., Sheth, A.: The internet of things: a survey from the data-centric perspective. In: Managing and Mining Sensor Data, pp. 383–428. Springer (2013)Google Scholar
  3. 3.
    Baaziz, A., Quoniam, L.: How to use big data technologies to optimize operations in upstream petroleum industry. Int. J. Innov. (IJI) 1(1), 30–42 (2013)Google Scholar
  4. 4.
    Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C.: A neural probabilistic language model. J. Mach. Learning Res. 3, 1137–1155 (2003)Google Scholar
  5. 5.
    Bettencourt, L.M.A.: The uses of big data in cities. Santa Fe Institute working paper 2013-09-029, September 2013.
  6. 6.
    Bosch MongoDB white-paper: IoT and big data. Technical report, October 2014.
  7. 7.
    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011)CrossRefzbMATHGoogle Scholar
  8. 8.
    Brasco, C., Eklund, N., Shah, M., Marthaler, D.: Predictive modeling of high-bypass turbofan engine deterioration. In: Proceedings of the Annual Conference of the Prognostics and Health Management Society (PHM 2013), vol. 4. PHM Society (2013).
  9. 9.
    Bui, N., Zorzi, M.: Health care applications: a solution based on the internet of things. In: Proceedings of the 4th International Symposium on Applied Sciences in Biomedical and Communication Technologies, ISABEL ’11, pp. 131:1–131:5. ACM, New York, NY, USA (2011).
  10. 10.
    Byrnes, N.: Cities find rewards in cheap technologies. MIT Technology Review, November 2014.
  11. 11.
    Chui, M., Löffler, M., Roberts, R.: The internet of things. McKinsey Quarterly 2, 1–9 (2010).
  12. 12.
    Cognizant Report: Reaping the benefits of the internet of things. Technical Report, May 2014.
  13. 13.
    Crankshaw, D., Bailis, P., Gonzalez, J.E., Li, H., Zhang, Z., Franklin, M.J., Ghodsi, A., Jordan, M.I.: The missing piece in complex analytics: low latency, scalable model management and serving with velox. In: Conference on Innovative Data Systems Research (CIDR). Asilomar, CA (2014)Google Scholar
  14. 14.
    Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P., Yang, K., Le, Q.V., Ng, A.Y.: Large scale distributed deep networks. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 25, pp. 1223–1231. Curran Associates, Inc. (2012).
  15. 15.
    Deb, B., Shah, M., Evans, S., Mehta, M., Gargulak, A., Lasky, T.: Towards systems level prognostics in the cloud. In: Proceedings of the IEEE Conference on Prognostics and Health Management (PHM), pp. 1–6. IEEE (2013). ISBN:978-1-4673-5722-7Google Scholar
  16. 16.
    Dohr, A., Modre-Opsrian, R., Drobics, M., Hayn, D., Schreier, G.: The internet of things for ambient assisted living. In: Seventh International Conference on Information Technology: New Generations (ITNG), 2010, pp. 804–809. IEEE (2010)Google Scholar
  17. 17.
    Doukas, C., Maglogiannis, I.: Bringing IoT and cloud computing towards pervasive healthcare. In: 2012 Sixth International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing (IMIS), pp. 922–926, July 2012. doi: 10.1109/IMIS.2012.26
  18. 18.
    Feblowitz, J.: The big deal about big data in upstream oil and gas. IDC Energy Insights, October 2012Google Scholar
  19. 19.
    Feigelson, E.D., Babu, G.J.: Big data in astronomy. Significance 9(4), 22–25 (2012)CrossRefGoogle Scholar
  20. 20.
    Fung, B.C.M., Wang, K., Chen, R., Yu, P.S.: Privacy-preserving data publishing: A survey of recent developments. ACM Comput. Surv. 42(4), 14:1–14:53, June 2010. doi: 10.1145/1749603.1749605; ISSN:0360-0300; Google Scholar
  21. 21.
    Garcia, A.B., Bentes, C., de Melo, R.C., Zadrozny, B., Penna, T.J.P.: Sensor data analysis for equipment monitoring. Knowled. Inform. Syst. 28(2), 333–364 (2011). doi: 10.1007/s10115-010-0365-1; ISSN:0219-1377; Google Scholar
  22. 22.
    Ghose, A., Bhaumik, C., Das, D., Agrawal, A.K.: Mobile healthcare infrastructure for home and small clinic. In: Proceedings of the 2nd ACM International Workshop on Pervasive Wireless Healthcare, MobileHealth ’12, pp. 15–20. ACM, New York, NY, USA (2012). doi: 10.1145/2248341.2248347; ISBN:978-1-4503-1292-9;
  23. 23.
    Glas, B., Guajardo, J., Hacioglu, H., Ihle, M., Wehefritz, K., Yavuz, A.: Signal-based automotive communication security and its interplay with safety requirements. In: Proceedings of Embedded Security in Cars Conference, November 2012Google Scholar
  24. 24.
    Gonzalez, J.E., Xin, R.S., Dave, A., Crankshaw, D., Franklin, M.J., Stoica, I.: Graphx: graph processing in a distributed dataflow framework. In: 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14), pp. 599–613. USENIX Association, Broomfield, CO, October 2014. ISBN:978-1-931971-16-4;
  25. 25.
    Gubbi, J., Buyya, R., Marusic, S., Palaniswami, M.: Internet of things (IoT): a vision, architectural elements, and future directions. Future Gen. Comput. Syst. 29, 1645–1660 (2013)Google Scholar
  26. 26.
    Hems, A., Soofi, A., Perez, E.: Drilling for new business value: how innovative oil and gas companies are using big data to outmaneuver the competition. A Microsoft White Pater, May 2013Google Scholar
  27. 27.
    Hesla, L.: Particle physics tames big data. Symmetry 1 (2012)Google Scholar
  28. 28.
    IBM White Paper: Predictive maintenance for manufacturing. IBM (2011)Google Scholar
  29. 29.
    Japkowicz, N., Shah, M.: Evaluating Learning Algorithms: A classification perspective. Cambridge University Press (2011)Google Scholar
  30. 30.
    Jeske, M., Grüner, M., Weiß, F.: Big data in logistics: A DHL perspective on how to move beyond the hype. DHL Customer Solutions and Innovation, December 2013.
  31. 31.
    Joint DHL Bosch KIT Report: Self-driving vehicles in logistics: A DHL perspective on implications and use cases for the logistics industry. Technical report (2014).
  32. 32.
    Kleiner, A., Talwalkar, A., Sarkar, P., Jordan, M.I.: A scalable bootstrap for massive data. J. Royal Statis. Soc. 76, 795–816 (2013)MathSciNetCrossRefGoogle Scholar
  33. 33.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 25, pp. 1097–1105. Curran Associates, Inc. (2012).
  34. 34.
    Kurtz, J., Hoy, P., McHargue, L., Ward, J.: Improving operational and financial results through predictive maintenance. IBM Smarter Analytics Leadership Summit, Feb 2013Google Scholar
  35. 35.
    Lawson, S.: IoT groups are like an orchestra tuning up: the music starts in 2016. Computer World, Dec 2014.
  36. 36.
    Le, Q.V., Monga, R., Devin, M., Chen, K., Corrado, G.S., Dean, J., Ng, A.Y.: Building high-level features using large scale unsupervised learning. In: International Conference on Machine Learning (2012)Google Scholar
  37. 37.
    Lee, J., Lapira, E., Bagheri, B., Kao, H.: Recent advances and trends in predictive manufacturing systems in big data environment. Manuf. Lett. 1, 38–41 (2013)Google Scholar
  38. 38.
    Lee, J., Kao, H., Yang, S.: Service innovation and smart analytics for industry 4.0 and big data environment. Procedia CIRP 16, 3–8 (2014)Google Scholar
  39. 39.
    Leuth, K.L.: IoT market segments biggest opportunities in industrial manufacturing. IoT-Analytics (2014).
  40. 40.
    Lin, J., Keogh, E., Lonardi, S., Chiu, B.: A symbolic representation of time series, with implications for streaming algorithms. In: Proceedings of the 8th ACM SIGMOD workshop on Research issues in data mining and knowledge discovery, pp. 2–11. ACM (2003)Google Scholar
  41. 41.
    Mackey, L., Talwalkar, A., Jordan, M.I.: Distributed matrix completion and robust factorization. J. Mach. Learn. Res. (2014)Google Scholar
  42. 42.
    Markkanen, A., Shey, D.: The intersection of analytics and the internet of things. IEEE Internet of Things Newsletter, Nov 2014.
  43. 43.
    Marz, N., Warren, J.: Big data: principles and best practices of scalable realtime data systems. Manning Publications Co. (2015)Google Scholar
  44. 44.
    Matwin, S.: Privacy-preserving data mining techniques: survey and challenges. In: Discrimination and Privacy in the Information Society, pp. 209–221. Springer (2013)Google Scholar
  45. 45.
    McKinsey Study: Connected car, automotive value chain unbound. Technical report (2014)Google Scholar
  46. 46.
    Metz, R.: Ces 2015: Wearables everywhere. MIT Technology Review, January 2015.
  47. 47.
    Middleton, P., Kjeldsen, P., Tully, J.: Forecast: The Internet of Things, worldwide, 2013. Gartner, November 2013Google Scholar
  48. 48.
    Mind Commerce LLC Report: Big data in extraction and natural resource industries: Mining, water, timber, oil and gas 2014–2019. Technical report, July 2014.
  49. 49.
    MIT Business Report: Cities get smarter. Technical report (2015)Google Scholar
  50. 50.
    Nambiar, R., Bhardwaj, R., Sethi, A., Vargheese, R.: A look at challenges and opportunities of big data analytics in healthcare. In: 2013 IEEE International Conference on Big Data, pp. 17–22. IEEE (2013)Google Scholar
  51. 51.
    Navarro-Arribas, G., Torra, V.: Advanced Research in Data Privacy (2014)Google Scholar
  52. 52.
    Nicholson, R.: Big data in the oil and gas industry. IDC Energy Insights, September 2012Google Scholar
  53. 53.
    NIST Report: Workshop report on foundations for innovation in cyber-physical systems. Technical report, Jan 2013.
  54. 54.
    Orts, E., Spigonardo, J.: Sustainability in the age of big data. Special Report, Initiative for Global Environmental Leadership (IGEL), Knowledge at Wharton, September 2014.
  55. 55.
    Páez, D., Aparicio, F., de Buenaga, M., Ascanio, J.R.: Big data and IoT for chronic patients monitoring. In: Ubiquitous Computing and Ambient Intelligence. Personalisation and User Adapted Services, pp. 416–423. Springer (2014)Google Scholar
  56. 56.
    Pan, X., Jegelka, S., Gonzalez, J., Bradley, J.K., Jordan, M.: Parallel double greedy submodular maximization. In: Advances in Neural Information Processing Systems 22, (2014)Google Scholar
  57. 57.
    Poulymenopoulou, M., Malamateniou, F., Vassilacopoulos, G.: Machine learning for knowledge extraction from phr big data. Stud. Health Technol. Inform. 202, 36–39 (2013)Google Scholar
  58. 58.
    Reddy, A.S.: Reaping the benefits of the internet of things. Cognizant Reports, May 2014Google Scholar
  59. 59.
    Salakhutdinov, R.: Learning deep generative models. Ph.D. thesis, University of Toronto, Toronto, Canada (2009)Google Scholar
  60. 60.
    Seshadri, M.: Big data science challenging the oil industry. Energyworld (2013).
  61. 61.
    Socher, R., Pennington, J., Huang, E.H., Ng, A.Y., Manning, C.D.: Semi-supervised recursive autoencoders for predicting sentiment distributions. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’11, pp. 151–161. Association for Computational Linguistics, Stroudsburg, PA, USA, 2011. ISBN:978-1-937284-11-4.
  62. 62.
    Sowe, S.K., Kimata, T., Mianxiong, D., Zettsu, K.: Managing heterogeneous sensor data on a big data platform: IoT services for data-intensive science. In: 2014 IEEE 38th International Computer Software and Applications Conference Workshops (COMPSACW), pp. 295–300, July 2014. doi: 10.1109/COMPSACW.2014.52
  63. 63.
    Tracey, D., Sreenan, C.: A holistic architecture for the internet of things, sensing services and big data. In: 2013 13th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), pp. 546–553, May 2013. doi: 10.1109/CCGrid.2013.100
  64. 64.
    Turner, V., Gantz, J.F., Reinsel, D., Minton, S.: The digital universe of opportunities: rich data and the increasing value of the internet of things. IDC White Paper, April 2014.
  65. 65.
    Vandermerwe, S., Rada, J.: Servitization of business: adding value by adding services. Eur. Manage J. 6(6), 314–324 (1989)Google Scholar
  66. 66.
    Vermesan, O., Friess, P.: Internet of Things: Converging Technologies for Smart Environments and Integrated Ecosystems. River Publishers (2013)Google Scholar
  67. 67.
    Waller, M.A., Fawcett, S.E.: Data science, predictive analytics, and big data: a revolution that will transform supply chain design and management. J. Bus. Logist. 34(2), 77–84 (2013)CrossRefGoogle Scholar
  68. 68.
    Wang, Y., Bai, H., Stanton, M., Chen, W., Chang, E.Y.: Plda: parallel latent dirichlet allocation for large-scale applications. In: Proceedings of the 5th International Conference on Algorithmic Aspects in Information and Management, AAIM ’09, pp. 301–314. Springer-Verlag, Berlin, Heidelberg (2009). doi: 10.1007/978-3-642-02158-9_26; ISBN:978-3-642-02157-2; Google Scholar
  69. 69.
    Witten, B.: Top 10 IoT security mishaps 2014. In: Industrial Internet Consortium Web blog post. IIC (2014).
  70. 70.
    Yashiro, T., Kobayashi, S., Koshizuka, N., Sakamura, K.: An internet of things (IoT) architecture for embedded appliances. In: Humanitarian Technology Conference (R10-HTC), 2013 IEEE. Region, vol. 10, pp. 314–319 (2013). doi: 10.1109/R10-HTC.2013.6669062
  71. 71.
    Yavuz, A.A.: Practical immutable signature bouquets (pisb) for authentication and integrity in outsourced databases. In: Data and Applications Security and Privacy XXVI, pp. 179–194. Springer (2013)Google Scholar
  72. 72.
    Zaki, M., Neely, A.: Optimising asset management within complex service networks: the role of data. Cambridge Service Alliance, working paper:1–11 (2014)Google Scholar
  73. 73.
    Zanella, A., Bui, N., Castellani, A., Vangelista, L., Zorzi, M.: Internet of things for smart cities. IoT J IEEE, 1(1):22–32 (2014). doi: 10.1109/JIoT.2014.2306328; ISSN:2327-4662Google Scholar
  74. 74.
    Zaslavsky, A, Perera, C., Georgakopoulos, D.: Sensing as a service and big data. arXiv:1301.0159 (2013)
  75. 75.
    Zhai, K., Boyd-Graber, J., Asadi, N., Alkhouja, M.L.: Mr. lda: A flexible large scale topic modeling package using variational inference in mapreduce. In: Proceedings of the 21st International Conference on World Wide Web, WWW ’12, pp. 879–888, ACM, New York, NY, USA (2012). doi: 10.1145/2187836.2187955; ISBN:978-1-4503-1229-5;
  76. 76.
    Zhou, Z., Chawla, N., Jin, Y., Williams, G.: Big data opportunities and challenges: discussions from data analytics perspectives [discussion forum]. IEEE Comput. Intell. Magaz. 9(4), 62–74 (2014)CrossRefGoogle Scholar
  77. 77.
    Zicari, R.V., Akerkar, R. (ed.): Big data computing. In: Big Data: Challenges and Opportunities, pp. 103–128. Chapman and Hall/CRC (2013)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.Research and Technology Center - North AmericaPalo AltoUSA

Personalised recommendations