ISDI: A New Window-Based Framework for Integrating IoT Streaming Data from Multiple Sources

  • Doan Quang Tu
  • A. S. M. KayesEmail author
  • Wenny Rahayu
  • Kinh Nguyen
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 926)


Due to the rapid advancement in Internet of Things (IoT), myriad systems generate data of massive volume, variety and velocity which traditional databases are unable to manage effectively. Many organizations need to deal with these massive datasets that encounter different types of data (e.g., IoT streaming data, static data) in different formats coming from multiple sources. Different data integration mechanisms are designed to process mostly static data. Unfortunately, these techniques are not adequate to integrate IoT streaming data from multiple sources. In this paper, we identify the challenges of IoT streaming data integration (ISDI). A generic window-based ISDI approach is proposed to deal with IoT data in different formats and subsequently introduced the algorithms to integrate IoT streaming data obtained from multiple sources. In particular, we extend the basic windowing algorithm for real-time data integration and to deal with the timing alignment issue. We also introduce a de-duplication algorithm to deal with data redundancies and to demonstrate the useful fragments of the integrated data. We conduct several sets of experiments and quantify the performance of our proposed window-based ISDI approach. The experimental results, performed on several IoT datasets, show the efficiency of our proposed ISDI solution in terms of processing time.


IoT streaming data integration Timing alignment De-duplication Window-based integration 


  1. 1.
    Herland, M., Khoshgoftaar, T.M., Bauder, R.A.: Big data fraud detection using multiple medicare data sources. J. Big Data 5(1), 29 (2018)CrossRefGoogle Scholar
  2. 2.
    Chen, J., Chen, Y., Du, X., Li, C., Lu, J., Zhao, S., Zhou, X.: Big data challenge: a data management perspective. Front. Comput. Sci. 7(2), 157–164 (2013)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Bellahsène, Z., Bonifati, A., Rahm, E.: Schema Matching and Mapping. Springer, Berlin (2011)CrossRefGoogle Scholar
  4. 4.
    Sagi, T., Gal, A., Barkol, O., Bergman, R., Avram, A.: Multi-source uncertain entity resolution: transforming holocaust victim reports into people. Inf. Syst. 65, 124–136 (2017)CrossRefGoogle Scholar
  5. 5.
    Calbimonte, J.P., Corcho, O., Gray, A.J.: Enabling ontology-based access to streaming data sources. In: International Semantic Web Conference, pp. 96–111. Springer (2010)Google Scholar
  6. 6.
    Daraio, C., Lenzerini, M., Leporelli, C., Naggar, P., Bonaccorsi, A., Bartolucci, A.: The advantages of an ontology-based data management approach: openness, interoperability and data quality. Scientometrics 108(1), 441–455 (2016)CrossRefGoogle Scholar
  7. 7.
    Bifet, A., Gavalda, R.: Learning from time-changing data with adaptive windowing. In: Proceedings of the 2007 SIAM International Conference on Data Mining, SIAM, pp. 443–448 (2007)Google Scholar
  8. 8.
    Gama, J., Sebastião, R., Rodrigues, P.P.: On evaluating stream learning algorithms. Mach. Learn. 90(3), 317–346 (2013)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Pareek, A., Khaladkar, B., Sen, R., Onat, B., Nadimpalli, V., Lakshminarayanan, M.: Real-time ETL in Striim. In: Proceedings of the International Workshop on Real-Time Business Intelligence and Analytics, p. 3. ACM (2018)Google Scholar
  10. 10.
    Ahad, M.A., Biswas, R.: Dynamic merging based small file storage (DM-SFS) architecture for efficiently storing small size files in hadoop. Procedia Comput. Sci. 132, 1626–1635 (2018)CrossRefGoogle Scholar
  11. 11.
    Kayes, A., Han, J., Rahayu, W., Dillon, T., Islam, S., Colman, A.: A policy model and framework for context-aware access control to information resources. Comput. J. (2018)
  12. 12.
    Kayes, A., Rahayu, W., Dillon, T., Chang, E.: Accessing data from multiple sources through context-aware access control. In: TrustCom, pp. 551–559. IEEE (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Doan Quang Tu
    • 1
  • A. S. M. Kayes
    • 1
    Email author
  • Wenny Rahayu
    • 1
  • Kinh Nguyen
    • 1
  1. 1.La Trobe UniversityMelbourneAustralia

Personalised recommendations