ISDI: A New Window-Based Framework for Integrating IoT Streaming Data from Multiple Sources
Due to the rapid advancement in Internet of Things (IoT), myriad systems generate data of massive volume, variety and velocity which traditional databases are unable to manage effectively. Many organizations need to deal with these massive datasets that encounter different types of data (e.g., IoT streaming data, static data) in different formats coming from multiple sources. Different data integration mechanisms are designed to process mostly static data. Unfortunately, these techniques are not adequate to integrate IoT streaming data from multiple sources. In this paper, we identify the challenges of IoT streaming data integration (ISDI). A generic window-based ISDI approach is proposed to deal with IoT data in different formats and subsequently introduced the algorithms to integrate IoT streaming data obtained from multiple sources. In particular, we extend the basic windowing algorithm for real-time data integration and to deal with the timing alignment issue. We also introduce a de-duplication algorithm to deal with data redundancies and to demonstrate the useful fragments of the integrated data. We conduct several sets of experiments and quantify the performance of our proposed window-based ISDI approach. The experimental results, performed on several IoT datasets, show the efficiency of our proposed ISDI solution in terms of processing time.
KeywordsIoT streaming data integration Timing alignment De-duplication Window-based integration
- 5.Calbimonte, J.P., Corcho, O., Gray, A.J.: Enabling ontology-based access to streaming data sources. In: International Semantic Web Conference, pp. 96–111. Springer (2010)Google Scholar
- 7.Bifet, A., Gavalda, R.: Learning from time-changing data with adaptive windowing. In: Proceedings of the 2007 SIAM International Conference on Data Mining, SIAM, pp. 443–448 (2007)Google Scholar
- 9.Pareek, A., Khaladkar, B., Sen, R., Onat, B., Nadimpalli, V., Lakshminarayanan, M.: Real-time ETL in Striim. In: Proceedings of the International Workshop on Real-Time Business Intelligence and Analytics, p. 3. ACM (2018)Google Scholar
- 11.Kayes, A., Han, J., Rahayu, W., Dillon, T., Islam, S., Colman, A.: A policy model and framework for context-aware access control to information resources. Comput. J. (2018) https://doi.org/10.1093/comjnl/bxy065
- 12.Kayes, A., Rahayu, W., Dillon, T., Chang, E.: Accessing data from multiple sources through context-aware access control. In: TrustCom, pp. 551–559. IEEE (2018)Google Scholar