Abstract
In Data Warehouse (DW) environments, operational processes move data from sources to the warehouse. This includes data export, preparation, and loading usually performed using Extraction, Transformation and Loading (ETL) tools. Past research has treated DW ”as collections of materialized views” whose data is regularly refreshed and locally stored [1]. Requirements have changed and real time transactions are required to support on-line operational decision making. Traditional DW systems may impose unacceptable delays due to their batch nature. ETL techniques are difficult to scale up to address the challenge of data loading, performance and low latency to provide real-time decision support. We propose a new approach for designing real-time DW in which traditional ETL does not apply. Data is pre-analysed by agents in each data source before being pushed as needed to the DW. The approach has been evaluated in a simulated environment and some of the results are discussed here.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Sellis, T., Simitsis, A.: ETL workflows: From formal specification to optimization. In: Ioannidis, Y., Novikov, B., Rachev, B. (eds.) ADBIS 2007. LNCS, vol. 4690, pp. 1–11. Springer, Heidelberg (2007)
Srinivasan, S., Krishna, V., Holmes, S.: Web-log-driven business activity monitoring. IEEE Computer Society 38(3), 61–68 (2005)
Jaorg, T., Dessloch, S.: Near real-time data warehousing using state-of-the-art ETL tools. In: Castellanos, M., Dayal, U., Miller, R.J. (eds.) BIRTE 2009. LNBIP, vol. 41, pp. 100–117. Springer, Heidelberg (2010)
Yan, Y., Li, W., Xu, J.: Information value-driven near real-time decision support systems. In: 29th IEEE international conference on Distributed Computing Systems, ICDCS 2009, pp. 571–578 (2009)
Simitsis, A., Vassiliadis, P., Sellis, T.: Optimizing etl processes in data warehouses. In: 21st International Conference on Data Engineering, pp. 564–575 (2005)
Sutherland, J., Van den Heuvel, W.J.: Clinical process and data integration and evolution. In: 40th Annual Hawaii International Conference on System Sciences in IEEE Database (2007)
Raden, N.: Exploring the business imperative of real-time analytics. Hired Brains, Inc. Implementing Business Analytics (2010)
Terr, S.: Real-time data warehousing, vol. 101 (2004)
Etzion, O.: On real-time, right-time, latency, throughput and other time-oriented measurements (2007)
Nelson, G., Wright, J.: Real time decision support: Creating a flexible architecture for real time analytics (2005)
Javed, M., Nawaz, A.: Data load distribution by semi real time data warehouse. In: Proceedings of the 2010 Second International Conference on Computer and Network Technology, pp. 556–560 (2010)
Langseth, J.: Real-time data warehousing: Challenges and solutions (2004)
Taylor, R.: Concurrency in the data warehouse. In: 36th International Conference on Very Large Data Bases, VLDB 2010, pp. 724–727 (2000)
Halevy, A., Rajaraman, A., Ordille, J.: Database integration: The teenage years. In: VLDB 2006 Proceedings of the 32nd International Conference on Very Large Databases, pp. 9–16 (2006)
Castellanos, M., Casati, F., Shan, M., Dayal, U.: ibom: A platform for intelligent business operation management. In: 21st International Conference on Data Engineering (2005)
Dayal, U., Castellanos, M., Simitsis, A., Wilkinson, K.: Data integration flows for business intelligence. In: Proceedings of the 12th International Conference on Extending Database Technology: Advances in Database Technology, pp. 1–11 (2009)
Chieu, T., Zneg, L.: Real time perfomance monitoring for an enterprice information managemetn system. In: IEEE International Conference on e-Business Engineering, pp. 429–434 (2008)
Chavez, E., Finnie, G.: Empowering data sources to manage clinical data. In: 23rd IEEE International Symposium on Computer-Based Medical Systems, CBMS 2010 (2010)
Spil, T., Stegwee, R., Teitink, C.: Business intelligence in healthcare organization. In: 35th Annual Hawaii Internation Conference on System Sciences, p. 142b (2002)
Ferdous, S., Fegaras, L., Makedon, F.: Applying data warehousing technique in pervasive assistive environment. In: Proceedings of the 3rd International Conference on PErvasive Technologies Related to Assistive Environments (2010)
Lee, H., Park, K., Lee, B., Choi, J., Elmasri, R.: Issues in data fusion for health care monitoring. In: Proceedings of the 1st International Conference on PErvasive Technologies Related to Assistive Environments, vol. 3 (2008)
Yang, H., Zheng, J., Jiang, Y., Peng, C., Xiao, S.: Selecting critical clinical features for heart diseases diagnosis with real-coded genetic algorithm. In: Applied Soft. Computing, vol. 8, pp. 1105–1111 (2008)
Bellifemine, F., Caire, G., Poggi, A., Rimassa, G.: Jade a white paper. Technical report, Telecom Italia Lab (2003)
Yin, Y., Papadias, D.: Just-in-time processing of continuous queries. In: IEEE 24th International Conference on Data Engineering, pp. 1150–1159 (2008)
In: Nascimento, M., Zsu, T., Kossmann, D., Miller, R., Blakeley, J., Schiefer, K. (eds.) Proceedings of the 30th International Conference on Very Large Databases. Morgan Kaufmann, San Francisco (2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Chávez, E., Finnie, G. (2011). TTL: A Transformation, Transference and Loading Approach for Active Monitoring. In: Cuzzocrea, A., Dayal, U. (eds) Data Warehousing and Knowledge Discovery. DaWaK 2011. Lecture Notes in Computer Science, vol 6862. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-23544-3_10
Download citation
DOI: https://doi.org/10.1007/978-3-642-23544-3_10
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-23543-6
Online ISBN: 978-3-642-23544-3
eBook Packages: Computer ScienceComputer Science (R0)