Definitions
A hybrid system based on traditional database extensions refers to a federated system between Hadoop and an enterprise data warehouse (EDW). In such a system, a query may need to combine data stored in both Hadoop and an EDW.
Overview
The co-existence of Hadoop and enterprise data warehouses (EDWs), together with the new application requirement of correlating data stored in both systems, has created the need for a special federation between Hadoop-like big data platforms and EDWs. This entry presents the motivation behind such hybrid systems, highlights the unique challenges of building them, surveys existing hybrid solutions, and finally discusses potential future directions.
Introduction
More and more enterprises today start to embrace Hadoop-like big data technologies to process huge volumes of data and drive actionable insights. The Hadoop Distributed File System (HDFS)...
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Adali S, Seluk Candan K, Papakonstantinou Y, Subrahmanian VS (1996) Query caching and optimization in distributed mediator systems. In: Proceedings of the 1996 ACM SIGMOD international conference on management of data (SIGMOD’96), pp 137–146
Armbrust M, Xin RS, Lian C, Huai Y, Liu D, Bradley JK, Meng X, Kaftan T, Franklin MJ, Ghodsi A, Zaharia M (2015) Spark SQL: relational data processing in spark. In: Proceedings of the 2015 ACM SIGMOD international conference on management of data (SIGMOD’15), pp 1383–1394
Beyer KS, Ercegovac V, Gemulla R, Balmin A, Eltabakh MY, Kanne C-C, Özcan F, Shekita EJ (2011) Jaql: a scripting language for large scale semistructured data analysis. PVLDB 4(12):1272–1283
Bloom BH (1970) Space/time trade-offs in hash coding with allowable errors. Commun ACM 13(7):422–426
DeWitt DJ, Halverson A, Nehme RV, Shankar S, Aguilar-Saborit J, Avanes A, Flasza M, Gramling J (2013) Split query processing in Polybase. In: Proceedings of the 2013 ACM SIGMOD international conference on management of data (SIGMOD’13), pp 1255–1266
Gray SC, Ozcan F, Pereyra H, van der Linden B, Zubiri A (2015) SQL-on-hadoop without compromise: how big SQL 3.0 from IBM represents an important leap forward for speed, portability and robust functionality in SQL-on-hadoop solutions. http://public.dhe.ibm.com/common/ssi/ecm/sw/en/sww14019usen/SWW14019USEN.PDF
Hive (2018) Apache Hive. https://hive.apache.org/
Hortonworks (2015) Modern data architecture with apache hadoop – the hybrid data warehouse. https:// www.denodo.com/en/system/files/document-attachme- nts/wp-hortonworks-01-ab.pdf
Josifovski V, Schwarz P, Haas L, Lin E (2002) Garlic: a new flavor of federated query processing for Db2. In: Proceedings of the 2002 ACM SIGMOD international conference on management of data (SIGMOD’02), pp 524–532
Kornacker M, Behm A, Bittorf V, Bobrovytsky T, Ching C, Choi A, Erickson J, Grund M, Hecht D, Jacobs M, Joshi I, Kuff L, Kumar D, Leblang A, Li N, Pandis I, Robinson H, Rorke D, Rus S, Russell J, Tsirogiannis D, Wanderman-Milne S, Yoder M (2015) Impala: a modern, open-source SQL engine for hadoop. In: Proceedings of the 2015 conference on innovative data systems research (CIDR)
McClary D (2014) Oracle big data SQL: one fast query, all your data. https://blogs.oracle.com/datawarehousing/entry/oracle_big_data_sql_one
Olston C, Reed B, Srivastava U, Kumar R, Tomkins A (2008) Pig latin: a not-so-foreign language for data processing. In: Proceedings of the 2008 ACM SIGMOD international conference on management of data (SIGMOD’08), pp 1099–1110
Oracle (2012) High performance connectors for load and access of data from Hadoop to Oracle database. http:// www.oracle.com/technetwork/bdc/hadoop-loader/con- nectors-hdfs-wp-1674035.pdf
Özcan F, Hoa D, Beyer KS, Balmin A, Liu CJ, Li Y (2011) Emerging trends in the enterprise data analytics: connecting hadoop and Db2 warehouse. In: Proceedings of the 2011 ACM SIGMOD international conference on management of data (SIGMOD’11), pp 1161–1164
Papakonstantinou Y, Gupta A, Garcia-Molina H, Ullman JD (1995) A query translation scheme for rapid implementation of wrappers. In: Proceedings of the 1995 international conference on deductive and object-oriented databases (DOOD’95), pp 161–186
Pig (2018) Apache Pig. https://pig.apache.org
Portnoy D (2013) The furture of hybrid data warehouse-Hadoop implementations. https://www.slideshare.net/ DavidPortnoy/hybrid-data-warehouse-hadoop-implem- entations
Shan M-C, Ahmed R, Davis J, Du W, Kent W (1995) Pegasus: a heterogeneous information management system. In: Kim W (ed) Modern database systems. ACM Press/Addison-Wesley Publishing Co., New York/Reading, pp 664–682
Shankar R (2015) Top reasons for powering your enterprise data warehouse with hadoop. http://www.cignex. com/blog/top-reasons-powering-your-enterprise-data- warehouse-hadoop
SparkSQL (2018) Spark SQL. https://spark.apache.org/sql
Sqoop (2018) Apache Sqoop. http://sqoop.apache.org
Teradata (2013) Teradata connector for Hadoop. http://developer.teradata.com/connectivity/articles/teradata-connector-for-hadoop-now-available
Teradata (2016) Take a giant step with Teradata QueryGrid. http://blogs.teradata.com/data-points/take-a-giant-step-with-teradata-querygrid
Thusoo A, Sarma JS, Jain N, Shao Z, Chakka P, Anthony S, Liu H, Wyckoff P, Murthy R (2009) Hive: a warehousing solution over a map-reduce framework. Proc VLDB Endow (PVLDB) 2(2):1626–1629
Tian Y, Zou T, Ozcan F, Goncalves R, Pirahesh H (2015) Joins for hybrid warehouses: exploiting massive parallelism in hadoop and enterprise data warehouses. In: Proceedings of the 2015 international conference on extending database technology (EDBT’15), pp 373–384
Tian Y, Özcan F, Zou T, Goncalves R, Pirahesh H (2016) Building a hybrid warehouse: efficient joins between data stored in hdfs and enterprise warehouse. ACM Trans Database Syst 41(4):21:1–21:38
Tomasic A, Raschid L, Valduriez P (1998) Scaling access to heterogeneous data sources with DISCO. IEEE Trans Know Data Eng (TKDE). 10(5):808–823
Traverso M (2013) Presto: interacting with petabytes of data at facebook. https://www.facebook.com/notes/ facebook-engineering/presto-interacting-with-petaby- tes-of-data-at-facebook/10151786197628920
Vertica (2016) Hadoop integration guide – HP Vertica analytic database. https://my.vertica.com/docs/7.0.x/PDF/HP_Vertica_7.0.x_HadoopIntegration.pdf
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this entry
Cite this entry
Tian, Y. (2019). Hybrid Systems Based on Traditional Database Extensions. In: Sakr, S., Zomaya, A.Y. (eds) Encyclopedia of Big Data Technologies. Springer, Cham. https://doi.org/10.1007/978-3-319-77525-8_254
Download citation
DOI: https://doi.org/10.1007/978-3-319-77525-8_254
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-77524-1
Online ISBN: 978-3-319-77525-8
eBook Packages: Computer ScienceReference Module Computer Science and Engineering