Abstract
Real time business intelligence demands to execute OLAP queries on a current, up-to-date state of the transactional OLTP data. The currently exercised separation of transaction processing on the OLTP database and BI query processing on the data warehouse that is only periodically refreshed violates this goal. We propose to enhance the transactional database with highly effective query processing capabilities. We contrast different architectures proposed for achieving the real-time BI goal: versioning of the data and thereby separating the query from the transactions workload, continuous DW refreshing, heterogeneous workload management, update staging by periodically merging the update delta into the queryable main database, update and query batching, and our newly developed virtual memory snapshot mechanism based on hardware-supported shadowed pages. In our HyPer main memory database management system, the OLAP queries are executed on an arbitrarily current database snapshot that is created by forking the OLTP process. This snapshot is efficiently maintained by the operating system’s copy on write mechanism that is supported by the MMU hardware. To assess the performance of such hybrid DBMSs we propose a new OLTP&OLAP benchmark that combines the transaction processing functionality of the TPC-C benchmark with the query workload of the TPC-H benchmark in one mixed workload. Based on this benchmark we substantiate the claim that it is indeed possible to architect a hybrid system that can achieve the transaction throughput of dedicated in-memory OLTP systems and, in parallel, execute a BI workload on the same data at the same performance as dedicated OLAP systems, such as in-memory column stores.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Abouzour, M., Salem, K., Bumbulis, P.: Automatic Tuning of the Multiprogramming Level in Sybase SQL Anywhere. In: Proc. of the 2010 Workshop on Self-Managing Database Systems, SMDB (2010)
Aulbach, S., Seibold, M., Jacobs, D., Kemper, A.: Extesibility and Data Sharing in Evolving Multi-Tenancy Dtabases. In: ICDE (2011)
Berenson, H., Bernstein, P.A., Gray, J., Melton, J., O’Neil, E.J., O’Neil, P.E.: A Critique of ANSI SQL Isolation Levels. In: SIGMOD (1995)
Binnig, C., Hildenbrand, S., Färber, F.: Dictionary-based order-preserving string compression for main memory column stores. In: SIGMOD (2009)
Boncz, P.A., Manegold, S., Kersten, M.L.: Database Architecture Evolution: Mammals Flourished long before Dinosaurs became Extinct. PVLDB 2(2) (2009)
Comeau, B.: Introduction to DB2 9.5 Workload Management. Oracle (June 2007)
Curino, C., Zhang, Y., Jones, E.P.C., Madden, S.: Schism: a Workload-Driven Approach to Database Replication and Partitioning. In: VLDB 2010 (2010)
Funke, F., Kemper, A., Neumann, T.: Benchmarking Hybrid OLTP&OLAP Database Systems. In: The 14th BTW Conference on Database Systems for Business, Technology, and Web (BTW 2011) of the Gesellschaft für Informatik, GI (2011)
Harizopoulos, S., Abadi, D.J., Madden, S., Stonebraker, M.: OLTP through the looking glass, and what we found there. In: SIGMOD (2008)
Gmach, D., Krompass, S., Scholz, A., Wimmer, M., Kemper, A.: Adaptive quality of service management for enterprise services. ACM Trans. WEB 2(1) (2008)
Harizopoulos, S., Abadi, D.J., Madden, S., Stonebraker, M.: OLTP through the looking glass, and what we found there. In: SIGMOD (2008)
Héman, S., Zukowski, M., Nes, N.J., Sidirourgos, L., Boncz, P.A.: Positional update handling in column stores. In: SIGMOD Conference, pp. 543–554 (2010)
Héman, S., Zukowski, M., Nes, N.J., Sidirourgos, L., Boncz, P.A.: Positional update handling in column stores. In: SIGMOD Conference, pp. 543–554 (2010)
Jones, E.P.C., Abadi, D.J., Madden, S.: Low overhead concurrency control for partitioned main memory databases. In: SIGMOD (2010)
Johnson, R., Pandis, I., Stoica, R., Athanassoulis, M., Ailamaki, A.: Aether: A scalable approach to logging. In: VLDB (2010)
Krueger, J., Grund, M., Tinnefeld, C., Plattner, H., Zeier, A., Faerber, F.: Optimizing write performance for read optimized databases. In: Kitagawa, H., Ishikawa, Y., Li, Q., Watanabe, C. (eds.) DASFAA 2010. LNCS, vol. 5982, pp. 291–305. Springer, Heidelberg (2010)
Kallman, R., Kimura, H., Natkins, J., Pavlo, A., Rasin, A., Zdonik, S.B., Jones, E.P.C., Madden, S., Stonebraker, M., Zhang, Y., Hugg, J., Abadi, D.J.: H-Store: a high-performance, distributed main memory transaction processing system. PVLDB 1(2) (2008)
Krompass, S., Kuno, H.A., Wiener, J.L., Wilkinson, K., Dayal, U., Kemper, A.: A Testbed for Managing Dynamic Mixed Workloads. PVLDB 2(2) (2009)
Kemper, A., Neumann, T.: HyPer: A Hybrid OLTP&OLAP Main Memory Database System Based on Virtual Memory Snapshots. In: ICDE (2011)
Lorie, R.A.: Physical Integrity in a Large Segmented Database. TODS 2(1) (1977)
Neumann, T., Weikum, G.: x-RDF-3X: Fast Querying, High Update Rates, and Consistency for RDF Databases. In: VLDB (2010)
Ousterhout, J.K., Agrawal, P., Erickson, D., Kozyrakis, C., Leverich, J., Mazières, D., Mitra, S., Narayanan, A., Parulkar, G.M., Rosenblum, M., Rumble, S.M., Stratmann, E., Stutsman, R.: The case for RAMClouds: scalable high-performance storage entirely in DRAM. Operating Systems Review 43(4) (2009)
Oracle. Change Data Capture (2007), http://download.oracle.com/docs/cd/B28359_01/server.111/b28313/cdc.htm
Plattner, H.: A common database approach for OLTP and OLAP using an in-memory column database. In: SIGMOD (2009)
Schroeder, B., Harchol-Balter, M., Iyengar, A., Nahum, E.M., Wierman, A.: How to Determine a Good Multi-Programming Level for External Scheduling. In: ICDE, p. 60 (2006)
Stonebraker, M., Rowe, L.A., Hirohama, M.: The Implementation of Postgres. IEEE Trans. Knowl. Data Eng. 2(1), 125–142 (1990)
Tozer, S., Brecht, T., Aboulnaga, A.: Q-Cop: Avoiding bad query mixes to minimize client timeouts under heavy loads. In: ICDE, pp. 397–408 (2010)
VoltDB. Overview (March 2010), http://www.voltdb.com/_pdf/VoltDBOverview.pdf
VoltDB. VoltDB TPC-C-like Benchmark Comparison-Benchmark Description (May 2010), https://community.voltdb.com/node/134
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kemper, A., Neumann, T. (2011). One Size Fits all, Again! The Architecture of the Hybrid OLTP&OLAP Database Management System HyPer. In: Castellanos, M., Dayal, U., Markl, V. (eds) Enabling Real-Time Business Intelligence. BIRTE 2010. Lecture Notes in Business Information Processing, vol 84. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-22970-1_2
Download citation
DOI: https://doi.org/10.1007/978-3-642-22970-1_2
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-22969-5
Online ISBN: 978-3-642-22970-1
eBook Packages: Computer ScienceComputer Science (R0)