Entropy Aware Adaptive Compression for SQL Column Stores

  • K. T. SridharEmail author
  • Jimson Johnson
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 928)


With the advent of SQL column stores, compression has gained renewed interest and drawn considerable attention from both academia and industry. Unlike row stores, column stores use lightweight compression methods and, generally, compression granularity is at entire column level. In this paper we outline and explore an alternative compression strategy for column stores that works at a different granularity and adapts itself to data, on-the-fly, using a compression planner. The approach yields good compression ratios, facilitates compression during bulk data load and also mitigates some issues that arise from having to maintain global meta-data on compression. We describe its implementation in analytics database dbX, a cloud agnostic, columnar MPP SQL product and present experimental results.


Column store Database compression Compression planner 



We thank M.A. Sakkeer for his feedback in using modules of compression and decompression in bulk loading and query execution; Dipanjan Deb for operational cloud support and help with AWS runs.


  1. 1.
    Abadi, D., Boncz, P.A., Harizopoulos, S., Idreos, S., Madden, S.: The design and implementation of modern column-oriented database systems. Found. Trends Databases 5(3), 197–280 (2013)CrossRefGoogle Scholar
  2. 2.
    Abadi, D.J., Madden, S., Ferreira, M.: Integrating compression and execution in column-oriented database systems. In: SIGMOD 2006, pp. 671–682. ACM (2006)Google Scholar
  3. 3.
    Abadi, D.J., Madden, S., Hachem, N.: Column-stores vs. row-stores: how different are they really? In: SIGMOD 2008, pp. 967–980. ACM (2008)Google Scholar
  4. 4.
    Baklarz, G.: DB2 compression estimation tool. Technical report, IBM Corporation, Canada, October 2016Google Scholar
  5. 5.
    Binnig, C., Hildenbrand, S., Färber, F.: Dictionary-based order-preserving string compression for main memory column stores. In: SIGMOD 2009, pp. 283–296. ACM (2009)Google Scholar
  6. 6.
    Copeland, G.P., Khoshafian, S.: A decomposition storage model. In: SIGMOD, pp. 268–279. ACM (1985)Google Scholar
  7. 7.
    Damme, P., Habich, D., Hildebrandt, J., Lehner, W.: Lightweight data compression algorithms: an experimental survey. In: Proceedings of 20th EDBT, pp. 72–83 (2017)Google Scholar
  8. 8.
    Flajolet, P., Fusy, E., Gandouet, O., Mennier, F.: HyperLogLog: the analysis of a near-optimal cardinality estimation algorithm. In: Analysis of Algorithms 2007 (AofA07), pp. 127–146 (2007)Google Scholar
  9. 9.
    Goldstein, J., Ramakrishnan, R., Shaft, U.: Compressing relations and indexes. In: Proceedings of 14th ICDE, pp. 370–379. IEEE (1998)Google Scholar
  10. 10.
    Graefe, G., Shapiro, L.: Data compression and database performance. In: Proceedings of ACM/IEEE-CS Symposium on Applied Computing, pp. 22–27. IEEE (1991)Google Scholar
  11. 11.
    Hovestadt, M., Kao, O., Kliem, A., Warneke, D.: Evaluating adaptive compression to mitigate the effects of shared I/O in clouds. In: 25th IEEE IPDPS, pp. 1042–1051. IEEE (2011)Google Scholar
  12. 12.
    Idreos, S., Groffen, F., Nes, N., Manegold, S., Mullender, K.S., Kersten, M.L.: MonetDB: two decades of research in column-oriented database architectures. IEEE Data Eng. Bull. 35(1), 40–45 (2012)Google Scholar
  13. 13.
    Iyer, B.R., Wilhite, D.: Data compression support in databases. In: Proceedings of 20th VLDB, pp. 695–704 (1994)Google Scholar
  14. 14.
    Krintz, C., Sucu, S.: Adaptive on-the-fly compression. IEEE Trans. Parallel Distrib. Syst. 17(1), 15–24 (2006)CrossRefGoogle Scholar
  15. 15.
    Krueger, J., Grund, M., Tinnefeld, C., Plattner, H., Zeier, A., Faerber, F.: Optimizing write performance for read optimized databases. In: Kitagawa, H., Ishikawa, Y., Li, Q., Watanabe, C. (eds.) DASFAA 2010. LNCS, vol. 5982, pp. 291–305. Springer, Heidelberg (2010). Scholar
  16. 16.
    Lang, H., Mühlbauer, T., Funke, F., Boncz, P.A., Neumann, T., Kemper, A.: Data blocks: hybrid OLTP and OLAP on compressed storage using both vectorization and compilation. In: SIGMOD 2016, pp. 311–326. ACM (2016)Google Scholar
  17. 17.
    Lelewer, D.A., Hirschberg, D.S.: Data compression. ACM Comput. Surv. 19(3), 261–296 (1987)CrossRefGoogle Scholar
  18. 18.
    Poess, M., Potapov, D.: Data compression in oracle. In: Proceedings of 29th VLDB, pp. 761–770 (2003)Google Scholar
  19. 19.
    Raman, V., et al.: DB2 with BLU acceleration: so much more than just a column store. PVLDB 6(11), 1080–1091 (2013)Google Scholar
  20. 20.
    Sridhar, K.T., Sakkeer, M.A.: Optimizing database load and extract for big data era. In: Bhowmick, S.S., Dyreson, C.E., Jensen, C.S., Lee, M.L., Muliantara, A., Thalheim, B. (eds.) DASFAA 2014. LNCS, vol. 8422, pp. 503–512. Springer, Cham (2014). Scholar
  21. 21.
    Stonebraker, M., et al.: C-store: a column-oriented DBMS. In: Proceedings of the 31st International Conference on Very Large Data Bases, pp. 553–564. VLDB Endowment (2005)Google Scholar
  22. 22.
    TeraData: compression types supported by teradata database, release 16.0 (2016).
  23. 23.
    Viglas, S.: Just-in-time compilation for SQL query processing. PVLDB 6(11), 1190–1191 (2013)Google Scholar
  24. 24.
    Wust, J., Krüger, J., Gund, M., Hartmann, U., Plattner, H.: Sparse dictionaries for in-memory column stores. In: Proceedings of 4th DBKDA, pp. 25–33. IARIA (2012)Google Scholar
  25. 25.
    Zukowski, M., Héman, S., Nes, N., Boncz, P.A.: Super-scalar RAM-CPU cache compression. In: Proceedings of 22nd ICDE, pp. 59–71. IEEE (2006)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.XtremeData TechnologiesBangaloreIndia
  2. 2.XtremeData, Inc.SchaumburgUSA

Personalised recommendations