Agile Query Processing in Statistical Databases: A Process-In-Memory Approach

  • Shanshan Lu
  • Peiquan JinEmail author
  • Lin Mu
  • Shouhong Wan
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11775)


Statistical database systems are designed to answer queries on summarized data (or macro data), while queries on raw records are not allowed in such database systems. As macro data can offer aggregate information about the database, it is also an effective way to use statistical queries to provide analytical results in semantic databases. However, traditional statistical databases are proposed for security protection, i.e., hiding the raw records from user queries. Few studies are toward query optimizations on aggregate queries in statistical databases. In this paper, we propose a new process-in-memory (PIM) based processing scheme called agile query for accelerating queries in statistical databases. We present two new designs in the agile query. First, we propose an in-memory index to cache aggregate operators (e.g., sum, min, max, count, and average) in the main memory. The aggregate queries that hit in the in-memory index can be evaluated in the memory and no I/O operation will be incurred. Second, we propose to incrementally update the in-memory operator index so that we can ensure the consistency between the cached data and the original data records. We implement the agile query processing framework on top of MySQL and conduct experiments over various sizes of datasets to compare our design with the traditional method in MySQL. The results show that our proposal achieves up to 9 times higher throughput than MySQL under the skewed Zipf query set, and averagely gets about 2 times higher throughput under the random and uniform distributed queries.


Query processing Statistical database Processing in memory 



This work is partially supported by the National Key Research and Development Program of China (2018YFB0704404) and the National Science Foundation of China (61672479).


  1. 1.
    Gupta, A.: Statistical data management. In: Liu, L., Özsu, M.T. (eds.) Encyclopedia of Database Systems, 2nd edn. Springer, Boston (2018)Google Scholar
  2. 2.
    Rafanelli, M., Shoshani, A.: Storm: a statistical object representation model. In: Michalewicz, Z. (ed.) SSDBM 1990. LNCS, vol. 420, pp. 14–29. Springer, Heidelberg (1990). Scholar
  3. 3.
    Shoshani, A.: OLAP and statistical databases: similarities and differences. In: PODS, pp. 185–196 (1997)Google Scholar
  4. 4.
    Brankovic, L., Giggins, H.: Statistical database security. In: Petković, M., Jonker, W. (eds.) Security, Privacy, and Trust in Modern. Data Management Data-Centric Systems and Applications, pp. 167–181. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  5. 5.
    Shoshani, A., Olken, F., Wong, H.K.T.: Characteristics of scientific databases. In: VLDB, pp. 147–160 (1984)Google Scholar
  6. 6.
    Shoshani A. Statistical databases: characteristics, problems, and some solutions. In: VLDB, pp. 208–222 (1982)Google Scholar
  7. 7.
    Shoshani, A., Wong, H.K.T.: Statistical and scientific database issues. IEEE Trans. Software Eng. 10, 1040–1047 (1985)CrossRefGoogle Scholar
  8. 8.
    Lu, H., Vaidya, J., et al.: Statistical database auditing without query denial threat. INFORMS J. Comput. 27(1), 20–34 (2015)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Ryan, J.: A brief survey on the contribution of Mirka Miller to the security of statistical databases. Math. Comput. Sci. 12(3), 255–262 (2018)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Domingo-Ferrer, J.: Inference control in statistical databases. In: Liu, L., Özsu, M.T. (eds.) Encyclopedia of Database Systems, 2nd edn. Springer, Boston (2018)Google Scholar
  11. 11.
    Aggarwal, C.C., Yu, P.S.: A condensation approach to privacy preserving data mining. In: Bertino, E., et al. (eds.) EDBT 2004. LNCS, vol. 2992, pp. 183–199. Springer, Heidelberg (2004). Scholar
  12. 12.
    Skinner, G., Chang, E., et al.: Shield privacy Hippocratic security method for virtual community. In: IECON, pp. 472–479 (2004)Google Scholar
  13. 13.
    Baranczyk, S., Konik, R., et al.: Forecasting query access plan obsolescence: U.S. Patent 9, 990, 396 (2018)Google Scholar
  14. 14.
    Xike, X., Xingjun, H., Torben, P., Peiquan, J., Jinchuan, C.: OLAP over probabilistic data cubes I: Aggregating, materializing, and querying. In: ICDE, pp. 799–810 (2016)Google Scholar
  15. 15.
    Cormode, G., Korn, F., Muthukrishnan, S., Srivastava, D.: Summarizing two-dimensional data with skyline-based statistical descriptors. In: Ludäscher, B., Mamoulis, N. (eds.) SSDBM 2008. LNCS, vol. 5069, pp. 42–60. Springer, Heidelberg (2008). Scholar
  16. 16.
    Gemulla, R., Rösch, P., Lehner, W.: Linked bernoulli synopses: sampling along foreign keys. In: Ludäscher, B., Mamoulis, N. (eds.) SSDBM 2008. LNCS, vol. 5069, pp. 6–23. Springer, Heidelberg (2008). Scholar
  17. 17.
    Singh, S., Mayfield, C., Shah, R., Prabhakar, S., Hambrusch, S.: Query selectivity estimation for uncertain data. In: Ludäscher, B., Mamoulis, N. (eds.) SSDBM 2008. LNCS, vol. 5069, pp. 61–78. Springer, Heidelberg (2008). Scholar
  18. 18.
    Wang, F., Agrawal, G., Jin, R.: Query planning for searching inter-dependent deep-web databases. In: Ludäscher, B., Mamoulis, N. (eds.) SSDBM 2008. LNCS, vol. 5069, pp. 24–41. Springer, Heidelberg (2008). Scholar
  19. 19.
    Zhi, L., Peiquan, J., Xuan, S., et al.: CCF-LRU: a new buffer replacement algorithm for flash memory. IEEE Trans. Consum. Electron. 55(3), 1351–1359 (2009)CrossRefGoogle Scholar
  20. 20.
    Peiquan, J., Xike, X., Na, W., Lihua, Y.: Optimizing R-tree for flash memory. Expert Syst. Appl. 42(10), 4676–4686 (2015)CrossRefGoogle Scholar
  21. 21.
    Peiquan, J., Yi, O., Theo, H., Zhi, L.: AD-LRU: an efficient buffer replacement algorithm for flash-based databases. Data Knowl. Eng. 72, 83–102 (2012)CrossRefGoogle Scholar
  22. 22.
    Chen, K., Jin, P., Yue, L.: A novel page replacement algorithm for the hybrid memory architecture involving PCM and DRAM. In: Hsu, C.-H., Shi, X., Salapura, V. (eds.) NPC 2014. LNCS, vol. 8707, pp. 108–119. Springer, Heidelberg (2014). Scholar
  23. 23.
    Wu, Z., Jin, P., Yang, C., Yue, L.: APP-LRU: a new page replacement method for PCM/DRAM-based hybrid memory systems. In: Hsu, C.-H., Shi, X., Salapura, V. (eds.) NPC 2014. LNCS, vol. 8707, pp. 84–95. Springer, Heidelberg (2014). Scholar
  24. 24.
    Talati, N., Ali, A., et al.: Practical challenges in delivering the promises of real processing-in-memory machines. In: DATE, pp. 1628–1633 (2018)Google Scholar
  25. 25.
    Ahn, J., Yoo, S., Mutlu, O., et al.: PIM-enabled instructions: a low-overhead, locality-aware processing-in-memory architecture. In: ISCA, pp. 336–348 (2015)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Shanshan Lu
    • 1
  • Peiquan Jin
    • 1
    • 2
    Email author
  • Lin Mu
    • 1
  • Shouhong Wan
    • 1
    • 2
  1. 1.University of Science and Technology of ChinaHefeiChina
  2. 2.Key Laboratory of Electromagnetic Space InformationChinese Academy of SciencesHefeiChina

Personalised recommendations