Abstract
This paper describes and analyses optimization approaches, which make possible the exact calculation of millions of hierarchical count distinct measures over hundreds of billions data rows. Described approach evolved for several years, in parallel with the growth of tasks from a fast growing internet company, and was finally implemented as a PEAPM (Pipelined Exact Accumulation for Paralleled Measures) algorithm. Current version of an algorithm outputs exact values (not estimates), works in a single thread, in minutes using a general commodity hardware, and requires volume of RAM equal to the doubled size of required measures.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Flajolet, P., Fusy, É., Gandouet, O., Meunier, F.: HyperLogLog: the analysis of a near-optimal cardinality estimation algorithm. In: Discrete Mathematics and Theoretical Computer Science Proceedings, Nancy, France, AH, pp. 127–146. CiteSeerX 10.1.1.76.4286
Wang, G., Zhang, X., Tang, S., Wilson, C., Zheng, H., Zhao, B.Y.: Clickstream user behavior models. ACM Trans. Web 11(4), 1–37 (2017)
Banerjee, A., Ghosh, J.: Clickstream clustering using weighted longest common subsequences. In: Proceedings of Web Mining Workshop at the 1st SIAM Conference on Data Mining (2001)
Rönnbäck, L., Regardt, O., Bergholtz, M., Johannesson, P., Wohed, P.: Anchor modeling agile information modeling in evolving data environments. Data Knowl. Eng. 69(12), 1229–1253 (2010)
Golov, N., Ronnback, L.: Big data normalization for massively parallel processing databases. Comput. Stand. Interfaces 54(2), 86–93 (2017). https://doi.org/10.1016/j.csi.2017.01.009
Naspers takes full control of Russian classifieds site Avito in 1.16B dollars deal. https://techcrunch.com/2019/01/28/naspers-avito-1-16-billion/
Benchmarks of modern analytical databases for typical click stream analysis scenarious. https://clickhouse.yandex/benchmark.html
C++ Vertica extension implementing described algorithm. https://github.com/phil-88/vertica-udf
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Golov, N., Filatov, A., Bruskin, S. (2019). Efficient Exact Algorithm for Count Distinct Problem. In: England, M., Koepf, W., Sadykov, T., Seiler, W., Vorozhtsov, E. (eds) Computer Algebra in Scientific Computing. CASC 2019. Lecture Notes in Computer Science(), vol 11661. Springer, Cham. https://doi.org/10.1007/978-3-030-26831-2_5
Download citation
DOI: https://doi.org/10.1007/978-3-030-26831-2_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-26830-5
Online ISBN: 978-3-030-26831-2
eBook Packages: Computer ScienceComputer Science (R0)