Skip to main content
SpringerLink
Log in

Stable Distributions in Streaming Computations

  • Chapter
  • First Online:
Data Stream Management

Part of the book series: Data-Centric Systems and Applications ((DCSA))

Abstract

In many streaming scenarios, we need to measure and quantify the data that is seen. For example, we may want to measure the number of distinct IP addresses seen over the course of a day, compute the difference between incoming and outgoing transactions in a database system or measure the overall activity in a sensor network. In all of these examples, data can be modeled as massive, dynamic vectors. Here, we are interested in the well-known and widely used \(L_{p}\) norms of such vectors. These encompass the familiar Euclidean (root of sum of squares) and Manhattan (sum of absolute values) norms. We show how such \(L_{p}\) norms can be efficiently estimated for massive vectors presented in the streaming model. This is achieved by making succinct sketches of the data, which can be used as synopses of the vectors they summarize.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 54.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 69.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 99.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. N. Ailon, B. Chazelle, Approximate nearest neighbors and the fast Johnson–Lindenstrauss transform, in Proceedings of the ACM Symposium on Theory of Computing (2006)

    Google Scholar 

  2. N. Alon, Y. Matias, M. Szegedy, The space complexity of approximating the frequency moments, in Proceedings of the ACM Symposium on Theory of Computing (1996), pp. 20–29. Journal version in J. Comput. Syst. Sci. 58, 137–147 (1999)

    Google Scholar 

  3. Z. Bar-Yossef, T.S. Jayram, R. Kumar, D. Sivakumar, L. Trevisian, Counting distinct elements in a data stream, in Proceedings of RANDOM 2002 (2002), pp. 1–10

    Google Scholar 

  4. B. Brinkman, M. Charikar, On the impossibility of dimensionality reduction in \(L_{1}\), in IEEE Conference on Foundations of Computer Science (2003), pp. 514–523

    Google Scholar 

  5. A. Chakrabarti, G. Cormode, A. McGregor, A near-optimal algorithm for computing the entropy of a stream, in Proceedings of ACM-SIAM Symposium on Discrete Algorithms (2007)

    Google Scholar 

  6. J.M. Chambers, C.L. Mallows, B.W. Stuck, A method for simulating stable random variables. J. Am. Stat. Assoc. 71(354), 340–344 (1976)

    Article  MathSciNet  MATH  Google Scholar 

  7. M. Charikar, K. Chen, M. Farach-Colton, Finding frequent items in data streams, in Procedings of the International Colloquium on Automata, Languages and Programming (ICALP) (2002), pp. 693–703

    Chapter  Google Scholar 

  8. M.S. Charikar, Similarity estimation techniques from rounding algorithms, in Proceedings of the ACM Symposium on Theory of Computing (2002), pp. 380–388

    Google Scholar 

  9. G. Cormode, Sequence distance embeddings. PhD thesis, University of Warwick (2003)

    Google Scholar 

  10. G. Cormode, M. Datar, P. Indyk, S. Muthukrishnan, Comparing data streams using Hamming norms, in Proceedings of the International Conference on Very Large Data Bases (2002), pp. 335–345. Journal version in IEEE Trans. Knowl. Data Eng. 15(3), 529–541 (2003)

    Chapter  Google Scholar 

  11. G. Cormode, P. Indyk, N. Koudas, S. Muthukrishnan, Fast mining of tabular data via approximate distance computations, in Proceedings of the International Conference on Data Engineering (2002), pp. 605–616

    Chapter  Google Scholar 

  12. G. Cormode, S. Muthukrishnan, The string edit distance matching problem with moves, in Proceedings of ACM–SIAM Symposium on Discrete Algorithms (2002), pp. 667–676

    Google Scholar 

  13. G. Cormode, S. Muthukrishnan, Estimating dominance norms of multiple data streams, in Proceedings of the European Symposium on Algorithms (ESA). LNCS, vol. 2838 (2003)

    Google Scholar 

  14. G. Cormode, S. Muthukrishnan, S.C. Ṣahinalp, Permutation editing and matching via embeddings, in Proceedings of 28th International Colloquium on Automata, Languages and Programming, vol. 2076 (2001), pp. 481–492

    Chapter  Google Scholar 

  15. C. Cranor, T. Johnson, O. Spatscheck, V. Shkapenyuk, Gigascope: a stream database for network applications, in Proceedings of ACM SIGMOD International Conference on Management of Data (2003), pp. 647–651

    Google Scholar 

  16. M. Datar, N. Immorlica, P. Indyk, V.S. Mirrokni, Locality-sensitive hashing scheme based on \(p\)-stable distributions, in Symposium on Computational Geometry (2004)

    Google Scholar 

  17. C. Estan, G. Varghese, M. Fisk, Bitmap algorithms for counting active flows on high speed links, in Proceedings of the Internet Measurement Conference (2003), pp. 153–166

    Chapter  Google Scholar 

  18. J. Feigenbaum, S. Kannan, M. Strauss, M. Viswanathan, An approximate \(L_{1}\)-difference algorithm for massive data streams, in IEEE Conference on Foundations of Computer Science (1999), pp. 501–511

    Google Scholar 

  19. P. Flajolet, G.N. Martin, Probabilistic counting, in IEEE Conference on Foundations of Computer Science (1983), pp. 76–82. Journal version in J. Comput. Syst. Sci. 31, 182–209 (1985)

    Google Scholar 

  20. J. Fong, M. Strauss, An approximate \(L_{p}\)-difference algorithm for massive data streams, in Symposium on Theoretical Aspects of Computer Science (STACS) (2000), pp. 193–204

    Google Scholar 

  21. S. Ganguly, B. Lakshminath, Estimating entropy over data streams, in Proceedings of the European Symposium on Algorithms (ESA) (2006)

    Google Scholar 

  22. M. Garofalakis, A. Kumar, Correlating XML data streams using tree-edit distance embeddings, in Proceedings of ACM Principles of Database Systems (2003), pp. 143–154

    Google Scholar 

  23. P. Gibbons, Distinct sampling for highly-accurate answers to distinct values queries and event reports, in Proceedings of the International Conference on Very Large Data Bases (2001), pp. 541–550

    Google Scholar 

  24. P. Gibbons, S. Tirthapura, Estimating simple functions on the union of data streams, in ACM Symposium on Parallel Algorithms and Architectures (SPAA) (2001), pp. 281–290

    Google Scholar 

  25. A. Gilbert, S. Guha, P. Indyk, Y. Kotidis, S. Muthukrishnan, M. Strauss, Fast, small-space algorithms for approximate histogram maintenance, in Proceedings of the ACM Symposium on Theory of Computing (2002), pp. 389–398

    Google Scholar 

  26. P. Indyk, Stable distributions, pseudorandom generators, embeddings and data stream computation, in IEEE Conference on Foundations of Computer Science (2000), pp. 189–197

    Google Scholar 

  27. P. Indyk, Algorithmic aspects of geometric embeddings (invited tutorial), in IEEE Conference on Foundations of Computer Science (2001), pp. 10–35

    Google Scholar 

  28. P. Indyk, Algorithms for dynamic geometric problems over data streams, in Proceedings of the ACM Symposium on Theory of Computing (2004)

    Google Scholar 

  29. P. Indyk, R. Motwani, Approximate nearest neighbors: towards removing the curse of dimensionality, in Proceedings of the ACM Symposium on Theory of Computing (1998), pp. 604–613

    Google Scholar 

  30. W.B. Johnson, J. Lindenstrauss, Extensions of Lipschitz mapping into Hilbert space. Contemp. Math. 26, 189–206 (1984)

    Article  MathSciNet  MATH  Google Scholar 

  31. P. Li, Very sparse stable random projections, estimators and tail bounds for stable random projections. Technical report (2006). arXiv:cs.DS/0611114

  32. P. Li, T. Hastie, K.W. Church, Nonlinear estimators and tail bounds for dimension reduction in \(L_{1}\) using Cauchy random projections. J. Mach. Learn. Res. (2007)

    Google Scholar 

  33. J. Matoušek, Lectures on Discrete Geometry (Springer, Berlin, 2002)

    Book  MATH  Google Scholar 

  34. R. Motwani, P. Raghavan, Randomized Algorithms (Cambridge University Press, Cambridge, 1995)

    Book  MATH  Google Scholar 

  35. S. Muthukrishnan, Data streams: algorithms and applications, in Proceedings of ACM–SIAM Symposium on Discrete Algorithms (2003)

    Google Scholar 

  36. A. Naor, G. Schechtman, Planar earthmover is not in \(L_{1}\), in IEEE Conference on Foundations of Computer Science (2006)

    Google Scholar 

  37. N. Nisan, Pseudorandom generators for space-bounded computation. Combinatorica 12, 449–461 (1992)

    Article  MathSciNet  MATH  Google Scholar 

  38. J. Nolan, Stable distributions. Available from http://academic2.american.edu/~jpnolan/stable/chap1.ps

  39. M. Thorup, Y. Zhang, Tabulation based 4-universal hashing with applications to second moment estimation, in Proceedings of ACM-SIAM Symposium on Discrete Algorithms (2004), pp. 615–624

    Google Scholar 

  40. V.V. Uchaikin, V.M. Zolotarev, Chance and Stability: Stable Distributions and Their Applications (VSP, Utrecht, 1999)

    Book  MATH  Google Scholar 

  41. D. Woodruff, Optimal space lower bounds for all frequency moments, in Proceedings of ACM–SIAM Symposium on Discrete Algorithms (2004), pp. 167–175

    Google Scholar 

  42. V.M. Zolotarev, One Dimensional Stable Distributions. Translations of Mathematical Monographs, vol. 65 (Am. Math. Soc., Providence, 1983)

    MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Warwick, Coventry, CV4 7AL, UK

    Graham Cormode

  2. Laboratory for Computer Science, Massachusetts Institute of Technology, Cambridge, MA, USA

    Piotr Indyk

Authors
  1. Graham Cormode
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Piotr Indyk
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Graham Cormode .

Editor information

Editors and Affiliations

  1. University Campus - Kounoupidiana, School of ECE, Techn. Univ. of Crete University Campus - Kounoupidiana, Chania, Greece

    Minos Garofalakis

  2. Microsoft Corporation, Redmond, Washington, USA

    Johannes Gehrke

  3. Amazon India , Bangalore, India

    Rajeev Rastogi

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Cormode, G., Indyk, P. (2016). Stable Distributions in Streaming Computations. In: Garofalakis, M., Gehrke, J., Rastogi, R. (eds) Data Stream Management. Data-Centric Systems and Applications. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-28608-0_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-540-28608-0_14

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-28607-3

  • Online ISBN: 978-3-540-28608-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation