Learning-Theoretic Methods in Vector Quantization

  • T. Linder
Part of the International Centre for Mechanical Sciences book series (CISM, volume 434)


The principal goal of data compression (also known as source coding) is to replace data by a compact representation in such a manner that from this representation the original data can be reconstructed either perfectly, or with high enough accuracy. Generally, the representation is given in the form of a sequence of binary digits (bits) that can be used for efficient digital transmission or storage.


Vector Quantization Lagrangian Formulation Binary String Source Distribution Minimum Distortion 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. E. A. Abaya and G. L. Wise. On the existence of optimal quantizers. IEEE Trans. Inform. Theory, 28:937 — 940, Nov. 1982.Google Scholar
  2. E. A. Abaya and G. L. Wise. Convergence of vector quantizers with applications to optimal quantization. SIAM Journal on Applied Mathematics, 44: 183–189, 1984.MathSciNetCrossRefMATHGoogle Scholar
  3. M. Anthony and P. L. Bartlett. Neural Network Learning: Theoretical Foundations. Cambridge University Press, 1999.Google Scholar
  4. R. B. Ash. Probability and Measure Theory. Academic Press, New York, 2000.MATHGoogle Scholar
  5. P. Bartlett, T. Linder, and G. Lugosi. The minimax distortion redundancy in empirical quantizer design. IEEE Trans. Inform. Theory, IT-44(5): 1802–1813, Sep. 1998.Google Scholar
  6. T. Berger. Optimum quantizers and permutation codes. IEEE Trans. Inform. Theory, IT-18: 759–765, Nov. 1972.Google Scholar
  7. P. A. Chou. The distortion of vector quantizers trained on n vectors decreases to the optimum as Op (1/n). in Proc. IEEE Int. Symp. Information Theory (Trondheim, Norway, Jun. 27-Jul. 1, 1994 ), p. 457.Google Scholar
  8. P. A. Chou and B. J. Betts. When optimal entropy-constrained quantizers have only a finite number of codewords. in Proc. IEEE Int. Symp. Information Theory (Cambridge, MA, USA, Aug. 16–21, 1998 ), p. 97.Google Scholar
  9. P. A. Chou, T. Lookabaugh, and R. M. Gray. Entropy-constrained vector quantization. IEEE Trans. Acoust. Speech, Signal Processing, ASSP-37: 31–42, Jan. 1989.Google Scholar
  10. D. Cohn, E. Riskin, and R. Ladner. Theory and practice of vector quantizers trained on small training sets. IEEE Trans. on Pattern Analysis and Machine Intelligence, 16: 54–65, Jan. 1994.CrossRefGoogle Scholar
  11. P. C. Cosman, K. O. Perlmutter, S. M. Perlmutter, R. A. Olshen, and R. M. Gray. Training sequence size and vector quantizer performance. In Proceedings of Asilomar Conference on Signals, Systems, and Computers, pages 434–438, 1991.Google Scholar
  12. T. Cover and J. A. Thomas. Elements of Information Theory. Wiley, New York, 1991.CrossRefMATHGoogle Scholar
  13. L. Devroye, L. Györfi, and G. Lugosi. A Probabilistic Theory of Pattern Recognition. Springer, New York, 1996.MATHGoogle Scholar
  14. R. M. Dudley. Real Analysis and Probability. Chapman & Hall, New York, 1989.MATHGoogle Scholar
  15. R.M. Dudley. Central limit theorems for empirical measures. Annals of Probability, 6: 899–929, 1978.MathSciNetCrossRefMATHGoogle Scholar
  16. A. Gersho and R. M. Gray. Vector Quantization and Signal Compression. Kluwer, Boston, 1992.CrossRefMATHGoogle Scholar
  17. H. Gish and J. N. Pierce. Asymptotically efficient quantizing. IEEE Trans. Inform. Theory, IT-14: 676–683, Sep. 1968.Google Scholar
  18. S. Graf and H. Luschgy. Consistent estimation in the quantization problem for random vectors. In Transactions of the Twelfth Prague Conference on Information Theory, Statistical Decision Functions, Random Processes, pages 84–87, 1994.Google Scholar
  19. S. Graf and H. Luschgy. Rates of convergence for the empirical quantization error. preprint, 1999.Google Scholar
  20. S. Graf and H. Luschgy. Foundations of Quantization for Probability Distributions. Springer Verlag, Berlin, Heidelberg, 2000.Google Scholar
  21. R. M. Gray. Source Coding Theory. Kluwer, Boston, 1990.MATHGoogle Scholar
  22. R. M. Gray and L. D. Davisson. Quantizer mismatch. IEEE Trans. Communications, 23: 439–443, 1975.MathSciNetCrossRefMATHGoogle Scholar
  23. R. M. Gray, T. Linder, and J. Li. A Lagrangian formulation of Zador’s entropy-constrained quantization theorem. IEEE Trans. Inform. Theory,2001 (to appear).Google Scholar
  24. R. M. Gray and D. L. Neuhoff. Quantization. IEEE Trans. Inform. Theory, (Special Commemorative Issue), IT-44(6): 2325–2383, Oct. 1998.Google Scholar
  25. R. M. Gray, D. L. Neuhoff, and P. C. Shields. A generalization of Orstein’s J-distance with applications to information theory. Annals of Probability, 3: 315–328, 1975.MathSciNetCrossRefMATHGoogle Scholar
  26. A. György and T. Linder. Optimal entropy-constrained scalar quantization of a uniform source. IEEE Trans. Inform. Theory, IT-46:pp. 2704–2711, Nov. 2000.Google Scholar
  27. A. György and T. Linder. On the structure of optimal entropy-constrained scalar quantizers. IEEE Trans. Inform. Theory,2001 (to appear).Google Scholar
  28. A. György and T. Linder. On optimal Lagrangian entropy-constrained vector quantization. preprint, 2001.Google Scholar
  29. Y. Linde, A. Buzo, and R. M. Gray. An algorithm for vector quantizer design. IEEE Transactions on Communications, COM-28: 84–95, Jan. 1980.Google Scholar
  30. T. Linder. On the training distortion of vector quantizers. IEEE Trans. Inform. Theory, IT-46: 1617–1623, Jul. 2000.Google Scholar
  31. T. Linder, G. Lugosi, and K. Zeger. Rates of convergence in the source coding theorem, in empirical quantizer design, and in universal lossy source coding. IEEE Trans. Inform. Theory, 40: 1728–1740, Nov. 1994.MathSciNetCrossRefMATHGoogle Scholar
  32. T. Linder, G. Lugosi, and K. Zeger. Empirical quantizer design in the presence of source noise or channel noise. IEEE Trans. Inform. Theory, IT-43: 612–623, Mar. 1997.Google Scholar
  33. T. Linder, V. Tarokh, and K. Zeger. Existence of optimal prefix codes for infinite source alphabets. IEEE Trans. Inform. Theory, IT-43: 2026–2028, Nov. 1997.Google Scholar
  34. S. P. Lloyd. Least squared quantization in PCM. unpublished memorandum, Bell Lab., 1957; Also, IEEE Trans. Inform. Theory, vol. IT-28, no. 2, pp. 129–137., Mar. 1982.Google Scholar
  35. J. MacQueen. Some methods for classification and and analysis of multivariate observations. in Proc. 5th Berkeley Symp. on Mathematical Statistics and Probability, vol. 1, pp. 281–296, 1967.Google Scholar
  36. R Zador. Asymptotic quantization error of continuous signals and the quantization dimension. IEEE Trans. Inform. Theory, IT-28: 139–149, Mar. 1982.Google Scholar
  37. N. Merhav and J. Ziv. On the amount of side information required for lossy data compression. IEEE Trans. Inform. Theory, IT-43: 1112–1121, July 1997.Google Scholar
  38. D. Pollard. Strong consistency of k-means clustering. Annals of Statistics, 9, no. 1: 135140, 1981.Google Scholar
  39. D. Pollard. Quantization and the method of k-means. IEEE Trans. Inform. Theory, IT-28: 199–205, Mar. 1982.Google Scholar
  40. D. Pollard. A central limit theorem for k-means clustering. Annals of Probability, vol. 10, no. 4: 919–926, 1982.MathSciNetCrossRefMATHGoogle Scholar
  41. D. Pollard. Empirical Processes: Theory and Applications. NSF-CBMS Regional Conference Series in Probability and Statistics, Institute of Mathematical Statistics, Hayward, CA, 1990.Google Scholar
  42. R. T. Rockafellar. Convex Analysis. Princeton University Press, Princeton, NJ, 1970.MATHGoogle Scholar
  43. M. J. Sabin. Global convergence and empirical consistency of the generalized Lloyd algorithm. PhD thesis, Stanford Univ., 1984.Google Scholar
  44. M. J. Sabin and R. M. Gray. Global convergence and empirical consistency of the generalized Lloyd algorithm. IEEE Trans. Inform. Theory, IT-32: 148–155, Mar. 1986.Google Scholar
  45. K. Sayood. Introduction to Data Compression. Morgan Kaufmann Publishers, San Francisco, 2nd edition, 2001.Google Scholar
  46. E.V. Slud. Distribution inequalities for the binomial law. Annals of Probability, 5: 404412, 1977.Google Scholar
  47. H. Steinhaus. Sur la division des corp materiels en parties. Bull. Acad. Polon. Sci, IV: 80 1804, May 1956.Google Scholar
  48. S. Szarek. On the best constants in the Khintchine inequality. Studia Mathematica, 63: 197–208, 1976.MathSciNetGoogle Scholar
  49. V. N. Vapnik. Statistical Learning Theory. Wiley, New York, 1998.MATHGoogle Scholar
  50. P. Zador. Topics in the asymptotic quantization of continuous random variables. unpublished memorandum, Bell Laboratories, Murray Hill, NJ, Feb. 1966.Google Scholar
  51. A. J. Zeevi. On the performance of vector quantizers empirically designed from dependent sources. in Proceedings of Data Compression Conference, DCC’98, (J. Storer, M. Cohn, ed.) pp. 73–82, IEEE Computer Society Press, Los Alamitos, California, 1998.Google Scholar

Copyright information

© Springer-Verlag Wien 2002

Authors and Affiliations

  • T. Linder
    • 1
  1. 1.Queens’ University at KingstonKingstonCanada

Personalised recommendations