Predictive Complexity and Information

  • Michael V. Vyugin
  • Vladimir V. V’yugin
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2375)


A new notion of predictive complexity and corresponding amount of information are considered. Predictive complexity is a generalization of Kolmogorov complexity which bounds the ability of any algorithm to predict elements of a sequence of outcomes. We consider predictive complexity for a wide class of bounded loss functions which are generalizations of square-loss function. Relations between unconditional KG(x) and conditional KG(xy) predictive complexities are studied. We define an algorithm which has some “expanding property”. It transforms with positive probability sequences of given predictive complexity into sequences of essentially bigger predictive complexity. A concept of amount of predictive information IG(y: x) is studied. We show that this information is non-commutative in a very strong sense and present asymptotic relations between values IG(y: x), IG(x: y), KG(x) and KG(y).


Loss Function Binary Tree Computable Mapping Expert Advice Prediction Strategy 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Haussler, D., Kivinen, J., Warmuth, M. K. (1994) Tight worst-case loss bounds for predicting with expert advice. Technical Report UCSC-CRL-94-36, University of California at Santa Cruz, revised December 1994. Short version in P. Vitányi, editor, Computational Learning Theory, Lecture Notes in Computer Science, volume 904, pages 69–83, Springer, Berlin, 1995.Google Scholar
  2. 2.
    Cesa-Bianchi, N., Freund, Y., Helmbold, D. P., Haussler, D., Schapire, R. E., Warmuth, M. K. (1997) How to use expert advice. Journal of the ACM, 44, 427–485zbMATHCrossRefMathSciNetGoogle Scholar
  3. 3.
    Cormen, H., Leiserson, E., Rivest, R (1990) Introduction to Algorithms. New York: McGraw Hill.Google Scholar
  4. 4.
    Kalnishkan, Y. (1999) General linear relations among different types of predictive complexity. In. Proc. 10th international Conference on Algorithmic Learning Theory-ALT’ 99, v. 1720 of Lecture Notes in Artificial Intelligence, pp. 323–334, Springer-Verlag.Google Scholar
  5. 5.
    Li, M., Vitányi, P. (1997) An Introduction to Kolmogorov Complexity and Its Applications. Springer, New York, 2nd edition.zbMATHGoogle Scholar
  6. 6.
    Rogers, H. (1967) Theory of recursive functions and effective computability, New York: McGraw Hill.zbMATHGoogle Scholar
  7. 7.
    Vovk, V. (1990) Aggregating strategies. In M. Fulk and J. Case, editors, Proceedings of the 3rd Annual Workshop on Computational Learning Theory, pages 371–383, San Mateo, CA, 1990. Morgan Kaufmann.Google Scholar
  8. 8.
    Vovk, V. (1998) A game of prediction with expert advice. J. Comput. Syst. Sci., 56:153–173.zbMATHCrossRefMathSciNetGoogle Scholar
  9. 9.
    Vovk, V., Gammerman, A. (1999) Complexity estimation principle, The Computer Journal, 42:4, 318–322.zbMATHCrossRefGoogle Scholar
  10. 10.
    Vovk, V., Watkins, C. J. H. C. (1998) Universal portfolio selection, Proceedings of the 11th Annual Conference on Computational Learning Theory, 12–23.Google Scholar
  11. 11.
    V’yugin, V. V. (1999) Does snooping help? Technical report No CLRC-TR-99-06, December 1999, Computer Learning Research Centre, Royal Holloway, University of London.Google Scholar
  12. 12.
    Vyugin, M. V., V’yugin, V. V. (2001) Non-linear inequalities between Kolmogorov and predictive complexities, In Proc. Twelfth International Conference on Algorithmic Learning Theory—ALT’ 01, 190–204.Google Scholar
  13. 13.
    Zvonkin, A. K., Levin, L. A. (1970) The complexity of finite objects and the algorithmic concepts of information and randomness, Russ. Math. Surv. 25, 83–124.zbMATHCrossRefMathSciNetGoogle Scholar
  14. 14.
    Yamanishi, K. (1995) Randomized approximate aggregating strategies and their applications to prediction and discrimination, in Proceedings, 8th Annual ACM Conference on Computational Learning Theory, 83–90, Assoc. Comput. Mach., New York.CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2002

Authors and Affiliations

  • Michael V. Vyugin
    • 1
  • Vladimir V. V’yugin
    • 2
    • 3
  1. 1.Department of Computer Science, Royal HollowayUniversity of LondonEghamEngland
  2. 2.Institute for Information Transmission ProblemsRussian Academy of SciencesMoscow GSP-4Russia
  3. 3.Computer Learning Research Centre, Royal HollowayUniversity of LondonEghamEngland

Personalised recommendations