Skip to main content
SpringerLink
Log in

Maximizing Fun by Creating Data with Easily Reducible Subjective Complexity

  • Chapter
  • First Online:
Intrinsically Motivated Learning in Natural and Artificial Systems

Abstract

The Formal Theory of Fun and Creativity (1990–2010) [Schmidhuber, J.: Formal theory of creativity, fun, and intrinsic motivation (1990–2010). IEEE Trans. Auton. Mental Dev. 2(3), 230–247 (2010b)] describes principles of a curious and creative agent that never stops generating nontrivial & novel & surprising tasks and data. Two modules are needed: a data encoder and a data creator. The former encodes the growing history of sensory data as the agent is interacting with its environment; the latter executes actions shaping the history. Both learn. The encoder continually tries to encode the created data more efficiently, by discovering new regularities in it. Its learning progress is the wow-effect or fun or intrinsic reward of the creator, which maximizes future expected reward, being motivated to invent skills leading to interesting data that the encoder does not yet know but can easily learn with little computational effort. I have argued that this simple formal principle explains science and art and music and humor.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bakker, B., Schmidhuber, J.: Hierarchical reinforcement learning based on subgoal discovery and subpolicy specialization. In: Groen, F., Amato, N., Bonarini, A., Yoshida, E., and Kröse, B. (eds.) Proceedings of the 8th Conference on Intelligent Autonomous Systems IAS-8, pp. 438–445. IOS Press, Amsterdam (2004)

    Google Scholar 

  2. Balcan, M.,.Beygelzimer, A., Langford, J.: Agnostic active learning. J. Comput. Syst. Sci. 75(1), 78–89 (2009)

    MathSciNet  MATH  Google Scholar 

  3. Barto, A.G., Singh, S., Chentanez, N.: Intrinsically motivated learning of hierarchical collections of skills. In: Proceedings of International Conference on Developmental Learning (ICDL). MIT, Cambridge (2004)

    Google Scholar 

  4. Bense, M.: Einführung in die informationstheoretische Ästhetik. Grundlegung und Anwendung in der Texttheorie (Introduction to Information-Theoretical Aesthetics. Foundation and Application to Text Theory). Rowohlt Taschenbuch Verlag (1969)

    Google Scholar 

  5. Berlyne, D.E.: Novelty and curiosity as determinants of exploratory behavior. Br. J. Psychol. 41, 68–80 (1950)

    Google Scholar 

  6. Berlyne, D.E.: Conflict, Arousal, and Curiosity. McGraw Hill, New York (1960)

    Google Scholar 

  7. Birkhoff, G.D.: Aesthetic Measure. Harvard University Press, Cambridge (1933)

    MATH  Google Scholar 

  8. Blank, D., Meeden, L.: Introduction to the special issue on developmental robotics. Connect. Sci. 18(2) (2006)

    Google Scholar 

  9. Brafman, R.I., Tennenholtz, M.: R-MAX—a general polynomial time algorithm for near-optimal reinforcement learning. J. Mach. Learn. Res. 3, 213–231 (2002)

    MathSciNet  MATH  Google Scholar 

  10. Cohn, D.A.: Neural network exploration using optimal experiment design. In: Cowan, J., Tesauro, G., Alspector, J. (eds.) Advances in Neural Information Processing Systems 6, pp. 679–686. Morgan Kaufmann, San Francisco (1994)

    Google Scholar 

  11. Collingwood, R.G.: The Principles of Art. Oxford University Press, London (1938)

    Google Scholar 

  12. Cuccu, G., Luciw, M., Schmidhuber, J., Gomez, F.: Intrinsically motivated evolutionary search for vision-based reinforcement learning. In: Proceedings of the 2011 IEEE Conference on Development and Learning and Epigenetic Robotics IEEE-ICDL-EPIROB. IEEE (2011)

    Google Scholar 

  13. Dayan, P., Hinton, G.: Feudal reinforcement learning. In: Lippman, D.S., Moody, J.E., Touretzky, D.S. (eds.) Advances in Neural Information Processing Systems 5, pp. 271–278. Morgan Kaufmann, San Francisco (1993)

    Google Scholar 

  14. Fedorov, V.V.: Theory of Optimal Experiments. Academic, New York (1972)

    Google Scholar 

  15. Frank, H.G.: Kybernetische Analysen subjektiver Sachverhalte. Verlag Schnelle, Quickborn (1964)

    Google Scholar 

  16. Frank, H.G., Franke, H.W.: Ästhetische Information. Estetika informacio. Eine Einführung in die kybernetische Ästhetik. Kopäd Verlag (2002)

    Google Scholar 

  17. Franke, H.W.: Kybernetische Ästhetik. Phänomen Kunst, 3rd edn. Ernst Reinhardt Verlag, Munich (1979)

    Google Scholar 

  18. Friston, K.J., Daunizeau, J., Kilner, J., Kiebel, S.J.: Action and behavior: A free-energy formulation. Biol. Cybern. 102(3), 227–260 (2010)

    Google Scholar 

  19. Gold, K., Scassellati, B.: Learning acceptable windows of contingency. Connect. Sci. 18(2) (2006)

    Google Scholar 

  20. Gomez, F.J., Schmidhuber, J., Miikkulainen, R.: Efficient non-linear control through neuroevolution. J. Mach. Learn. Res. 9, 937–965 (2008)

    MathSciNet  MATH  Google Scholar 

  21. Goodman, N.: Languages of Art: An Approach to a Theory of Symbols. The Bobbs-Merrill Company, Indianapolis (1968)

    Google Scholar 

  22. Graves, A., Liwicki, M., Fernandez, S., Bertolami, R., Bunke, H., Schmidhuber, J.: A novel connectionist system for improved unconstrained handwriting recognition. IEEE Trans. Pattern Anal. Mach. Intell. 31(5) (2009)

    Google Scholar 

  23. Harlow, H.F., Harlow, M.K., Meyer, D.R.: Novelty and curiosity as determinants of exploratory behavior. J. Exp. Psychol. 41, 68–80 (1950)

    Google Scholar 

  24. Hart, S.: The Development of Hierarchical Knowledge in Robot Systems. Ph.D. Thesis, Department of Computer Science, University of Massachusetts Amherst (2009)

    Google Scholar 

  25. Hart, S., Sen, S., Grupen, R.: Intrinsically motivated hierarchical manipulation. In: Proceedings of the 2008 IEEE Conference on Robots and Automation (ICRA), Pasadena (2008)

    Google Scholar 

  26. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Google Scholar 

  27. Holland, J.H.: Adaptation in Natural and Artificial Systems. University of Michigan Press, Ann Arbor (1975)

    Google Scholar 

  28. Holland, J.H.: Properties of the bucket brigade. In: Proceedings of an International Conference on Genetic Algorithms. Lawrence Erlbaum, Hillsdale (1985)

    Google Scholar 

  29. Huffman, D.A.: A method for construction of minimum-redundancy codes. Proc. IRE 40, 1098–1101 (1952)

    Google Scholar 

  30. Hutter, M.: Universal Artificial Intelligence: Sequential Decisions based on Algorithmic Probability. Springer, Berlin (2005). (On J. Schmidhuber’s SNF grant 20-61847)

    MATH  Google Scholar 

  31. Hwang, J., Choi, J., Oh, S., II, R.J.M.: Query-based learning applied to partially trained multilayer perceptrons. IEEE Trans. Neural Netw. 2(1), 131–136 (1991)

    Google Scholar 

  32. Itti, L., Baldi, P.F.: Bayesian surprise attracts human attention. In: Advances in Neural Information Processing Systems 19, pp. 547–554. MIT, Cambridge (2005)

    Google Scholar 

  33. Kaelbling, L.P., Littman, M.L., Moore, A.W.: Reinforcement learning: A survey. J. AI Res. 4, 237–285 (1996)

    Google Scholar 

  34. Kant, I.: Critik der reinen Vernunft (1781)

    Google Scholar 

  35. Klyubin, A.S., Polani, D., Nehaniv, C.L.: Empowerment: A universal agent-centric measure of control. In: Congress on Evolutionary Ccomputation (CEC-05), IEEE (2005)

    Google Scholar 

  36. Kolmogorov, A.N.: Three approaches to the quantitative definition of information. Prob. Inform. Transm. 1, 1–11 (1965)

    Google Scholar 

  37. Kuhn, T.: The Structure of Scientific Revolutions. University of Chicago Press, Chicago (1962)

    Google Scholar 

  38. Kuipers, B., Beeson, P., Modayil, J., Provost, J.: Bootstrap learning of foundational representations. Connect. Sci. 18(2) (2006)

    Google Scholar 

  39. Kullback, S.: Statistics and Information Theory. Wiley, New York (1959)

    MATH  Google Scholar 

  40. Lenat, D.B.: Theory formation by heuristic search. Mach. Learn. 21 (1983)

    Google Scholar 

  41. Levin, L.A.: Universal sequential search problems. Prob. Inform. Transm. 9(3), 265–266 (1973)

    MathSciNet  Google Scholar 

  42. Li, L., Littman, M.L., Walsh, T.J.: Knows what it knows: A framework for self-aware learning. In: Proceedings of the Twenty-Fifth International Conference on Machine Learning (ICML-08) (2008)

    Google Scholar 

  43. Li, M., Vitányi, P.M.B.: An Introduction to Kolmogorov Complexity and Its Applications, 2nd edn. Springer, Berlin (1997)

    MATH  Google Scholar 

  44. Littman, M.L., Ackley, D.H.: Adaptation in constant utility non-stationary environments. In: Belew, R.K., Booker, L. (eds.) Proceedings of the Fourth International Conference on Genetic Algorithms, pp. 136–142. Morgan Kaufmann, San Mateo (1991)

    Google Scholar 

  45. Luciw, M., Graziano, V., Ring, M., Schmidhuber, J.: Artificial curiosity with planning for autonomous perceptual and cognitive development. In: Proceedings of the First Joint Conference on Development Learning and on Epigenetic Robotics ICDL-EPIROB, Frankfurt (2011)

    Google Scholar 

  46. MacKay, D.J.C.: Information-based objective functions for active data selection. Neural Comput. 4(2), 550–604 (1992)

    Google Scholar 

  47. Moles, A.: Information Theory and Esthetic Perception. University of Illinois Press, Illinois (1968)

    Google Scholar 

  48. Nake, F.: Ästhetik als Informationsverarbeitung. Springer, Berlin (1974)

    Google Scholar 

  49. Ngo, H., Ring, M., Schmidhuber, J.: Compression progress-based curiosity drive for developmental learning. In: Proceedings of the 2011 IEEE Conference on Development and Learning and Epigenetic Robotics IEEE-ICDL-EPIROB. IEEE (2011)

    Google Scholar 

  50. Olsson, L., Nehaniv, C.L., Polani, D.: From unknown sensors and actuators to actions grounded in sensorimotor perceptions. Connect. Sci. 18(2) (2006)

    Google Scholar 

  51. Oudeyer, P.-Y., Kaplan, F.: What is intrinsic motivation? a typology of computational approaches. Front. Neurorobot. 1 (2006)

    Google Scholar 

  52. Oudeyer, P.-Y., Kaplan, F., Hafner, V.F.: Intrinsic motivation systems for autonomous mental development. IEEE Trans. Evol. Comput. 11(2), 265–286 (2007)

    Google Scholar 

  53. Piaget, J.: The Child’s Construction of Reality. Routledge and Kegan Paul, London (1955)

    Google Scholar 

  54. Plutowski, M., Cottrell, G., White, H.: Learning Mackey-Glass from 25 examples, plus or minus 2. In: Cowan, J., Tesauro, G., Alspector, J. (eds.) Advances in Neural Information Processing Systems, vol. 6, pp. 1135–1142. Morgan Kaufmann, San Francisco (1994)

    Google Scholar 

  55. Poland, J., Hutter, M.: Strong asymptotic assertions for discrete MDL in regression and classification. In: Annual Machine Learning Conference of Belgium and the Netherlands (Benelearn-2005), Enschede (2005)

    Google Scholar 

  56. Popper, K.R.: All Life Is Problem Solving. Routledge, London (1999)

    Google Scholar 

  57. Provost, J., Kuipers, B.J., Miikkulainen, R.: Developing navigation behavior through self-organizing distinctive state abstraction. Connect. Sci. 18(2) (2006)

    Google Scholar 

  58. Raskin, V.: Semantic Mechanisms of Humor. Dordrecht/Boston/Lancaster (1985)

    Google Scholar 

  59. Rechenberg, I.: Evolutionsstrategie - Optimierung technischer Systeme nach Prinzipien der biologischen Evolution. Dissertation. Published 1973 by Fromman-Holzboog (1971)

    Google Scholar 

  60. Ring, M.B.: Incremental development of complex behaviors through automatic construction of sensory-motor hierarchies. In: Birnbaum, L., Collins, G. (eds.) Machine Learning: Proceedings of the Eighth International Workshop, pp. 343–347. Morgan Kaufmann, San Francisco (1991)

    Google Scholar 

  61. Ring, M.B.: Continual Learning in Reinforcement Environments. Ph.D. Thesis, University of Texas at Austin, Austin (1994)

    Google Scholar 

  62. Rissanen, J.: Modeling by shortest data description. Automatica 14, 465–471 (1978)

    MATH  Google Scholar 

  63. Rückstieß, T., Sehnke, F., Schaul, T., Wierstra, D., Yi, S., Schmidhuber, J.: Exploring parameter space in reinforcement learning. Paladyn J. Behav. Robot. 1(1), 14–24 (2010)

    Google Scholar 

  64. Schaul, T., Pape, L., Glasmachers, T., Graziano, V., Schmidhuber, J.: Coherence Progress: A Measure of Interestingness Based on Fixed Compressors. In: Fourth Conference on Artificial General Intelligence (AGI) (2011a)

    Google Scholar 

  65. Schaul, T., Sun, Y., Wierstra, D., Gomez, F., Schmidhuber, J.: Curiosity-Driven Optimization. In: IEEE Congress on Evolutionary Computation (CEC), New Orleans (2011b)

    Google Scholar 

  66. Schembri, M., Mirolli, M., Baldassarre, G.: Evolving internal reinforcers for an intrinsically motivated reinforcement-learning robot. In: Demiris, Y., Scassellati, B., Mareschal, D. (eds.) The 6th IEEE International Conference on Development and Learning (ICDL2007), pp. 282–287. Imperial College, London. IEEE Catalog Number: 07EX1740C, Library of Congress: 2007922394 (2007)

    Google Scholar 

  67. Schlesinger, M.: Decomposing infants’ object representations: A dual-route processing account. Connect. Sci. 18(2) (2006)

    Google Scholar 

  68. Schmidhuber, J.: A local learning algorithm for dynamic feedforward and recurrent networks. Connect. Sci. 1(4), 403–412 (1989a)

    Google Scholar 

  69. Schmidhuber, J.: The neural bucket brigade. In: Pfeifer, R., Schreter, Z., Fogelman, Z., Steels, L. (eds.) Connectionism in Perspective, pp. 439–446. Elsevier, North-Holland, Amsterdam (1989b)

    Google Scholar 

  70. Schmidhuber, J.: Dynamische neuronale Netze und das fundamentale raumzeitliche Lernproblem. Dissertation, Institut für Informatik, Technische Universität München (1990a)

    Google Scholar 

  71. Schmidhuber, J.: An on-line algorithm for dynamic reinforcement learning and planning in reactive environments. In: Proceedings of the IEEE/INNS International Joint Conference on Neural Networks, San Diego, vol. 2, pp. 253–258 (1990b)

    Google Scholar 

  72. Schmidhuber, J.: Adaptive curiosity and adaptive confidence. Technical Report FKI-149-91, Institut für Informatik, Technische Universität München. See also Schmidhuber (1991b) (1991a)

    Google Scholar 

  73. Schmidhuber, J.: Curious model-building control systems. In: Proceedings of the International Joint Conference on Neural Networks, Singapore, vol. 2, pp. 1458–1463. IEEE (1991b)

    Google Scholar 

  74. Schmidhuber, J.: A possibility for implementing curiosity and boredom in model-building neural controllers. In: Meyer, J.A., Wilson, S.W. (eds.) Proceedings of the of the International Conference on Simulation of Adaptive Behavior: From Animals to Animats, pp. 222–227. MIT/Bradford Books (1991c)

    Google Scholar 

  75. Schmidhuber, J.: Reinforcement learning in Markovian and non-Markovian environments. In: Lippman, D.S., Moody, J.E., Touretzky, D.S. (eds.) Advances in Neural Information Processing Systems 3 (NIPS 3), pp. 500–506. Morgan Kaufmann, San Francisco (1991d)

    Google Scholar 

  76. Schmidhuber, J.: Learning complex, extended sequences using the principle of history compression. Neural Comput. 4(2), 234–242 (1992)

    Google Scholar 

  77. Schmidhuber, J.: Femmes fractales. Report IDSIA-99-97, IDSIA, Switzerland, (1997)

    Google Scholar 

  78. Schmidhuber, J.: Low-complexity art. Leonardo, J. Int. Soc. Arts Sci. Technol. 30(2), 97–103 (1997b)

    Google Scholar 

  79. Schmidhuber, J.: What’s interesting? Technical Report IDSIA-35-97, IDSIA (1997c). ftp://ftp.idsia.ch/pub/juergen/interest.ps.gz; extended abstract in Proc. Snowbird’98, Utah, 1998; see also Schmidhuber (2002a)

  80. Schmidhuber, J.: Exploring the predictable. In: Ghosh, A., Tsuitsui, S. (eds.) Advances in Evolutionary Computing, pp. 579–612. Springer, Berlin (2002a)

    Google Scholar 

  81. Schmidhuber, J.: Hierarchies of generalized Kolmogorov complexities and nonenumerable universal measures computable in the limit. Int. J. Found. Comput. Sci. 13(4), 587–612 (2002b)

    MathSciNet  MATH  Google Scholar 

  82. Schmidhuber, J.: The Speed Prior: a new simplicity measure yielding near-optimal computable predictions. In: Kivinen, J., Sloan, R.H. (eds.) Proceedings of the 15th Annual Conference on Computational Learning Theory (COLT 2002), Lecture Notes in Artificial Intelligence, pp. 216–228. Springer, Sydney (2002c)

    Google Scholar 

  83. Schmidhuber, J.: Optimal ordered problem solver. Mach. Learn. 54, 211–254 (2004)

    MATH  Google Scholar 

  84. Schmidhuber, J.: Completely self-referential optimal reinforcement learners. In: Duch, W., Kacprzyk, J., Oja, E., Zadrozny, S. (eds.) Artificial Neural Networks: Biological Inspirations - ICANN 2005, LNCS 3697, pp. 223–233. Springer (2005a). Plenary talk

    Google Scholar 

  85. Schmidhuber, J.: Gödel machines: Towards a technical justification of consciousness. In: Kudenko, D., Kazakov, D., Alonso, E. (eds.) Adaptive Agents and Multi-Agent Systems III (LNCS 3394), pp. 1–23. Springer, Berlin (2005b)

    Google Scholar 

  86. Schmidhuber, J.: Developmental robotics, optimal artificial curiosity, creativity, music, and the fine arts. Connect. Sci. 18(2), 173–187 (2006a)

    Google Scholar 

  87. Schmidhuber, J.: Gödel machines: Fully self-referential optimal universal self-improvers. In: Goertzel, B., Pennachin, C. (eds.) Artificial General Intelligence, pp. 199–226. Springer, Berlin (2006b). Variant available as arXiv:cs.LO/0309048

    Google Scholar 

  88. Schmidhuber, J.: The new AI: General & sound & relevant for physics. In: Goertzel, B., Pennachin, C. (eds.) Artificial General Intelligence, pp. 175–198. Springer, Berlin (2006c). Also available as TR IDSIA-04-03, arXiv:cs.AI/0302012

    Google Scholar 

  89. Schmidhuber, J.: Randomness in physics. Nature 439(3), 392 (2006d). Correspondence

    Google Scholar 

  90. Schmidhuber, J.: 2006: Celebrating 75 years of AI - history and outlook: The next 25 years. In: Lungarella, M., Iida, F., Bongard, J., Pfeifer, R. (eds.) 50 Years of Artificial Intelligence, LNAI, vol. 4850, pp. 29–41. Springer, Berlin (2007a). Preprint available as arXiv:0708.4311

    Google Scholar 

  91. Schmidhuber, J.: New millennium AI and the convergence of history. In: Duch, W., Mandziuk, J. (eds.) Challenges to Computational Intelligence, vol. 63, pp. 15–36. Studies in Computational Intelligence, Springer, Berlin (2007b). Also available as arXiv:cs.AI/0606081

    Google Scholar 

  92. Schmidhuber, J.: Simple algorithmic principles of discovery, subjective beauty, selective attention, curiosity & creativity. In: Proceedings of the 10th International Conf. on Discovery Science (DS 2007), LNAI, vol. 4755, pp. 26–38. Springer, Berlin. Joint invited lecture for ALT 2007 and DS 2007, Sendai, Japan, 2007 (2007c)

    Google Scholar 

  93. Schmidhuber, J.: Driven by compression progress. In: Lovrek, I., Howlett, R.J., Jain, L.C. (eds.) Knowledge-Based Intelligent Information and Engineering Systems KES-2008, Lecture Notes in Computer Science, vol. 5177, Part I, p. 11. Springer, Berlin. Abstract of invited keynote (2008)

    Google Scholar 

  94. Schmidhuber, J.: Art & science as by-products of the search for novel patterns, or data compressible in unknown yet learnable ways. In: Botta, M. (ed.) Multiple ways to design research. Research cases that reshape the design discipline, Swiss Design Network - Et al. Edizioni, pp. 98–112. Springer, Berelin (2009a)

    Google Scholar 

  95. Schmidhuber, J.: Compression progress: The algorithmic principle behind curiosity and creativity (with applications of the theory of humor). 40 min video of invited talk at Singularity Summit 2009, New York City (2009b). http://www.vimeo.com/7441291. 10 min excerpts: http://www.youtube.com/watch?v=Ipomu0MLFaI .

  96. Schmidhuber, J.: Driven by compression progress: A simple principle explains essential aspects of subjective beauty, novelty, surprise, interestingness, attention, curiosity, creativity, art, science, music, jokes. In: Pezzulo, G., Butz, M.V., Sigaud, O., Baldassarre, G. (eds.) Anticipatory Behavior in Adaptive Learning Systems. From Psychological Theories to Artificial Cognitive Systems, LNCS, vol. 5499, pp. 48–76. Springer, Berlin (2009c)

    Google Scholar 

  97. Schmidhuber, J.: Simple algorithmic theory of subjective beauty, novelty, surprise, interestingness, attention, curiosity, creativity, art, science, music, jokes. J. Soc. Instrum. Control Eng. 48(1), 21–32 (2009d)

    Google Scholar 

  98. Schmidhuber, J.: Ultimate cognition à la Gödel. Cogn. Comput. 1(2), 177–193 (2009e)

    Google Scholar 

  99. Schmidhuber, J.: Artificial scientists & artists based on the formal theory of creativity. In: Eric, B., Marcus, H., Emanuel, K. (eds.) Artificial General Intelligence. Springer, Berlin (2010a)

    Google Scholar 

  100. Schmidhuber, J.: Formal theory of creativity, fun, and intrinsic motivation (1990–2010): IEEE Trans. Auton. Mental Dev. 2(3), 230–247 (2010b)

    Google Scholar 

  101. Schmidhuber, J.: A formal theory of creativity to model the creation of art. In: McCormack, J., d’Inverno, M. (eds.) Computational Creativity. MIT, Cambridge (2012)

    Google Scholar 

  102. Schmidhuber, J., Gomez, F., Graves, A.: Sequence Learning with Artificial Recurrent Neural Networks. Cambridge University Press, Cambridge (2012) (in preparation)

    Google Scholar 

  103. Schmidhuber, J., Heil, S.: Sequential neural text compression. IEEE Trans. Neural Netw. 7(1), 142–146 (1996)

    Google Scholar 

  104. Schmidhuber, J., Wahnsiedler, R.: Planning simple trajectories using neural subgoal generators. In: Meyer, J.A., Roitblat, H.L., Wilson, S.W. (eds.) Proc. of the 2nd International Conference on Simulation of Adaptive Behavior, pp. 196–202. MIT, Cambridge (1992)

    Google Scholar 

  105. Schmidhuber, J., Zhao, J., Wiering, M.: Shifting inductive bias with success-story algorithm, adaptive Levin search, and incremental self-improvement. Mach. Learn. 28, 105–130 (1997)

    Google Scholar 

  106. Schwefel, H.P.: Numerische Optimierung von Computer-Modellen. Dissertation. Published 1977 by Birkhäuser, Basel (1974)

    Google Scholar 

  107. Sehnke, F., Osendorfer, C., Rückstieß, T., Graves, A., Peters, J., Schmidhuber, J.: Parameter-exploring policy gradients. Neural Netw. 23(4), 551–559 (2010)

    Google Scholar 

  108. Seung, H.S., Opper, M., Sompolinsky, H.: Query by committee. In: COLT ’92: Proceedings of the Fifth Annual Workshop on Computational Learning Theory, pp. 287–294. ACM, New York (1992)

    Google Scholar 

  109. Shannon, C.E.: A mathematical theory of communication (parts I and II): Bell Syst. Techn. J. XXVII, 379–423 (1948)

    Google Scholar 

  110. Singh, S., Barto, A.G., Chentanez, N.: Intrinsically motivated reinforcement learning. In: Advances in Neural Information Processing Systems 17 (NIPS). MIT, Cambridge (2005)

    Google Scholar 

  111. Singh, S., Lewis, R.L., Barto, A.G.: Where do rewards come from? In: Taatgen, N., van Rijn, H. (eds.) Proceedings of the 31st Annual Conference of the Cognitive Science Society. Austin (2009)

    Google Scholar 

  112. Solomonoff, R.J.: A formal theory of inductive inference. Part I. Inform. Control 7, 1–22 (1964)

    MathSciNet  MATH  Google Scholar 

  113. Solomonoff, R.J.: Complexity-based induction systems. IEEE Trans. Inform. Theory IT-24(5), 422–432 (1978)

    MATH  Google Scholar 

  114. Storck, J., Hochreiter, S., Schmidhuber, J.: Reinforcement driven information acquisition in non-deterministic environments. In: Proceedings of the International Conference on Artificial Neural Networks, Paris, vol. 2, pp. 159–164. EC2 & Cie (1995)

    Google Scholar 

  115. Strehl, A., Langford, J., Kakade, S.: Learning from logged implicit exploration data. Technical Report arXiv:1003.0120 (2010)

    Google Scholar 

  116. Stronger, D., Stone, P.: Towards autonomous sensor and actuator model induction on a mobile robot. Connect. Sci. 18(2) (2006)

    Google Scholar 

  117. Sutton, R., Barto, A.: Reinforcement Learning: An Introduction. MIT, Cambridge (1998)

    MATH  Google Scholar 

  118. Sutton, R.S., McAllester, D.A., Singh, S.P., Mansour, Y.: Policy gradient methods for reinforcement learning with function approximation. ü In: Solla, S.A., Leen, T.K., Müller, K.-R. (eds.) Advances in Neural Information Processing Systems 12 [NIPS Conference, Denver, Colorado, USA, November 29 – December 4(1999)], pp. 1057–1063. MIT, Cambridge (1999)

    Google Scholar 

  119. Wallace, C.S., Boulton, D.M.: An information theoretic measure for classification. Comput. J. 11(2), 185–194 (1968)

    MATH  Google Scholar 

  120. Wallace, C.S., Freeman, P.R.: Estimation and inference by compact coding. J. R. Stat. Soc. B 49(3), 240–265 (1987)

    MathSciNet  MATH  Google Scholar 

  121. Watkins, C., Dayan, P.: Q-learning. Mach. Learn. 8(3/4), 279–292 (1992)

    MATH  Google Scholar 

  122. Wiering, M., Schmidhuber, J.: HQ-learning. Adap. Behav. 6(2), 219–246 (1998a)

    Google Scholar 

  123. Wiering, M.A., Schmidhuber, J.: Fast online Q(λ): Mach. Learn. 33(1), 105–116 (1998b)

    MATH  Google Scholar 

  124. Wierstra, D., Schaul, T., Peters, J., Schmidhuber, J.: Fitness expectation maximization. In: Proceedings of Parallel Problem Solving from Nature (PPSN 2008) (2008)

    Google Scholar 

  125. Wilson, S.: ZCS: A zeroth level classifier system. Evol. Comput. 2, 1–18 (1994)

    Google Scholar 

  126. Wundt, W.M.: Grundzüge der Phvsiologischen Psychologie. Engelmann, Leipzig (1874)

    Google Scholar 

Download references

Acknowledgements

Thanks to Benjamin Kuipers, Herbert W. Franke, Marcus Hutter, Andy Barto, Jonathan Lansey, Michael Littman, Julian Togelius, Faustino J. Gomez, Giovanni Pezzulo, Gianluca Baldassarre, Martin Butz, Moshe Looks, Mark Ring, and several anonymous reviewers for useful comments that helped to improve this chapter or earlier papers on this subject. This research has received funds from the European Commission 7th Framework Programme (FP7/2007–2013), “Challenge 2: Cognitive Systems, Interaction, Robotics,” Grant Agreement No. ICT-IP-231722, Project “IM-CLeVeR: Intrinsically Motivated Cumulative Learning Versatile Robots.”

Author information

Authors and Affiliations

  1. IDSIA, Galleria 2, 6928, Manno-Lugano, Switzerland

    Jürgen Schmidhuber

  2. University of Lugano & SUPSI, Manno-Lugano, Switzerland

    Jürgen Schmidhuber

Authors
  1. Jürgen Schmidhuber
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jürgen Schmidhuber .

Editor information

Editors and Affiliations

  1. Istituto di Scienze e Tecnologie della Cognizione Consiglio Nazionale delle Ricerche, Rome, Italy

    Gianluca Baldassarre  & Marco Mirolli  & 

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Schmidhuber, J. (2013). Maximizing Fun by Creating Data with Easily Reducible Subjective Complexity. In: Baldassarre, G., Mirolli, M. (eds) Intrinsically Motivated Learning in Natural and Artificial Systems. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-32375-1_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-32375-1_5

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-32374-4

  • Online ISBN: 978-3-642-32375-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation