Skip to main content
SpringerLink
Log in

From humans to humanoids: The optimal control framework

  • Review Article
  • Published:
Paladyn

Abstract

In the last years of research in cognitive control, neuroscience and humanoid robotics have converged to different frameworks which aim, on one side, at modeling and analyzing human motion, and, on the other side, at enhancing motor abilities of humanoids. In this paper we try to cover the gap between the two areas, giving an overview of the literature in the two fields which concerns the production of movements. First, we survey computational motor control models based on optimality principles; then, we review available implementations and techniques to transfer these principles to humanoid robots, with a focus on the limitations and possible improvements of the current implementations. Moreover, we propose Stochastic Optimal Control as a framework to take into account delays and noise, thus catching the unpredictability aspects typical of both humans and humanoids systems. Optimal Control in general can also easily be integrated with Machine Learning frameworks, thus resulting in a computational implementation of human motor learning. This survey is mainly addressed to roboticists attempting to implement human-inspired controllers on robots, but can also be of interest for researchers in other fields, such as computational motor control.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Adams, B., Breazeal, C., Brooks, R. A., and Scassellati, B. (2000). Humanoid robots: a new kind of tool. IEEE Intelligent Systems, 15(4):25–31.

    Article  Google Scholar 

  2. Alexander, R. M. (1997). A minimum energy cost hypothesis for human arm trajectories. Biological cybernetics, 76(2):97–105.

    Article  MATH  Google Scholar 

  3. Andersen, R. A., Snyder, L. H., Bradley, D. C., and Xing, J. (1997). Multimodal representation of space in the posterior parietal cortex and its use in planning movements. Annual review of Neuroscience, 20(1):303–330.

    Article  Google Scholar 

  4. Arechavaleta, G., Laumond, J.-P., Hicheur, H., and Berthoz, A. (2008). An optimality principle governing human walking. IEEE Transactions on Robotics, 24:5–14.

    Article  Google Scholar 

  5. Argall, B. D., Chernova, S., Veloso, M., and Browning, B. (2009). A survey of robot learning from demonstration. Robotics and Autonomous Systems, 57(5):469–483.

    Article  Google Scholar 

  6. Atkeson, C. G., Hale, J. G., Pollick, F., Riley, M., Kotosaka, S., Schaul, S., Shibata, T., Tevatia, G., Ude, A., Vijayakumar, S., Kawato, E., and Kawato, M. (2000). Using humanoid robots to study human behavior. IEEE Intelligent Systems and Their Applications, 15(4):46–56.

    Article  Google Scholar 

  7. Atkeson, C. G. and Stephens, B. (2007). Multiple balance strategies from one optimization criterion. 2007 7th IEEE-RAS Int. Conf. on Humanoid Robots}, pages 57–6

    Google Scholar 

  8. Barambones, O. and Etxebarria, V. (2002). Robust neural control for robotic manipulators. Automatica, 38:235–242.

    Article  MATH  Google Scholar 

  9. Bauml, B., Wimbock, T., and Hirzinger, G. (2010). Kinematically optimal catching a flying ball with a hand-arm-system. In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, pages 2592–2599, Taipei, Taiwan.

  10. Bellman, R. (1957). Dynamic Programming. Princeton University Press, Princeton, NJ.

    MATH  Google Scholar 

  11. Ben-Itzhak, S. and Karniel, A. (2008). Minimum acceleration criterion with constraints implies bang-bang control as an underlying principle for optimal trajectories of arm reaching movements. Neural Computation, 20(3):779–812.

    Article  MathSciNet  MATH  Google Scholar 

  12. Bennequin, D., Fuchs, R., Berthoz, A., and Flash, T. (2009). Movement timing and invariance arise from several geometries. PLoS Computational Biology, 5(7):e1000426.

    Article  MathSciNet  Google Scholar 

  13. Bernstein, N. (1967). The Co-ordination and Regulation of Movements. Oxford, UK: Pergamo.

    Google Scholar 

  14. Berret, B., Chiovetto, E., Nori, F., and Pozzo, T. (2011a). Evidence for composite cost functions in arm movement planning: An inverse optimal control approach. PLoS Computational Biology, 7(10):e1002183.

    Article  MathSciNet  Google Scholar 

  15. Berret, B., Darlot, C., Jean, F., Pozzo, T., Papaxanthis, C., and Gauthier, J.-P. (2008). The inactivation principle: mathematical solutions minimizing the absolute work and biological implications for the planning of arm movements. PLoS Computational Biology, 4(10):e1000194.

    Article  MathSciNet  Google Scholar 

  16. Berret, B., Ivaldi, S., Nori, F., and Sandini, G. (2011b). Stochastic optimal control with variable impedance manipulators in presence of uncertainties and delayed feedback. In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, pages 4354–4359.

  17. Bertsekas, D. P. (1995). Dynamic Programming and Optimal Control. Athena Scientific.

  18. Bertsekas, D. P. and Tsitsiklis, J. N. (1996). Neuro-dynamic programming. Athena Scientific.

  19. Biess, A., Liebermann, D. G., and Flash, T. (2007). A computational model for redundant human three-dimensional pointing movements: integration of independent spatial and temporal motor plans simplifies movement dynamics. The Journal of Neuroscience, 27(48):13045–13064.

    Article  Google Scholar 

  20. Billard, A., Calinon, S., Dillmann, R., and Schaal, S. (2007). Handbook of Robotics (Siciliano, B. and Khatib, O. Eds), Robot Programming by Demonstration, pages 1371–1394. Springer.

  21. Blair, J. and Iwasaki, T. (2011). Optimal Gaits for Mechanical Rectifier Systems. IEEE Transactions on Automatic Control, 56(1):59–71.

    Article  MathSciNet  Google Scholar 

  22. Braganza, D., Dixon, W. E., Dawson, D. M., and Xian, B. (2005). Tracking control for robot manipulators with kinematic and dynamic uncertainty. In 44th IEEE Conf. on Decision and Control.

  23. Braun, D. A., Aertsen, A., Wolpert, D. M., and Mehring, C. (2009). Learning optimal adaptation strategies in unpredictable motor tasks. The Journal of Neuroscience, 29(20):6472–6478.

    Article  Google Scholar 

  24. Bryson, A. E. and Ho, Y.-C. (1975). Applied Optimal Control: Optimization, Estimation, and Control. John Wiley & Sons Inc.

  25. Buchli, J., Stulp, F., Theodorou, E., and Schaal, S. (2011). Learning variable impedance control. The Int. Journal of Robotics Research, 30(7):820–833.

    Article  Google Scholar 

  26. Buchli, J., Theodorou, E., Stulp, F., and Schaal, S. (2010). Variable impedance control — a reinforcement learning approach. In Robotics Science and Systems.

  27. Butz, M., Pedersen, G., and Stalph, P. (2009). Learning sensorimotor control structures with XCSF: redundancy exploitation and dynamic control. In 11th Annual Conf. on Genetic and Evolutionary Computation, pages 1171–1178.

  28. Caccavale, F., Chiaverini, S., and Siciliano, B. (1997). Secondorder kinematic control of robot manipulators with jacobian damped least-squares inverse: theory and experiments. IEEE/ASME Transactions on Mechatronics, 2(3):188–194.

    Article  Google Scholar 

  29. Cardinali, L., Frassinetti, F., Brozzoli, C., Urquizar, C., Roy, A. C., and Farnè, A. (2009). Tool-use induces morphological updating of the body schema. Current Biology, 19(12):R478–9.

    Article  Google Scholar 

  30. Cesqui, B., d’Avella, A., Portone, A., and Lacquaniti, F. (2012). Catching a ball at the right time and place: individual factors matter. PLoS one, 7(2):e31770.

    Article  Google Scholar 

  31. Chevallereau, C. and Aoustin, Y. (2001). Optimal reference trajectories for walking and running of a biped robot. Robotica, 19:557–569.

    Article  Google Scholar 

  32. Chiaverini, S., Egeland, O., and Kanestrom, R. K. (1991). Achieving user-defined accuracy with damped least-squares inverse kinematics. In 5th Int. Conf. on Advanced Robotics, pages 672–677.

  33. Crevecoeur, F., Thonnard, J.-L., and Lefèvre, P. (2009). Optimal integration of gravity in trajectory planning of vertical pointing movements. Journal of neurophysiology, 102(2):786–796.

    Article  Google Scholar 

  34. da Silva, M., Durand, F., and Popovic, J. (2009). Linear Bellman combination for control of character animation. ACM Transactions on Graphics, 28(3):1.

    Article  Google Scholar 

  35. Dahiya, R. S., Metta, G., Valle, M., and Sandini, G. (2010). Tactile sensing: From humans to humanoids. IEEE Transactions on Robotics, 26(1):1–20.

    Article  Google Scholar 

  36. Dapena, J. (1980a). Mechanics of rotation in the fosbury-flop. Medicine and Science in Sports and Exercise, 12(1):45–53.

    Google Scholar 

  37. Dapena, J. (1980b). Mechanics of translation in the fosbury-flop. Medicine and Science in Sports and Exercise, 12(1):37–44.

    Google Scholar 

  38. Dapena, J. (2002). The evolution of high jumping technique: Biomechanical analysis. In of 20th Internat. Symp. Biomech. Sports, C ceres, Spain.

  39. De Santis, A., Siciliano, B., De Luca, A., and Bicchi, A. (2008). An atlas of physical human-robot interaction. Mechanism and Machine Theory, 43(3):253–270.

    Article  MATH  Google Scholar 

  40. Desmurget, M. and Grafton, S. (2000). Forward modeling allows feedback control for fast reaching movements. Trends in Cognitive Sciences, 4:423–431.

    Article  Google Scholar 

  41. Diedrichsen, J., Shadmehr, R., and Ivry, R. B. (2010a). The coordination of movement: optimal feedback control and beyond. Trends in Cognitive Sciences, 14(1):31–39.

    Article  Google Scholar 

  42. Diedrichsen, J., White, O., Newman, D., and Lally, N. (2010b). Use-dependent and error-based learning of motor behaviors. Journal of Neuroscience, 30(15):5159–5166.

    Article  Google Scholar 

  43. Diehl, M., Bock, H. G., Diedam, H., and Wieber, P. B. (2006). Fast Motions in Biomechanics and Robotics (Diehl, M. and Mombaur, K. Eds), vol. 340, Fast Direct Multiple Shooting algorithms for optimal robot control, pages 65–93. LNCIS, Springer.

  44. Diehl, M., Ferreau, H. J., and Haverbeke, N. (2009). Nonlinear Model Predictive Control (Magni, L. et al. Eds), vol. 384, Effcient numerical methods for nonlinear MPC and Moving Horizon estimation, pages 541–550. LNCIS, Springer.

  45. Dominici, N., Ivanenko, Y. P., Cappellini, G., d’Avella, A., Mond, V., Cicchese, M., Fabiano, A., Silei, T., Di Paolo, A., Giannini, C., Poppele, R. E., and Lacquaniti, F. (2011). Locomotor primitives in newborn babies and their development. Science, 334(6058):997–999.

    Article  Google Scholar 

  46. Dupree, K., Patre, P., Johnson, M., and Dixon, W. (2009). Inverse optimal adaptive control of a nonlinear euler-lagrange system, part i: Full state feedback. In 48th IEEE Conf. on Decision and Control, pages 321–326.

  47. Feldman, A. G. and Levin, M. F. (1995). The origin and use of positional frames of reference in motor control. Behavioral and Brain Sciences, 18(4):723–744.

    Article  Google Scholar 

  48. Fitts, P. (1954). The information capacity of the human motor system in controlling the amplitude of movement. J. Exp. Psychol., 47(6):381–391.

    Article  Google Scholar 

  49. Flash, T. and Hogan, N. (1985). The coordination of arm movements: an experimentally confirmed mathematical model. The Journal of Neuroscience, 5(7):1688–1703.

    Google Scholar 

  50. Franklin, D. W., Burdet, E., Tee, K. P., Osu, R., Chew, C.-M., Milner, T. E., and Kawato, M. (2008). CNS learns stable, accurate, and effcient movements using a simple algorithm. The Journal of Neuroscience, 28(44):11165–11173.

    Article  Google Scholar 

  51. Franklin, D. W., So, U., Burdet, E., and Kawato, M. (2007). Visual feedback is not necessary for the learning of novel dynamics. PloS one, 2(12):e1336.

    Article  Google Scholar 

  52. Friston, K. (2010). The free-energy principle: a unified brain theory? Nature Reviews, 11:127–138.

    Article  Google Scholar 

  53. Ganesh, G., Albu-Schaffer, A., Haruno, M., Kawato, M., and Burdet, E. (2010). Biomimetic motor behavior for simultaneous adaptation of force, impedance and trajectory in interaction tasks. In IEEE Int. Conf. on Robotics and Automation, pages 2705–2711.

  54. Gepshtein, S., Seydell, A., and Trommershäuser, J. (2007). Optimality of human movement under natural variations of visualmotor uncertainty. Journal of Vision, 7(5):1–18.

    Article  Google Scholar 

  55. Gienger, M., Janssen, H., and Goerick, C. (2005). Task-oriented whole body motion for humanoid robots. In 5th IEEE-RAS Int. Conf. on Humanoid Robots, pages 238–244.

  56. Guigon, E. (2011). Motor Control (Danion, F. and Latash, M.L. Eds), Models and Architectures for motor control: Simple or complex?, pages 478–502. Oxford University Press.

  57. Guigon, E., Baraduc, P., and Desmurget, M. (2008a). Computational motor control: feedback and accuracy. European Journal of Neuroscience, 27(4):1003–1016.

    Article  Google Scholar 

  58. Guigon, E., Baraduc, P., and Desmurget, M. (2008b). Optimality, stochasticity and variability in motor behavior. Journal of Computational Neuroscience, 24(1):57–68.

    Article  Google Scholar 

  59. Haddadin, S., Albu-Schäffer, A., and Hirzinger, G. (2010). Safety analysis for a human-friendly manipulator. Int. Journal of Social Robotics, 2:235–252.

    Article  Google Scholar 

  60. Hansen, N., Muller, S. D., and Koumoutsakos, P. (2003). Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evolutionary Computation, 11(1):1–18.

    Article  Google Scholar 

  61. Harris, C. M. and Wolpert, D. M. (1998). Signal-dependent noise determines motor planning. Nature, 394(6695):780–784.

    Article  Google Scholar 

  62. Harris, C. M. and Wolpert, D. M. (2006). The main sequence of saccades optimizes speed-accuracy trade-off. Biological Cybernetics, 95(1):21–29.

    Article  MathSciNet  MATH  Google Scholar 

  63. Hauser, H., Neumann, G., Ijspeert, A. J., and Maass, W. (2011). Biologically inspired kinematic synergies enable linear balance control of a humanoid robot. Biological cybernetics, 104(4-5):235–249.

    Article  MathSciNet  MATH  Google Scholar 

  64. He, G.-P. and Geng, Z.-Y. (2007). Optimal motion planning of a one-legged hopping robot. In IEEE Int. Conf. on Robotics and Biomimetics, pages 1178–1183, Sanya, China.

  65. Heidrich-Meisner, V. and Igel, C. (2008). Similarities and differences between policy gradient methods and evolution strategies. In 16th Europ. Symp. on Artificial Neural Networks (ESANN), pages 149–154.

  66. Herbort, O. and Butz, M. (2011). The continuous end-state comfort effect: weighted integration of multiple biases. Psychological Research, pages 1–19.

  67. Ijspeert, A. J., Nakanishi, J., and Schaal, S. (2003). Learning attractor landscapes for learning motor primitives. In Advances in Neural Information Processing Systems 15, volume 15, pages 1547–1554.

    Google Scholar 

  68. Ivaldi, S., Baglietto, M., Metta, G., and Zoppoli, R. (2009). Nonlinear Model Predictive Control (Magni, L. et al. Eds), vol. 384, An application of receding-horizon neural control in humanoid robotics, pages 541–550. LNCIS, Springer.

  69. Ivaldi, S., Fumagalli, M., Nori, F., Baglietto, M., and Metta, G. (2010). Approximate optimal control for reaching and trajectory planning in a humanoid robot. In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, pages 1290–1296, Taipei, Taiwan.

  70. Ivaldi, S., Fumagalli, M., Randazzo, M., Nori, F., Metta, G., and Sandini, G. (2011). Computing robot internal/external wrenches by means of inertial, tactile and F/T sensors: theory and implementation on the iCub. In 11th IEEE-RAS Int.Conf. on Humanoid Robots, pages 521–528.

    Google Scholar 

  71. Izawa, J., Rane, T., Donchin, O., and Shadmehr, R. (2008). Motor adaptation as a process of reoptimization. The Journal of Neuroscience, 28(11):2883–2891.

    Article  Google Scholar 

  72. Kadiallah, A., Liaw, G., Burdet, E., Kawato, M., and Franklin, D. W. (2008). Impedance control is tuned to multiple directions of movement. In IEEE Int. Eng. Med. Biol. Soc. Conf., pages 5358-5361.

  73. Kajita, S., Kanehiro, F., Kaneko, K., Fujiwara, K., Harada, K., Yokoi, K., and Hirukawa, H. (2003). Resolved momentum control: Humanoid motion planning based on the linear and angular momentum. In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, volume 2, pages 1644–1650.

    Google Scholar 

  74. Kaneko, Y., Nakano, E., Osu, R., Wada, Y., and Kawato, M. (2005). Trajectory formation based on the minimum commanded torque change model using euler-poisson equation. Systems and Computers in Japan, 36:92–103.

    Article  Google Scholar 

  75. Kanoun, O., Yoshida, E., and Laumond, J.-P. (2009). An optimization formulation for footsteps planning. In IEEE-RAS Int. Conf. on Humanoid Robots, pages 202–207, Paris, France.

  76. Kappen, H. J. (2005). A linear theory for control of non-linear stochastic systems. Physical Review Letters, 95:200–201.

    Article  MathSciNet  Google Scholar 

  77. Kim, Y. H., Lewis, F. L., and Dawson, D. M. (2000). Intelligent optimal control of robotic manipulators using neural networks. Automatica, 36:1355–1364.

    Article  MathSciNet  MATH  Google Scholar 

  78. Kirk, D. E. (1970). Optimal control theory: An Introduction. Prentice-Hall, New Jersey.

    Google Scholar 

  79. Kober, J. and Peters, J. (2008). Policy search for motor primitives in robotics. Advances in Neural Information Processing Systems (NIPS), pages 1–8.

  80. Kodl, J., Ganesh, G., and Burdet, E. (2011). The CNS Stochastically Selects Motor Plan Utilizing Extrinsic and Intrinsic Representations. PLoS one, 6(9):e24229.

    Article  Google Scholar 

  81. Konczak, J. and Dichgans, J. (1997). The development toward stereotypic arm kinematics during reaching in the first 3 years of life. Experimental Brain Research, 117:346–354.

    Article  Google Scholar 

  82. Kormushev, P., Calinon, S., and Caldwell, D. (2010). Robot motor skill coordination with em-based reinforcement learning. In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, pages 3232–3237.

  83. Krstic, M. (2009). Inverse optimal adaptive control: the interplay between update laws, control laws, and lyapunov functions. In American Control Conf., pages 1250–1255.

  84. Kuo, A. (2005). An optimal state estimation model of sensory integration in human postural balance. Journal of Neural Engineering, 2:S235–S249.

    Article  Google Scholar 

  85. Lacquaniti, F., Terzuolo, C., and Viviani, P. (1983). The law relating kinematic and figural aspects of drawing movements. Acta Psychologica, 54:115–130.

    Article  Google Scholar 

  86. Lan, N. and Crago, P. E. (1994). Optimal control of antagonistic muscle stiffness during voluntary movements. Biological cybernetics, 71(2):123–135.

    Article  MATH  Google Scholar 

  87. Lengagne, S., Ramdani, N., and Fraisse, P. (2009). Planning and fast re-planning of safe motions for humanoid robots: Application to a kicking motion. In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, pages 441–446.

  88. Lenzi, T., Vitiello, N., McIntyre, J., Roccella, S., and Carrozza, M. C. (2011). A robotic model to investigate human motor control. Biological Cybernetics, 105(1):1–19.

    Article  MATH  Google Scholar 

  89. Li, W. and Todorov, E. (2004). Iterative linear quadratic regulator applied to nonlinear biological movement systems. In 1st Int. Conf. on Informatics in Control, Automation and Robotics, pages 222–229.

  90. Lockhart, D. and Ting, L. (2007). Optimal sensorimotor transformations for balance. Nature Neuroscience, 10(10):1329–1336.

    Article  Google Scholar 

  91. MacKenzie, I. S. (1992). Fitts’ law as a research and design tool in human-computer interaction. Human-Computer Interaction, 7:91–139.

    Article  Google Scholar 

  92. Marin, D. and Sigaud, O. (2012). Towards fast and adaptive optimal control policies for robots: A direct policy search approach. In Proceedings Robotica, pages 21–26.

  93. Matsui, T. (2008). A new optimal control model for reproducing two-point reaching movements of human three-joint arm with wrist joint’s freezing mechanism. In IEEE Int. Conf. on Robotics and Biomimetics, pages 383–388.

  94. Matsui, T., Honda, M., and Nakazawa, N. (2006). A new optimal control model for reproducing human arm’s two-point reaching movements: a modified minimum torque change model. In IEEE Int. Conf. on Robotics and Biomimetics, pages 1541–1546.

  95. Matsui, T., Takeshita, K., and Shibusawa, T. (2009). Effectiveness of human three-joint arm’s optimal control model characterized by hand-joint’s freezing mechanism in reproducing constrained reaching movement characteristics. In ICROS-SICE Int. Joint Conf., pages 1206–1211.

  96. Mettin, U., Shiriaev, A. S., Freidovich, L. B., and Sampei, M. (2010). Optimal ball pitching with an underactuated model of a human arm. In IEEE Int. Conf. on Robotics and Automation, pages 5009–5014.

  97. Mistry, M., Theodorou, E., Liaw, G., Yoshioka, T., Schaal, S., and Kawato, M. (2008). Adaptation to a sub-optimal desired trajectory. In Society for Neuroscience — Symp. on Advances in Computational Motor Control, Washington DC, USA.

  98. Mitrovic, D., Klanke, S., and Vijayakumar, S. (2010). From Motor to Interaction Learning in Robotics (Sigaud, O. and Peters, J. Eds), vol. 264, Adaptive Optimal Feedback Control with Learned Internal Dynamics Models, pages 65–84. Springer-Verlag.

    Article  Google Scholar 

  99. Mombaur, K., Laumond, J.-P., and Yoshida, E. (2008). An optimal control model unifying holonomic and nonholonomic walking. In 8th IEEE-RAS Int. Conf. on Humanoid Robots, Daejeon, Korea.

  100. Mombaur, K., Truong, A., and Laumond, J.-P. (2010). From human to humanoid locomotion: an inverse optimal control approach. Autonomous Robots, 28:369–383.

    Article  Google Scholar 

  101. Morasso, P. (1983). Three dimensional arm trajectories. Biological Cybernetics, 48:187–194.

    Article  Google Scholar 

  102. Mugan, J. and Kuipers, B. (2009). Autonomously learning an action hierarchy using a learned qualitative state representation. In Int. Joint Conf. on Artificial Intelligence.

  103. Mussa-Ivaldi, F. A., Giszter, S. F., and Bizzi, E. (1994). Linear combinations of primitives in vertebrate motor control. Proc. National Academy of Sciences USA, 91(16):7534–7538.

    Article  Google Scholar 

  104. Nagengast, A. J., Braun, D. A., and Wolpert, D. M. (2011). Risksensitivity and the mean-variance trade-off: decision making in sensorimotor control. Proceedings Biological Sciences / The Royal Society, 278(1716):2325–2332.

    Article  Google Scholar 

  105. Nakamura, Y. (1991). Advanced Robotics: redundancy and optimization. Addison Wesley.

  106. Nakano, E., Imamizu, H., Osu, R., Uno, Y., Gomi, H., Yoshioka, T., and Kawato, M. (1999). Quantitative examinations of internal representations for arm trajectory planning: Minimum commanded torque change model. Journal of Neurophysiology, 81:2140–2155.

    Google Scholar 

  107. Nakaoka, S., Nakazawa, A., Yokoi, K., Hirukawa, H., and Ikeuchi, K. (2003). Generating whole body motions for a biped humanoid robot from captured human dances. In IEEE Int. Conf. on Robotics and Automation, volume 3, pages 3905–3910.

    Google Scholar 

  108. Nelson, W. L. (1983). Physical principles for economies of skilled movements. Biological Cybernetics, 46:135–147.

    Article  MATH  Google Scholar 

  109. Nishii, J. and Murakami, T. (2002). Energetic optimality of arm trajectory. In Int. Conf. on Biomechanics of Man, pages 30-33.

  110. Nori, F. and Frezza, R. (2005). A control theory approach to the analysis and synthesis of the experimentally observed motion primitives. Biological Cybernetics, 93(5):323–342.

    Article  MathSciNet  MATH  Google Scholar 

  111. Oksendal, B. (1995). Stochastic Differential Equations. Springer Berlin, 4th edition.

  112. Parker, G. A. and Smith, J. M. (1990). Optimality theory in evolutionary biology. Nature, 348(6296):27–33.

    Article  Google Scholar 

  113. Pattacini, U., Nori, F., Natale, L., Metta, G., and Sandini, G. (2010). An experimental evaluation of a novel minimum-jerk cartesian controller for humanoid robots. In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, Taipei, Taiwan.

  114. Peters, J. and Schaal, S. (2007). Natural actor-critic. Neurocomputing, 71:1180–1190.

    Article  Google Scholar 

  115. Peters, J. and Schaal, S. (2008). Reinforcement learning of motor skills with policy gradients. Neural networks, 21:682–97.

    Article  Google Scholar 

  116. Pollard, N. S., Hodgins, J. K., Riley, M. J., and Atkeson, C. G. (2002). Adapting human motion for the control of a humanoid robot. In IEEE Int. Conf. on Robotics and Automation, volume 2, pages 1390–1397.

    Google Scholar 

  117. Pontryagin, L. S., Boltyanskii, V. G., Gamkrelidze, R. V., and Mishchenko, E. F. (1964). The Mathematical Theory of Optimal Processes. Pergamon Press.

  118. Pouget, A. and Snyder, L. (2000). Computational approaches to sensorimotor transformations. Nature Neuroscience, 3:1192–1198.

    Article  Google Scholar 

  119. Pozzo, T., Berthoz, A., and Lefort, L. (1990). Head stabilisation during various locomotor tasks in humans. i. normal subjects. Experimental Brain Research, 82:97–106.

    Article  Google Scholar 

  120. Ribas-Fernandes, J. J. F., Solway, A., Diuk, C., McGuire, J. T., Barto, A. G., Niv, Y., and Botvinick, M. M. (2011). A neural signature of hierarchical reinforcement learning. Neuron, 71(2):370–379.

    Article  Google Scholar 

  121. Richardson, M. J. E. and Flash, T. (2000). On the emulation of natural movements by humanoid robots. In IEEE-RAS Int. Conf. on Humanoids Robots.

  122. Richardson, M. J. E. and Flash, T. (2002). Comparing smooth arm movements with the two-thirds power law and the related segmented-control hypothesis. Journal of Neuroscience, 22(18):8201–8211.

    Google Scholar 

  123. Rigoux, L., Sigaud, O., Terekhov, A., and Guigon, E. (2010). Movement duration as an emergent property of reward directed motor control. In Annual Symp. Advances in Computational Motor Control.

  124. Rubinstein, R. Y. (1997). Optimization of computer simulation models with rare events. European Journal of Operational Research, 99(1):89–112.

    Article  Google Scholar 

  125. Salini, J., Padois, V., and Bidaud, P. (2011). Synthesis of complex humanoid whole-body behavior: a focus on sequencing and tasks transitions. In IEEE Int. Conf. on Robotics and Automation, pages 1283–1290.

  126. Sastry, S. and Bodson, M. (1994). Adaptive Control: Stability, Convergence, and Robustness. Advanced Reference Series (Engineering). Prentice-Hall.

  127. Schaal, S. (1997). Advances in Neural Information Processing Systems (Mozer, M.C. et al. Eds), Learning from demonstration, pages 1040–1046. MIT Press.

  128. Schaal, S., Peters, J., Nakanishi, J., and Ijspeert, A. J. (2003). Learning movement primitives. In Int. Symp. on Robotics Research (ISRR), pages 561-572.

  129. Schaal, S. and Schweighofer, N. (2005). Computational motor control in humans and robots. Current Opinion in Neurobiology, 15:675–682.

    Article  Google Scholar 

  130. Scheidt, R. A., Reinkensmeyer, D. J., Conditt, M. A., Rymer, W. Z., and Mussa-Ivaldi, F. A. (2000). Persistence of motor adaptation during constrained, multi-joint, arm movements. Journal of Neurophysiology, 84(2):853–862.

    Google Scholar 

  131. Schmidt, R. A. (1975). A schema theory of discrete motor skill learning. Psychological review, 82(4):225.

    Article  Google Scholar 

  132. Schmidt, R. A., Zelaznik, H., Hawkins, B., Frank, J. S., and Quinn, J. T. (1979). Motor output variability: a theory for the accuracy of rapid motor acts. Psychol. Rev., 47:415–51.

    Article  Google Scholar 

  133. Schöner, G. and Kelso, J. A. (1988). Dynamic pattern generation in behavioral and neural systems. Science, 239(4847):1513.

    Article  Google Scholar 

  134. Schultz, G. and Mombaur, K. (2010). Modeling and optimal control of human-like running. IEEE/ASME Trans. on Mechatronics, 15(5):783–792.

    Article  Google Scholar 

  135. Scott, S. (2004). Optimal feedback control and the neural basis of volitional motor control. Nature Reviews Neuroscience, 5:532–546.

    Article  Google Scholar 

  136. Scott Kelso, J. A. (1982). Human motor behavior: an introduction. Lawrence Erlbaum Associates.

  137. Seki, H. and Tadakuma, S. (2004). Minimum jerk control of power assisting robot based on human arm behavior characteristics. In IEEE Int. Conf. on System, Man and Cybernetics, volume 1, pages 722–727.

    Google Scholar 

  138. Sentis, L. and Khatib, O. (2005). Synthesis of whole-body behaviors through hierarchical control of behavioral primitives. The Int. Journal of Humanoid Robotics, 2(4):505–518.

    Article  Google Scholar 

  139. Shadmehr, R. and Krakauer, J. W. (2008). A computational neuroanatomy for motor control. Experimental Brain Research, 185:359–381.

    Article  Google Scholar 

  140. Shadmehr, R., Orban de Xivry, J.-J., Xu-Wilson, M., and Shih, T.-Y. (2010). Temporal discounting of reward and the cost of time in motor control. The Journal of Neuroscience, 30(31):10507–10516.

    Article  Google Scholar 

  141. Shadmehr, R. and Wise, S. (2005). The Computational Neurobiology of Reaching and Pointing: a foundation for Motor Learning. MIT Press.

  142. Shiller, Z. and Dubowsky, S. (1991). On computing the global time-optimal motions of robotic manipulators in the presence of obstacles. IEEE Transactions on Robotics and Automation, 7(6):785–797.

    Article  Google Scholar 

  143. Sigaud, O., Salaün, C., and Padois, V. (2011). On-line regression algorithms for learning mechanical models of robots: a survey. Robotics and Autonomous Systems, 51:1117–1125.

    Google Scholar 

  144. Simmons, G. and Demiris, Y. (2005). Optimal robot arm control using the minimum variance model. Journal of Robotic Systems, 22(11):677–690.

    Article  MATH  Google Scholar 

  145. Stengel, R. F. (1994). Optimal Control and Estimation. Dover Publications.

  146. Stulp, F., Buchli, J., Ellmer, A., Mistry, M., Theodorou, E., and Schaal, S. (2011). Reinforcement learning of impedance control in stochastic force fields. In IEEE Int. Conf. on Development and Learning, volume 2,pages 1–6.

    Article  Google Scholar 

  147. Stulp, F. and Sigaud, O. (2012). Path integral policy improvement with covariance matrix adaptation. In 29th Int. Conf. on Machine Learning.

  148. Sun, F. C., Li, H. X., and Li, L. (2002). Robot discrete adaptive control based on dynamic inversion using dynamical neural networks. Automatica, 38:1977–1983.

    Article  MATH  Google Scholar 

  149. Tanaka, H., Krakauer, J. W., and Qian, N. (2006). An optimization principle for determining movement duration. Journal of Neurophysiology, 95:38750–3886.

    Article  Google Scholar 

  150. Terekhov, A. V. and Zatsiorsky, V. M. (2011). Analytical and numerical analysis of inverse optimization problems: conditions of uniqueness and computational methods. Biological Cybernetics, 104:75–93.

    Article  MathSciNet  MATH  Google Scholar 

  151. Theodorou, E., Buchli, J., and Schaal, S. (2010). Reinforcement learning of motor skills in high dimensions: a path integral approach. In Int. Conf. on Robotics and Automation, pages 2397–2403. IEEE.

  152. Thrommershäuser, J., Maloney, L. T., and Landy, M. S. (2008). Decision making, movement planning, and statistical decision theory. Trends in Cognitive Sciences, 12(8):291–297.

    Article  Google Scholar 

  153. Tlalolini, D., Chevallereau, C., and Aoustin, Y. (2011). Humanlike walking: Optimal motion of a bipedal robot with toerotation motion. IEEE/ASME Transactions on Mechatronics, 16(2):310–320.

    Article  Google Scholar 

  154. Todorov, E. (2004). Optimality principles in sensorimotor control. Nature Neuroscience, 7(9):907–915.

    Article  Google Scholar 

  155. Todorov, E. (2005). Stochastic optimal control and estimation methods adapted to the noise characteristics of the sensorimotor system. Neural computation, 17(5):1084–1108.

    Article  MathSciNet  MATH  Google Scholar 

  156. Todorov, E. (2009a). Compositionality of optimal control laws. Advances in Neural Information Processing Systems, 3:2–6.

    Google Scholar 

  157. Todorov, E. (2009b). Effcient computation of optimal actions. Proc. Natl. Acad. Sci. USA, 106(28):11478–11483.

    Article  MATH  Google Scholar 

  158. Todorov, E. and Jordan, M. I. (2002). Optimal feedback control as a theory of motor coordination. Nature Neuroscience, 5(11):1226–1235.

    Article  Google Scholar 

  159. Todorov, E. and Jordan, M. I. (2003). A minimal intervention principle for coordinated movement. In Advances in Neural Information Processing Systems, volume 15, pages 27–34.

    Google Scholar 

  160. Todorov, E. and Li, W. (2005). A generalized iterative LQG method for locally-optimal feedback control of constrained nonlinear stochastic systems. In American Control Conf., pages 300-306.

  161. Toussaint, M., Gienger, M., and Goerick, C. (2007). Optimization of sequential attractor-based movement for compact behaviour generation. In IEEE-RAS Int. Conf. on Humanoid Robots, pages 122–129.

  162. Tuan, T., Souères, P., Taïx, M., and Guigon, E. (2008). A principled approach to biological motor control for generating humanoid robot reaching movements. In IEEE Int. Conf. Biomedical Robotics and Biomechatronics, pages 783–788.

  163. Uno, Y., Kawato, M., and Suzuki, R. (1989). Formation and control of optimal trajectory in human multijoint arm movement. Biological Cybernetics, 61:89–101.

    Article  Google Scholar 

  164. Vidyasagar, M. (1987). Control System Synthesis: A factorization approach. MIT Press.

  165. Vijayakumar, S. and Schaal, S. (2000). Locally Weighted Projection Regression: An O(n) Algorithm for Incremental Real Time Learning in High Dimensional Space. In 7th Int. Conf. on Machine Learning, pages 1079–1086.

  166. Viviani, P. (1986). Generation and modulation of action patterns (Heuer, H. and Fromm, C. Eds), Do units of motor action really exist?, pages 201–216. Springer-Verlag.

  167. Viviani, P. and Flash, T. (1995). Minimum-jerk, two-thirds power law, and isochrony: converging approaches to movement planning. Journal of Experimental Psychology: Human Perception and Performance, 21:32–53.

    Article  Google Scholar 

  168. Viviani, P. and Stucchi, N. (1992). Biological movements look constant: Evidence of motor perceptual interactions. Journal of Experimental Psychology: Human Perception and Performance, 18:603–623.

    Article  Google Scholar 

  169. Volkinshtein, D. and Meir, R. (2011). Delayed feedback control requires an internal forward model. Biological cybernetics, 105(1):41–53.

    Article  MathSciNet  MATH  Google Scholar 

  170. Wächter, A. and Biegler, L. (2006). On the implementation of a primal-dual interior point filter line search algorithm for largescale nonlinear programming. Mathematical Programming, 106:25–57.

    Article  MathSciNet  MATH  Google Scholar 

  171. Wada, Y., Kaneko, Y., Nakano, E., Osu, R., and Kawato, M. (2001). Quantitative examinations for multi joint arm trajectory planning-using a robust calculation algorithm of the minimum commanded torque change trajectory. Neural Networks, 14(4–5):381–393.

    Article  Google Scholar 

  172. Wada, Y., Yamanaka, K., Soga, Y., Tsuyuki, K., and Kawato, M. (2006). Can a kinetic optimization criterion predict both arm trajectory and final arm posture? Annual Int. Conf. of the IEEE Eng. Med. Biol. Society, 1:1197–200.

    Google Scholar 

  173. Whitman, E. C. and Atkeson, C. G. (2009). Control of a walking biped using a combination of simple policies. In IEEE-RAS Int. Conf. on Humanoid Robots, pages 520–527, Paris, France.

  174. Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3–4):229–256.

    MATH  Google Scholar 

  175. Wolpert, D. M. and Flanagan, J. R. (2001). Motor prediction. Current Biology, 18(18):729–732.

    Article  Google Scholar 

  176. Wolpert, D. M., Ghahramani, Z., and Jordan, M. I. (1995). Are arm trajectories planned in kinematic or dynamic coordinates? an adaptation study. Experimental Brain Research, 103:460–470.

    Article  Google Scholar 

  177. Wolpert, D. M., Miall, R. C., and Kawato, M. (1998). Internal models in the cerebellum. Trends in Cognitive Sciences, 2(9):338–347.

    Article  Google Scholar 

  178. Yoshida, E., Esteves, C., Kanoun, O., Poirier, M., Mallet, A., Laumond, J.-P., and Yokoi, K. (2010). Motion Planning for Humanoid Robots (Harada, K. et al. Eds), Planning Whole-body Humanoid Locomotion, Reaching, and Manipulation, pages 99–128. Springer.

  179. Zhao, H. and Chen, D. (1996). Optimal motion planning for flexible space robots. In IEEE Int. Conf. on Robotics and Automation, pages 393–398.

Download references

Author information

Authors and Affiliations

  1. Institut des Systèmes Intelligents et de Robotique — CNRS UMR 7222, Université Pierre et Marie Curie, Pyramide Tour 55 — Boîte Courrier 173, 4 Place Jussieu, 75252, Paris CEDEX 05, France

    Serena Ivaldi & Olivier Sigaud

  2. Robotics, Brain and Cognitive Sciences Dept. Istituto Italiano di Tecnologia, Via Morego 30, 16136, Genova, Italy

    Bastien Berret & Francesco Nori

Authors
  1. Serena Ivaldi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Olivier Sigaud
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bastien Berret
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Francesco Nori
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Serena Ivaldi.

About this article

Cite this article

Ivaldi, S., Sigaud, O., Berret, B. et al. From humans to humanoids: The optimal control framework. Paladyn 3, 75–91 (2012). https://doi.org/10.2478/s13230-012-0022-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.2478/s13230-012-0022-3

Keywords

Search

Navigation