Deep Dynamic Programming: Optimal Control with Continuous Model Learning of a Nonlinear Muscle Actuated Arm

  • Andrew G. LonsberryEmail author
  • Alexander J. Lonsberry
  • Roger D. Quinn
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10384)


We outline a new technique for on-line continuous model learning control and demonstrate its utility by controlling a simulated 2-DOF arm actuated by 6 muscles as well as on an inverted pendulum. Work presented is part of an effort to develop controllers for human appendages rendered inoperable by paralysis. Computerized control provides an alternative to neural regeneration by means of electric muscle stimulation. It has been demonstrated that paralyzed individuals can regain self-powered mobility via use of external muscle controllers. A barrier to proliferation of the technology, is the difficulty in control over the living system which is highly nonlinear and unique to each individual. Here we demonstrate a novel, continuous model learning technique to simultaneously learn and control continuous, non-linear systems. The technique expands upon vanilla Q-learning and dynamic programming. Unlike typical Q-learning, where the action-value function updates are only for the most recent set of states visited and stored in memory, the method presented also generates updates to the action-value function for unvisited state-space and state-space visited far in the past. This is made feasible by giving the agent the ability to continually learn and update explicit local models of the environment and of itself, which we encapsulated in a set of deep neural networks.


Optimal control Deep learning Continuous model learning 


  1. 1.
    Tahara, K., Arimoto, S., Sekimoto, M., Luo, Z.: On control of reaching movements for musculo-skeletal redundant arm model. Appl. Bionics Biomech. 6(1), 57–72 (2009)CrossRefGoogle Scholar
  2. 2.
    Mashima, H., Akazawa, K., Kushima, H., Fujii, K.: The force-load-velocity relation and viscous-like force in the frog skeletal muscle. Jpn. J. Physiol. 22, 103–120 (1972)CrossRefGoogle Scholar
  3. 3.
    Haeufle, D.F.B., Gunther, M., Bayer, A., Schmitt, S.: Hill-type muscle model with serial damping and eccentric fore-velocity relation. J. Biomech. 47(6), 1531–1536 (2014)CrossRefGoogle Scholar
  4. 4.
    Berkowitz, M., Harvery, C., Greene, C., et al.: The Economic Consequences of Traumatic Spinal Cord Injury. Demos Press, New York (1992)Google Scholar
  5. 5.
    Triolo, R.J., Bogie, K.: Lower extremity applications of functional neuromuscular stimulation after spinal cord injury. Top. SCI Rehabil. 5(1), 44–65 (1999)Google Scholar
  6. 6.
    Kobetic, R., Triolo, R.J., Marsolais, E.B.: Muscle selection and walking performance of multichannel FES systems for ambulation in paraplegia. IEEE Trans. Rehabil. Eng. 5(1), 23–29 (1997)CrossRefGoogle Scholar
  7. 7.
    Uhlir, J.P., Triolo, R.J., Kobetic, R.: The use of selective electrical stimulation of the quadriceps to improve standing function in paraplegia. IEEE Trans. Rehabil. Eng. 8(4), 514–522 (2000)CrossRefGoogle Scholar
  8. 8.
    Triolo, R.J., Wibowo, M., Uhlir, J., Kobetic, R., Kirsh, R.: Effects of stimulated hip extension moment and position on upper-limb support forces during FES-induced standing-a technical not. J. Rehabil. Res. Dev. 38(5), 545–555 (2001)Google Scholar
  9. 9.
    Solomonow, M., Reisin, E., Aguilar, E., Baratta, R.V., Best, R., D’Ambrosia, R.: Reciprocating gait orthosis powered with electrical muscle stimulation (RGO II). Part II: medical evaluation of 70 paraplegic patients. Orthopedics 10(5), 411–418 (1997)Google Scholar
  10. 10.
    Betz, R.R., Rosenfeld, E., Triolo, R.J., Robinson, D.E., Gardner, E.R., Maurer, A.: Bone mineral content in children with spinal cord injury. Poster Presented at American Spinal Cord Injury Association, San Diego, CA, May 1988Google Scholar
  11. 11.
    Marsolais, E.B., Kobetic, R., Polando, G., Ferguson, K., Tashman, S., Gaudioi, R., Nandurkar, S., Lehneis, H.R.: The Case Western Reserve University hybrid gait orthosis. J. Spinal Cord Med. 23(2), 100–108 (2000)CrossRefGoogle Scholar
  12. 12.
    Farris, R., Quintero, H., Goldfarb, M.: Preliminary evaluation of a powered lower limb orthosis to aid waling paraplegic individuals. IEEE Trans. Neural Syst. Rehabil. Eng. 19(6), 652–659 (2011)CrossRefGoogle Scholar
  13. 13.
    Watkins, C.J., Dayan, P.: Q-learning. Mach. Learn. 8(3–4), 279–292 (1992)zbMATHGoogle Scholar
  14. 14.
    Silver, D., et al.: Deterministic policy gradient algorithms. In: Proceedings of the 31st International Conference on Machine Learning (ICML 2014) (2014)Google Scholar
  15. 15.
    Mnih, V., et al.: Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013)

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Andrew G. Lonsberry
    • 1
    Email author
  • Alexander J. Lonsberry
    • 1
  • Roger D. Quinn
    • 1
  1. 1.Department of Mechanical and Aerospace EngineeringCase Western Reserve UniversityClevelandUSA

Personalised recommendations