Deep Dynamic Programming: Optimal Control with Continuous Model Learning of a Nonlinear Muscle Actuated Arm

Abstract

We outline a new technique for on-line continuous model learning control and demonstrate its utility by controlling a simulated 2-DOF arm actuated by 6 muscles as well as on an inverted pendulum. Work presented is part of an effort to develop controllers for human appendages rendered inoperable by paralysis. Computerized control provides an alternative to neural regeneration by means of electric muscle stimulation. It has been demonstrated that paralyzed individuals can regain self-powered mobility via use of external muscle controllers. A barrier to proliferation of the technology, is the difficulty in control over the living system which is highly nonlinear and unique to each individual. Here we demonstrate a novel, continuous model learning technique to simultaneously learn and control continuous, non-linear systems. The technique expands upon vanilla Q-learning and dynamic programming. Unlike typical Q-learning, where the action-value function updates are only for the most recent set of states visited and stored in memory, the method presented also generates updates to the action-value function for unvisited state-space and state-space visited far in the past. This is made feasible by giving the agent the ability to continually learn and update explicit local models of the environment and of itself, which we encapsulated in a set of deep neural networks.