Background and Problem Specification

Abstract

This chapter presents background on sequential decision making and reinforcement learning as well as the specification of the problems that this book is addressing. I begin by presenting a formal description of sequential decision making problems as Markov Decision Processes. Then I describe the reinforcement learning problem. Next, I explain the difference between model-free and model-based approaches and present example algorithms of each class. I present details on using model-based RL in factored domains. In Section 2.2.4, I present an important aspect of model-based RL, planning, along with the uct planning algorithm. In the next section, I formally define the class of domains this book is focused on: time-constrained domains where learning in very few samples is critical. Finally, I present a specific example of a domain from this class and demonstrate how each of the RL for Robotics Challenges are present in this domain.

This chapter contains material from two publications: (Hester and Stone, 2011, 2012b).