Consideration of State Representation for Semi-autonomous Reinforcement Learning of Sailing Within a Navigable Area

Abstract

To sail quickly to a goal within a navigable area, complex control of the rudder and sail is required. Sailors must determine the current action with consideration of the time series of states; i.e., both current and future states. Reinforcement learning is an appropriate method for learning a complex problem, such as sailing. In this paper, we apply the navigable area such that a robotic sailor must avoid touching a boundary. To realise a higher layer of sailing architecture, the action space is simplified and discretised to the degree of the sailboat direction change. Moreover, we utilize semi-autonomous reinforcement learning, also known as imitation learning, in which a human selects an action and a robot updates its Q-values to evaluate pairs of states and actions until the robot’s action selection is equivalent to the human’s. For semi-autonomous learning, as well as for normal reinforcement learning, a representation of the state space is important. The state representation should be defined so that the state space is discretised to specify a desirable action, thereby removing any redundancy if possible. In this paper, we verify and investigate the possibility of state representation.