Trade-Off between Exploration and Exploitation

You could be reading this or you could be writing that long overdue article. The tradeoff is a natural one and it is the foundation of one of the oldest problems in animal evolution: How should an organism choose between exploiting a resource or exploring to find new resources? This problem is known as the exploration versus exploitation trade-off (EvE) and it cuts across domains – in decision research, visual attention, animal foraging, memory search, and computer science, the same basic problem repeatedly surfaces.

The diversity of situations in which this trade-off arises has, in turn, led to a diversity of solutions. In reinforcement learning, rules that achieve appropriate learning rates have elements of exploration intermixed with longer periods of exploitation (Sutton and Barto 1998). Rules such as the epsilon-greedy algorithminvolve a greedy exploitative component that always chooses options with the highest known payoffs intermixed with some small (epsilon) amount of random...