Self-Play Experiments Revisited

Summary

This chapter re-assesses the results of self-play experiments which feature handicaps in search depth, speed, or time between otherwise identical program versions. These experiments allow to study the relation of search effort and playing strength in game-playing programs. We formulate a mathematical framework for the statistical analysis of such experiments that only requires the winning percentages and the numbers of games to be known. The framework covers both direct self-play and round-robin (self-) play. We re-analyze the published results of renowned self-play experiments in computer chess, computer checkers, and computer Othello by means of our framework.

Our analyses show that no experiment provided confident empirical evidence demonstrating the existence of diminishing returns for additional search in computer self-play. The findings of our analyses prove all past claims by other researchers wrong who explicitly stated the contrary. In particular, our results contradict the recently published allegations of an obvious “chess anomaly” because the statistically confident conclusions of the experiments are identical for chess, checkers, and Othello alike. Based on the analyses of hypothetical match results we conjecture that 400 games per program version are necessary to demonstrate diminishing returns for additional search in computer self-play. Similarly, at least 1000 games per program version are required to quantify this widely expected yet still unproven phenomenon with good confidence.

The computer-chess related sections of the chapter are accepted for publication as “Self-Play Experiments in Computer Chess Revisited” in the proceedings of the 9th Conference on Advances in Computer Chess held in Paderborn, June 1999.