Evaluating Adaptive Paired Comparison Experiments



Paired comparison experiments are effective tools when the characteristics of the objects cannot be measured directly. In paired comparison experiments the characteristics of the objects are estimated from the result of the comparisons. The concept of paired comparison experiments was introduced by Thurstone (1927). The method by Scheffé (1952) is widely used for complete paired comparison experiments and the method by Bradley and Terry (1952) is popularly used in incomplete paired comparison experiments.

In incomplete paired comparison experiments, the design of the experiment, that is, how to form the pairs to be compared, is crucial to successful analysis. Many methods including adaptive experimental designs are proposed. The tournament systems in sports and other competitions are typical examples of such designs, but their statistical properties are not fully investigated. In this paper, we discuss how tournament systems may be evaluated and propose a new criterion. We also give examples of evaluating tournaments based on the proposed criterion.


Pairing System Strong Player Paired Comparison Method Round Robin Tournament Winning Percentage 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bradley RA, Terry ME (1952) Rank Analysis of Incomplete Block Designs. Biometrika 39:324-345MATHMathSciNetGoogle Scholar
  2. 2.
    Elo AE (1978) The Rating of Chessplayers Past and Present (second edition). ARCOGoogle Scholar
  3. 3.
  4. 4.
    Fujino Y, Takeuchi K (1988) Mathematical Science in Sports (in Japanese), Kyoritsu PublishingGoogle Scholar
  5. 5.
    Glickman ME (2005) Adaptive paired comparison design. Journal of statistical planning and inference 127:279-293MATHCrossRefMathSciNetGoogle Scholar
  6. 6.
    Hashimoto T, Nagashima J, Iida H (2002) A proposal of tournament system verification by the simulation –a case study using the World Computer Shogi Championship– (in Japanese). Proc. The 7th Game Programming Workshop, pp 101-108Google Scholar
  7. 7.
    ISO 5492:1992 (1992) Sensory Analysis –Vocabulary. ISOGoogle Scholar
  8. 8.
    Japan Chess Association. http://www.jca-chess.com/
  9. 9.
    Kawai T, Kawashima S, Iida H (2005) A new ranking measurement based on tournament records (in Japanese). IPSJ SIG Technical Report (2005-GI-13), pp 27-34Google Scholar
  10. 10.
    Kayama K (2003) Evaluation of Pairing Systems on Swiss-system Tournament by Simulation (in Japanese). Proc. The 8th Game Programming WorkshopGoogle Scholar
  11. 11.
    Marchand E (2002) On the Comparison between Standard and Random Knockout Tournament. The Statistician 51:169-178MathSciNetGoogle Scholar
  12. 12.
    McGarry T, Schutz RW (1997) Efficacy of traditional sport tournament structures. Journal of the Operational Research Society 48:65-74Google Scholar
  13. 13.
    Nozaki Y, Minakawa H, Suzuki T (2006) A Study on Ranking Participants in Competitions (in Japanese). Proc. The 36th JSQC Annual Congress, pp 187-190Google Scholar
  14. 14.
    Nozaki Y, Minakawa H, Suzuki T (2007) Experimental Design for Adaptive Paired Comparisons. Proc. The 5th ANQ Congress, 10 pages (CD-ROM)Google Scholar
  15. 15.
    Sakoda Y, Suzuki T (2005) A Study on Sports Tournament Systems. Proc. The 19th Asia Quality SymposiumGoogle Scholar
  16. 16.
    Scheffé H (1952) An Analysis of Variance for Paired Comparisons. Journal of American Statistical Association 147:381-400CrossRefGoogle Scholar
  17. 17.
  18. 18.
    Swiss Perfect Home Page. http://www.swissperfect.com/
  19. 19.
    Takizawa T, Kakinoki Y (2002) A Pairing System and Its Effectiveness in the World Computer Shogi Championships (in Japanese). Proc. The 7th Game Programming Workshop, pp 93-100Google Scholar
  20. 20.
    Thurstone LL (1927) A Law of Comparative Judgment. Psychological Review 34:278-286Google Scholar

Copyright information

© Physica-Verlag Heidelberg 2010

Authors and Affiliations

  • Tomomichi Suzuki
    • 1
  • Seiichi Yasui
    • 1
  • Yoshikazu Ojima
    • 1
  1. 1.Department of Industrial AdministrationTokyo University of ScienceChibaJAPAN

Personalised recommendations