Experimental Analysis of Algorithms

  • Catherine C. McGeoch
Part of the Nonconvex Optimization and Its Applications book series (NOIA, volume 62)


This chapter surveys methodological issues that arise in experimental research on optimization algorithms. Guidelines are presented for selecting research problems, input classes, and program metrics; for ensuring correctness and reproducibility of the results; for applying appropriate data analysis techniques and presenting conclusions.


Solution Quality Data Analysis Technique Input Property Wall Clock Time Input Instance 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Ahuja, R.K., Magnanti, T.L., and Orlin, J.B. (1993). Network Flows: Theory, Algorithms and Applications. Prentice Hall. Chapter 18: Computational Testing of Algorthms.Google Scholar
  2. Alizadeh, F. and Goldberg, A.V. (1993). Implementing the push-relabel method for the maximum flow problem on a connection machine. In Network Flows and Matching: First DIMACS Implementation Challenge. American Mathematical Society. DIMACS Series in Discrete Mathematics and Theoretical Computer Science, Volume 12.Google Scholar
  3. Anderson, R.J. (1997). The role of experiment in the theory of algorithms. Position paper, to appear in Proceedings of the Fifth DIMACS Challenge. Available at Scholar
  4. Barr, R.S., Golden, B.L., Kelly, J.P., Resende, M.G.C., and Stewart, W.R. (1995). Designing and reporting on computational experiments with heuristic methods. Journal of Heuristics, 1 (1): 9–32.zbMATHCrossRefGoogle Scholar
  5. Beardwood, J., Halton, J.H., and Hammersley, J.M. (1959). The shortest path through many points. Proceedings of the Cambridge Philosophical Society, 55:299–327.Google Scholar
  6. Bentley, J.L. (1982). Writing Efficient Programs. Prentice-Hall, Englewood Cliffs, New Jersey.Google Scholar
  7. Bentley, J.L. (1991). Tools for experiments on algorithms. In Rashid, R.F., editor, CMU Computer Science: A 25 ’ Anniversary Commermorative,pages 99–123. ACM Press.Google Scholar
  8. Bentley, J.L., Johnson, D.S., Leighton, T., and McGeoch, C.C. (1983). An experimental study of bin packing. In Proceedings of 21S t Annual Allerton Conference on Communication, Control, and Computing, pages 51–60.Google Scholar
  9. Box, G.E.P., Hunter, W.G., and Hunter, J.S. (1978). Statistics for Experimenters. John Wiley and Sons.Google Scholar
  10. Bratley, P., Fox, B.L., and Schrage, L.E. (1983). A Guide to Simulation. Springer-Verlag.Google Scholar
  11. Chambers, J.M., Cleveland, W.S., Kleiner, B., and Tukey, P.A. (1983). Graphical Methods for Data Analysis. Duxbury Press, Boston.zbMATHGoogle Scholar
  12. Cheng, R.C.H. (1982). The use of antithetic variates in computer simulation. Journal of the Operational Research Society, 33: 229–237.zbMATHGoogle Scholar
  13. Cleveland, W.S. (1993). Visualizing Data. Hobart Press, Summit, New Jersey.Google Scholar
  14. Coffman, Jr., E.G. and Lueker, G.S. (1991). Probabilistic Analysis of Packing and Partitioning Algorithms. Series in Discrete Mathematics and Optimization. Wiley-Interscience.Google Scholar
  15. Cohen, P.R. (1995). Empirical Methods for Artificial Intelligence. The MIT Press, Cambridge, Massachusetts and London, England.Google Scholar
  16. Crowder, H.P., Dembo, R.S., and Mulvey, J.M. (1978). Reporting computational experiments in mathematical programming. Mathematical Programming, 15: 316–329.MathSciNetzbMATHCrossRefGoogle Scholar
  17. Fishman, G.S. and Rubin, D.S. (1992). Bounding the variance in Monte Carlo experiments. Operations Research Letters, 11 (243–248).Google Scholar
  18. Fishman, G.S. and Yarberry, L.S. (1997). An implementation of the batch means method. INFORMS Journal on Computing,9(3):296310.Google Scholar
  19. Fredman, M.L., Johnson, D.S., McGeoch, L.A., and Ostheihmer, G. (1995). Data structures for travelling salesmen. Journal of Algorithms, 18 (3): 432–479.MathSciNetzbMATHCrossRefGoogle Scholar
  20. Golden, B.L., Assad, A.A., Wasil, E.A., and Baker, E. (1986). Experimentation in optimization. European Journal of Operational Research, 27: 1–16.MathSciNetzbMATHCrossRefGoogle Scholar
  21. Golden, B.L. and Stewart, W.R. (1985). Empirical analysis of algorithms. In Lawler, E.L., Lenstra, J.K., Kan, A.H.G. Rinnooy, and Shmoys, D.B., editors, The Traveling Salesman Problem: a guided tour of combinatorial optimization,chapter 7, pages 207–249. Wiley Interscience.Google Scholar
  22. Greenberg, H.J. (1990). Computational testing: Why, how, and how much. ORSA Journal on Computing, 2 (1): 94–97.CrossRefGoogle Scholar
  23. Higle, J.L. (1998). Variance reduction and objective function evaluation in stochastic linear programs. INFORMS Journal on Computing, 10 (2): 236–247.MathSciNetzbMATHCrossRefGoogle Scholar
  24. Hooker, J.N. (1994). Needed: an empirical science of algorithms. Operations Research, 42: 201–212.zbMATHCrossRefGoogle Scholar
  25. Hooker, J.N. (1996). Testing heuristics: We have it all wrong. Journal of Heuristics, 1: 33–42.CrossRefGoogle Scholar
  26. Jackson, R.H.F., Boggs, P.T., Nash, S.G., and Powell, S. (1991). Guidelines for reporting results of computational experiments. report of the ad hoc committee. Mathematical Programming, 49: 413–425.MathSciNetCrossRefGoogle Scholar
  27. Johnson, D.S. (1996). A theoretician’s guide to the experimental analysis of algorithms. To appear in Proceedings of the Fifth DIMACS Challenge. Available at“dsj/papers/ Scholar
  28. Johnson, D.S. and McGeoch, C.C., editors (1991). Network Flows and Matching: Proceedings of the First DIMACS Implementation Challenge,volume 12 of DIMACS Series in Discrete Mathematics and Theoretical Computer Science. American Mathematical Society.Google Scholar
  29. Johnson, D.S. and 11–1cGeoch, L.A. (1997). The traveling salesman problem: a case study. In Aarts, E. and Lenstra, J.K., editors, Local Search in Combinatorial Optimization, chapter 8. Wiley-Interscience. Series in Discrete Mathematics and Optimization.Google Scholar
  30. Johnson, D.S., McGeoch, L.A., and Rothberg, E.E. (1996). Asymptotic experimental analysis for the Held-Karp traveling salesman bound. In Proceedings of the 7 th Annual ACM-SIAM Symposium on Discrete Algorithms, pages 341–350.Google Scholar
  31. Johnson, D.S. and Trick, M.A., editors (1993). Cliques, Coloring and Satisfiability: Proceedings of the Second DIMACS Implementation Challenge,volume 26 of DIMACS Series in Discrete Mathematics and Theoretical Computer Science. American Mathematical Society.Google Scholar
  32. Knuth, D.E. (1993). The Stanford GraphBase. ACM Press.Google Scholar
  33. LaMarca, A. and Ladner, R.E. (1996). The influence of caches on the performance of heaps. ACM Journal of Experimental Algorithmics, 1.
  34. McGeoch, C.C. (1986). Experimental analysis of algorithms. Technical Report CMU-CS-87–124, Carnegie Mellon University, Department of Computer Science. Ph.D. Thesis.Google Scholar
  35. McGeoch, C.C. (1992). Analyzing algorithms by simulation: Variance reduction techniques and simulation speedups. ACM Computing Surveys, 245 (2): 195–212.CrossRefGoogle Scholar
  36. McGeoch, C.C. (1996). Toward an experimental method for algorithm simulation. INFORMS Journal on Computing, 1 (1): 1–15.MathSciNetCrossRefGoogle Scholar
  37. McGeoch, C.C. (2001). Experimental analysis of optimization algorithms. In Pardalos, P.M. and Resende, M.G.C., editors, Handbook of Applied Optimization. Oxford University Press.Google Scholar
  38. McGeoch, C.C. and Moret, B.M.E. (1999). How to present a paper on experimental work with algorithms. SIGACT News, 30 (4).Google Scholar
  39. McGeoch, C.C., Sanders, P., Fleischer, R., Cohen, P., and Precup, D. (2001). Searching for big-oh in the data: Inferring asymptotic complexity from experiments. To appear in a volume of Lecture Notes in Computer Science, Springer-Verlag (an early version appeared as “How to find Big-Oh in your data (and how not to)” by C.C. Mc-Google Scholar
  40. Geoch, P. Cohen, and D. Precup, Second International Symposium on Intelligent Data Analysis (IDA-97), London, August 1997 ). Tukey, J.W. (1977). Exploratory Data Analysis. Addison-Wesley.Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2002

Authors and Affiliations

  • Catherine C. McGeoch
    • 1
  1. 1.Department of Mathematics and Computer ScienceAmherst CollegeAmherstUSA

Personalised recommendations