Testing, Evaluation and Performance of Optimization and Learning Systems

  • D. Whitley
  • J. P. Watson
  • A. Howe
  • L. Barbulescu


Benchmarks and test suites are widely used to evaluate optimization and learning systems. The advantage is that these test problems provide an objective means of comparing systems. The potential disadvantage is that systems can become overfitted to work well on benchmarks and therefore that good performance on benchmarks does not generalize to real world problems. The meaning and significance of benchmarks is examined in light of theoretical results such as “No Free Lunch.” The “structure” of common benchmarks is also explored.


Local Optimum Problem Instance Free Lunch Path Relinking Free Lunch Theorem 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    R. Barr, ?. Golden, J. Kelly, M Resende, and Jr. W. Stewart. Designing and Reporting on Computational Experiments with Heuristic Methods. Journal of Heuristics, 1:9–32, 1995.MATHCrossRefGoogle Scholar
  2. 2.
    K. D. Boese, A. B. Kahng, and S. Muddu. A new adaptive multi-start technique for combinatorial global optimizations. Operations Research Letters, 16/2:101–113, 1994.MathSciNetCrossRefGoogle Scholar
  3. 3.
    T. Cormen, C. Leiserson, and R. Rivest. Introduction to Algorithms. McGraw Hill, New York, 1990.MATHGoogle Scholar
  4. 4.
    J. Culberson. On the Futility of Blind Search. Evolutionary Computation, 6(2): 109–127, 1999.CrossRefGoogle Scholar
  5. 5.
    K. De Jong, M. Potter, and Wm. Spears. Using Problem Generators to Explore the Effects of Epistasis. In T. Bäck, editor, Proc. of the 7th Int’l. Conf. on GAs, pages 338–345. Morgan Kaufmann, 1998.Google Scholar
  6. 6.
    J.N. Hooker. Testing Heuristics: We Have it All Wrong. Journal of Heuristics, 1:33–42, 1995.MATHCrossRefGoogle Scholar
  7. 7.
    E. Horowitz and S. Sahni. Fundamentals of Computer Algorithms. Computer Science Press, 1978.Google Scholar
  8. 8.
    Alexander R. Kan. Machine Scheduling Problems: Classification, complexity and computations. Martinus Nijhoff, The Hague, 1976.Google Scholar
  9. 9.
    David Mitchell, Bart Selman, and Hector Levesque. Hard and easy distribution of sat problems. In Proceedings of the Tenth National Conference on Artificial Intelligence, San Jose, CA, 1992.Google Scholar
  10. 10.
    Tom Mitchell. Machine Learning. McGraw Hill, 1997.Google Scholar
  11. 11.
    Christos H. Papadimitriou. Computational Complexity. Addison-Wesley Publishing, Co., 1994.Google Scholar
  12. 12.
    C.R. Reeves and T. Yamada. Genetic Algorithms, Path Relinking, and the Flowshop Sequencing Problem. Journal of Evolutionary Computation, 6(1):45–60, 1998.CrossRefGoogle Scholar
  13. 13.
    C. Schumacher, M. Vose, and D. Whitley. The No Free Lunch and Problem Description Length. In Genetic and Evolutionary Computation Conference GECCO-00, pages 565–570. Morgan Kaufmann, 2001.Google Scholar
  14. 14.
    ?. Selman, ?. Kautz, and B. Cohen. Local search strategies for satisfiability testing. In Trick and Johnson, editors, Second DIM ACS Challenge on Cliques, Coloring and Satisfiability, 1993.Google Scholar
  15. 15.
    E. Taillard. Some efficient heuristic methods for the flow shop sequencing problem. European Journal of Operations Research, 47:65–74, 1990.MathSciNetMATHCrossRefGoogle Scholar
  16. 16.
    E. Taillard. Benchmarks for basic scheduling problems. European Journal of Operations Research, 64:278–285, 1993.MATHCrossRefGoogle Scholar
  17. 17.
    J.P. Watson, L. Barbulescu, D. Whitley, and A. Howe. Artificial test suites for flowshop scheduling and algorithm performance. In Proceedings of the Sixteenth National Conference on Artificial Intelligence, 1999.Google Scholar
  18. 18.
    J.P. Watson, L. Barbulescu, D. Whitley, and A. Howe. Contrasting Structured and Random Permutation Flow-Shop Scheduling Problems. INFORMS Journal on Computing, 2002.Google Scholar
  19. 19.
    D. Whitley. Functions as Permutations: Regarding No Free Lunch, Walsh Analysis and Summary Statistics. In Schoenauer, Deb, Rudolph, Lutton, Merelo, and Schwefel, editors, Parallel Problem Solving from Nature, 6, pages 169–178. Springer, 2000.Google Scholar
  20. 20.
    David H. Wolpert and William G. Macready. No free lunch theorems for search. Technical Report SFI-TR-95-02-010, Santa Fe Institute, July 1995.Google Scholar
  21. 21.
    David H. Wolpert and William G. Macready. No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation, 4:67–82, 1997.CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London 2002

Authors and Affiliations

  • D. Whitley
    • 1
  • J. P. Watson
    • 1
  • A. Howe
    • 1
  • L. Barbulescu
    • 1
  1. 1.Colorado State UniversityUSA

Personalised recommendations