Testing, Evaluation and Performance of Optimization and Learning Systems
Benchmarks and test suites are widely used to evaluate optimization and learning systems. The advantage is that these test problems provide an objective means of comparing systems. The potential disadvantage is that systems can become overfitted to work well on benchmarks and therefore that good performance on benchmarks does not generalize to real world problems. The meaning and significance of benchmarks is examined in light of theoretical results such as “No Free Lunch.” The “structure” of common benchmarks is also explored.
KeywordsLocal Optimum Problem Instance Free Lunch Path Relinking Free Lunch Theorem
Unable to display preview. Download preview PDF.
- 5.K. De Jong, M. Potter, and Wm. Spears. Using Problem Generators to Explore the Effects of Epistasis. In T. Bäck, editor, Proc. of the 7th Int’l. Conf. on GAs, pages 338–345. Morgan Kaufmann, 1998.Google Scholar
- 7.E. Horowitz and S. Sahni. Fundamentals of Computer Algorithms. Computer Science Press, 1978.Google Scholar
- 8.Alexander R. Kan. Machine Scheduling Problems: Classification, complexity and computations. Martinus Nijhoff, The Hague, 1976.Google Scholar
- 9.David Mitchell, Bart Selman, and Hector Levesque. Hard and easy distribution of sat problems. In Proceedings of the Tenth National Conference on Artificial Intelligence, San Jose, CA, 1992.Google Scholar
- 10.Tom Mitchell. Machine Learning. McGraw Hill, 1997.Google Scholar
- 11.Christos H. Papadimitriou. Computational Complexity. Addison-Wesley Publishing, Co., 1994.Google Scholar
- 13.C. Schumacher, M. Vose, and D. Whitley. The No Free Lunch and Problem Description Length. In Genetic and Evolutionary Computation Conference GECCO-00, pages 565–570. Morgan Kaufmann, 2001.Google Scholar
- 14.?. Selman, ?. Kautz, and B. Cohen. Local search strategies for satisfiability testing. In Trick and Johnson, editors, Second DIM ACS Challenge on Cliques, Coloring and Satisfiability, 1993.Google Scholar
- 17.J.P. Watson, L. Barbulescu, D. Whitley, and A. Howe. Artificial test suites for flowshop scheduling and algorithm performance. In Proceedings of the Sixteenth National Conference on Artificial Intelligence, 1999.Google Scholar
- 18.J.P. Watson, L. Barbulescu, D. Whitley, and A. Howe. Contrasting Structured and Random Permutation Flow-Shop Scheduling Problems. INFORMS Journal on Computing, 2002.Google Scholar
- 19.D. Whitley. Functions as Permutations: Regarding No Free Lunch, Walsh Analysis and Summary Statistics. In Schoenauer, Deb, Rudolph, Lutton, Merelo, and Schwefel, editors, Parallel Problem Solving from Nature, 6, pages 169–178. Springer, 2000.Google Scholar
- 20.David H. Wolpert and William G. Macready. No free lunch theorems for search. Technical Report SFI-TR-95-02-010, Santa Fe Institute, July 1995.Google Scholar