Two Possible Paradoxes in Numerical Comparisons of Optimization Algorithms
Comparison strategies of benchmarking optimization algorithms are considered. Two strategies, namely “C2” and “C2+”, are defined. Existing benchmarking methods can be regarded as different applications of them. Mathematical models are developed for both “C2” and “C2+”. Based on these models, two possible paradoxes, namely the cycle ranking and the survival of the non-fittest, are deduced for three optimization algorithms’ comparison. The probabilities of these two paradoxes are calculated. It is shown that the value and the parity of the number of test problems affect the probabilities significantly. When there are only dozens of test problems, there is about 75% probability to obtain a normal ranking result for three optimization algorithms’ numerical comparison, about 9% for cycle ranking, and 16% for survival of the non-fittest.
KeywordsOptimization algorithm Benchmarking Paradox Survival of the non-fittest Cycle ranking
This work was supported by National Key R&D Program of China (No. 2016YFD0400206), NSF of China (No. 61773119) and NSF of Guangdong Province (No. 2015A030313648).
- 2.Hansen, N., Auger, A., Ros, R., Finck, S. and Pošík P.: Comparing results of 31 algorithms from the black-box optimization benchmarking bbob-2009. In: Proceedings of the 12th annual conference companion on genetic and evolutionary computation, pp. 1689–1696 (2010)Google Scholar
- 3.Awad, N.H., Ali, M.Z., Liang, J.J., Qu, B.Y., Suganthan, P.N.: Problem definitions and evaluation criteria for the CEC2017 special session and competition on single objective bound constrained real parameter numerical optimization. Nanyang Technological University, Singapore, Technical report, November 2016Google Scholar
- 4.Hansen, N., Auger, A., Mersmann, O., Tušar, T., Brockhoff, D.: Coco: A platform for comparing continuous optimizers in a black-box setting. ArXiv e-prints arXiv:1603.08785 (2016)
- 6.Valle, Y., Venayagamoorthy, G.K., Mohagheghi, S., Hernandez, J.-C., Harley, R.G.: Particle swarm optimization: Basic concepts, variants and applications in power systems. Inf. Sci. 12, 171–195 (2008)Google Scholar
- 13.Yang, M., Omidvar, M.N., Li, C., Li, X., Cai, Z., Kazimipour, B., Yao, X.: Efficient resource allocation in cooperative co-evolution for large-scale global optimization. IEEE Trans. Cybern. 21, 493–505 (2017)Google Scholar
- 20.Hansen N., Auger A., Brockhoff D., Tušar D., and Tušar T.: Coco: Performance assessment. ArXiv e-prints arXiv:1605.03560 (2016)
- 22.Dwork C., Kumar R., Naor M., and Sivakumar D.: Rank aggregation methods for the web. In: Proceedings of the 10th International Conference on World Wide Web, pp. 613–622. ACM (2001)Google Scholar