Two Possible Paradoxes in Numerical Comparisons of Optimization Algorithms

  • Qunfeng Liu
  • Wei Chen
  • Yingying Cao
  • Yun LiEmail author
  • Ling WangEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10955)


Comparison strategies of benchmarking optimization algorithms are considered. Two strategies, namely “C2” and “C2+”, are defined. Existing benchmarking methods can be regarded as different applications of them. Mathematical models are developed for both “C2” and “C2+”. Based on these models, two possible paradoxes, namely the cycle ranking and the survival of the non-fittest, are deduced for three optimization algorithms’ comparison. The probabilities of these two paradoxes are calculated. It is shown that the value and the parity of the number of test problems affect the probabilities significantly. When there are only dozens of test problems, there is about 75% probability to obtain a normal ranking result for three optimization algorithms’ numerical comparison, about 9% for cycle ranking, and 16% for survival of the non-fittest.


Optimization algorithm Benchmarking Paradox Survival of the non-fittest Cycle ranking 



This work was supported by National Key R&D Program of China (No. 2016YFD0400206), NSF of China (No. 61773119) and NSF of Guangdong Province (No. 2015A030313648).


  1. 1.
    Gaviano, M., Kvasov, D., Lera, D., Sergeyev, Y.D.: Algorithm 829: software for generation of classes of test functions with known local and global minima for global optimization. ACM Trans. Math. Softw. 9, 469–480 (2003)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Hansen, N., Auger, A., Ros, R., Finck, S. and Pošík P.: Comparing results of 31 algorithms from the black-box optimization benchmarking bbob-2009. In: Proceedings of the 12th annual conference companion on genetic and evolutionary computation, pp. 1689–1696 (2010)Google Scholar
  3. 3.
    Awad, N.H., Ali, M.Z., Liang, J.J., Qu, B.Y., Suganthan, P.N.: Problem definitions and evaluation criteria for the CEC2017 special session and competition on single objective bound constrained real parameter numerical optimization. Nanyang Technological University, Singapore, Technical report, November 2016Google Scholar
  4. 4.
    Hansen, N., Auger, A., Mersmann, O., Tušar, T., Brockhoff, D.: Coco: A platform for comparing continuous optimizers in a black-box setting. ArXiv e-prints arXiv:1603.08785 (2016)
  5. 5.
    Gong, M., Wang, Z., Zhu, Z., Jiao, L.: A similarity-based multiobjective evolutionary algorithm for deployment optimization of near space communication system. IEEE Trans. Evol. Comput. 21, 878–897 (2017)CrossRefGoogle Scholar
  6. 6.
    Valle, Y., Venayagamoorthy, G.K., Mohagheghi, S., Hernandez, J.-C., Harley, R.G.: Particle swarm optimization: Basic concepts, variants and applications in power systems. Inf. Sci. 12, 171–195 (2008)Google Scholar
  7. 7.
    Wang, Y., Xu, B., Sun, G., Yang, S.: A two-phase differential evolution for uniform designs in constrained experimental domains. IEEE Trans. Evol. Comput. 21, 665–680 (2017)CrossRefGoogle Scholar
  8. 8.
    Dolan, E.D., Moŕe, J.J.: Benchmarking optimization software with performance profiles. Math. Program. 91, 201–213 (2002)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Liu, Q., Zeng, J.: Global optimization by multilevel partition. J. Glob. Optim. 61, 47–69 (2015)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Liu, Q., Zeng, J., Yang, G.: MrDIRECT: a multilevel robust DIRECT algorithm for global optimization problems. J. Glob. Optim. 62, 205–227 (2015)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Moŕe, J., Wild, S.: Benchmarking derivative-free optimization algorithms. SIAM J. Optim. 20, 172–191 (2009)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Omidvar, M.N., Yang, M., Mei, Y., Li, X., Yao, X.: Dg2: A faster and more accurate differential grouping for large-scale black-box optimization. IEEE Trans. Evol. Comput. 21, 929–942 (2017)CrossRefGoogle Scholar
  13. 13.
    Yang, M., Omidvar, M.N., Li, C., Li, X., Cai, Z., Kazimipour, B., Yao, X.: Efficient resource allocation in cooperative co-evolution for large-scale global optimization. IEEE Trans. Cybern. 21, 493–505 (2017)Google Scholar
  14. 14.
    Li, X., Yao, X.: Cooperatively coevolving particle swarms for large scale optimization. IEEE Trans. Evol. Comput. 16, 210–224 (2012)CrossRefGoogle Scholar
  15. 15.
    Qin, Q., Cheng, S., Zhang, Q., Li, L., Shi, Y.: Particle swarm optimization with interswarm interactive learning strategy. IEEE Trans. Cybern. 46, 2238–2251 (2015)CrossRefGoogle Scholar
  16. 16.
    Gong, Y.-J., Li, J.-J., Zhou, Y., Li, Y., Chung, H.S.-H., Shi, Y.-H., Zhang, J.: Genetic learning particle swarm optimization. IEEE Trans. Cybern. 46, 2277–2290 (2016)CrossRefGoogle Scholar
  17. 17.
    Yang, Q., Chen, W.-N., Gu, T., Zhang, H., Deng, J.D., Li, Y., Zhang, J.: Segment-based predominant learning swarm optimizer for large-scale optimization. IEEE Trans. Cybern. 47, 2896–2910 (2017)CrossRefGoogle Scholar
  18. 18.
    Liu, Q.: Order-2 stability analysis of particle swarm optimization. Evol. Comput. 23, 187–216 (2015)CrossRefGoogle Scholar
  19. 19.
    Liu, Q., Chen, W.-N., Deng, J.D., Gu, T., Zhang, H., Yu, Z., Zhang, J.: Benchmarking stochastic algorithms for global optimization problems by visualizing confidence intervals. IEEE Trans. Cybern. 47, 2924–2937 (2017)CrossRefGoogle Scholar
  20. 20.
    Hansen N., Auger A., Brockhoff D., Tušar D., and Tušar T.: Coco: Performance assessment. ArXiv e-prints arXiv:1605.03560 (2016)
  21. 21.
    Maassen, H., Bezembinder, T.: Generating random weak orders and the probability of a Condorcet winner. Soc. Choice Welf. 19, 517–532 (2002)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Dwork C., Kumar R., Naor M., and Sivakumar D.: Rank aggregation methods for the web. In: Proceedings of the 10th International Conference on World Wide Web, pp. 613–622. ACM (2001)Google Scholar
  23. 23.
    Cucuringu, M.: Sync-rank: Robust ranking, constrained ranking and rank aggregation via eigenvector and SDP synchronization. IEEE Trans. Netw. Sci. Eng. 3, 58–79 (2016)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Li, Y.H., Zhan, Z.-H., Lin, S.J., Zhang, J., Luo, X.N.: Competitive and cooperative particle swarm optimization with information sharing mechanism for global optimization problems. Inf. Sci. 293, 370–382 (2015)CrossRefGoogle Scholar
  25. 25.
    Chen, W.-N., Zhang, J., Lin, Y., Chen, N., Zhan, Z.-H., Chung, H.S.-H., Li, Y., Shi, Y.-H.: Particle swarm optimization with an aging leader and challengers. IEEE Trans. Evol. Comput. 17, 241–258 (2013)CrossRefGoogle Scholar
  26. 26.
    Liu, Q., Wei, W., Yuan, H., Zhan, Z.-H., Li, Y.: Topology selection for particle swarm optimization. Inf. Sci. 363, 154–173 (2016)CrossRefGoogle Scholar
  27. 27.
    Rios, L.M., Sahinidis, N.V.: Derivative-free optimization: a review of algorithms and comparison of software implementations. J. Glob. Optim. 56, 1247–1293 (2013)MathSciNetCrossRefGoogle Scholar
  28. 28.
    Paulavičius, R., Sergeyev, Y.D., Kvasov, D.E., Žlinskas, J.: Globally-biased DISIMPL algorithm for expensive global optimization. J. Glob. Optim. 59, 545–567 (2014)MathSciNetCrossRefGoogle Scholar
  29. 29.
    Deemen, A.V.: On the empirical relevance of condorcet’s paradox. Pub. Choice 158, 311–330 (2014)CrossRefGoogle Scholar
  30. 30.
    Gehrlein, W.V.: Condorcet’s Paradox. Springer, Berlin (2006)zbMATHGoogle Scholar
  31. 31.
    Wolpert, D.H., Macready, W.G.: No free lunch theorems for optimization. IEEE Trans. Evol. Comput. 1, 67–82 (1997)CrossRefGoogle Scholar
  32. 32.
    Diss, M., Gehrlein, W.V.: Borda’s Paradox and weighted scoring rules. Soc. Choice Welf. 38, 121–136 (2012)MathSciNetCrossRefGoogle Scholar
  33. 33.
    Gehrlein, W.V., Lepelley, D.: On the probability of observing Borda’s paradox. Soc. Choice Welf. 35, 1–23 (2015)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.School of Computer Science and Network SecurityDongguan University of TechnologyDongguanChina
  2. 2.Department of AutomationTsingHua UniversityBeijingChina

Personalised recommendations