Abstract
Comparison strategies of benchmarking optimization algorithms are considered. Two strategies, namely “C2” and “C2+”, are defined. Existing benchmarking methods can be regarded as different applications of them. Mathematical models are developed for both “C2” and “C2+”. Based on these models, two possible paradoxes, namely the cycle ranking and the survival of the non-fittest, are deduced for three optimization algorithms’ comparison. The probabilities of these two paradoxes are calculated. It is shown that the value and the parity of the number of test problems affect the probabilities significantly. When there are only dozens of test problems, there is about 75% probability to obtain a normal ranking result for three optimization algorithms’ numerical comparison, about 9% for cycle ranking, and 16% for survival of the non-fittest.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Gaviano, M., Kvasov, D., Lera, D., Sergeyev, Y.D.: Algorithm 829: software for generation of classes of test functions with known local and global minima for global optimization. ACM Trans. Math. Softw. 9, 469–480 (2003)
Hansen, N., Auger, A., Ros, R., Finck, S. and Pošík P.: Comparing results of 31 algorithms from the black-box optimization benchmarking bbob-2009. In: Proceedings of the 12th annual conference companion on genetic and evolutionary computation, pp. 1689–1696 (2010)
Awad, N.H., Ali, M.Z., Liang, J.J., Qu, B.Y., Suganthan, P.N.: Problem definitions and evaluation criteria for the CEC2017 special session and competition on single objective bound constrained real parameter numerical optimization. Nanyang Technological University, Singapore, Technical report, November 2016
Hansen, N., Auger, A., Mersmann, O., Tušar, T., Brockhoff, D.: Coco: A platform for comparing continuous optimizers in a black-box setting. ArXiv e-prints arXiv:1603.08785 (2016)
Gong, M., Wang, Z., Zhu, Z., Jiao, L.: A similarity-based multiobjective evolutionary algorithm for deployment optimization of near space communication system. IEEE Trans. Evol. Comput. 21, 878–897 (2017)
Valle, Y., Venayagamoorthy, G.K., Mohagheghi, S., Hernandez, J.-C., Harley, R.G.: Particle swarm optimization: Basic concepts, variants and applications in power systems. Inf. Sci. 12, 171–195 (2008)
Wang, Y., Xu, B., Sun, G., Yang, S.: A two-phase differential evolution for uniform designs in constrained experimental domains. IEEE Trans. Evol. Comput. 21, 665–680 (2017)
Dolan, E.D., Moŕe, J.J.: Benchmarking optimization software with performance profiles. Math. Program. 91, 201–213 (2002)
Liu, Q., Zeng, J.: Global optimization by multilevel partition. J. Glob. Optim. 61, 47–69 (2015)
Liu, Q., Zeng, J., Yang, G.: MrDIRECT: a multilevel robust DIRECT algorithm for global optimization problems. J. Glob. Optim. 62, 205–227 (2015)
Moŕe, J., Wild, S.: Benchmarking derivative-free optimization algorithms. SIAM J. Optim. 20, 172–191 (2009)
Omidvar, M.N., Yang, M., Mei, Y., Li, X., Yao, X.: Dg2: A faster and more accurate differential grouping for large-scale black-box optimization. IEEE Trans. Evol. Comput. 21, 929–942 (2017)
Yang, M., Omidvar, M.N., Li, C., Li, X., Cai, Z., Kazimipour, B., Yao, X.: Efficient resource allocation in cooperative co-evolution for large-scale global optimization. IEEE Trans. Cybern. 21, 493–505 (2017)
Li, X., Yao, X.: Cooperatively coevolving particle swarms for large scale optimization. IEEE Trans. Evol. Comput. 16, 210–224 (2012)
Qin, Q., Cheng, S., Zhang, Q., Li, L., Shi, Y.: Particle swarm optimization with interswarm interactive learning strategy. IEEE Trans. Cybern. 46, 2238–2251 (2015)
Gong, Y.-J., Li, J.-J., Zhou, Y., Li, Y., Chung, H.S.-H., Shi, Y.-H., Zhang, J.: Genetic learning particle swarm optimization. IEEE Trans. Cybern. 46, 2277–2290 (2016)
Yang, Q., Chen, W.-N., Gu, T., Zhang, H., Deng, J.D., Li, Y., Zhang, J.: Segment-based predominant learning swarm optimizer for large-scale optimization. IEEE Trans. Cybern. 47, 2896–2910 (2017)
Liu, Q.: Order-2 stability analysis of particle swarm optimization. Evol. Comput. 23, 187–216 (2015)
Liu, Q., Chen, W.-N., Deng, J.D., Gu, T., Zhang, H., Yu, Z., Zhang, J.: Benchmarking stochastic algorithms for global optimization problems by visualizing confidence intervals. IEEE Trans. Cybern. 47, 2924–2937 (2017)
Hansen N., Auger A., Brockhoff D., Tušar D., and Tušar T.: Coco: Performance assessment. ArXiv e-prints arXiv:1605.03560 (2016)
Maassen, H., Bezembinder, T.: Generating random weak orders and the probability of a Condorcet winner. Soc. Choice Welf. 19, 517–532 (2002)
Dwork C., Kumar R., Naor M., and Sivakumar D.: Rank aggregation methods for the web. In: Proceedings of the 10th International Conference on World Wide Web, pp. 613–622. ACM (2001)
Cucuringu, M.: Sync-rank: Robust ranking, constrained ranking and rank aggregation via eigenvector and SDP synchronization. IEEE Trans. Netw. Sci. Eng. 3, 58–79 (2016)
Li, Y.H., Zhan, Z.-H., Lin, S.J., Zhang, J., Luo, X.N.: Competitive and cooperative particle swarm optimization with information sharing mechanism for global optimization problems. Inf. Sci. 293, 370–382 (2015)
Chen, W.-N., Zhang, J., Lin, Y., Chen, N., Zhan, Z.-H., Chung, H.S.-H., Li, Y., Shi, Y.-H.: Particle swarm optimization with an aging leader and challengers. IEEE Trans. Evol. Comput. 17, 241–258 (2013)
Liu, Q., Wei, W., Yuan, H., Zhan, Z.-H., Li, Y.: Topology selection for particle swarm optimization. Inf. Sci. 363, 154–173 (2016)
Rios, L.M., Sahinidis, N.V.: Derivative-free optimization: a review of algorithms and comparison of software implementations. J. Glob. Optim. 56, 1247–1293 (2013)
Paulavičius, R., Sergeyev, Y.D., Kvasov, D.E., Žlinskas, J.: Globally-biased DISIMPL algorithm for expensive global optimization. J. Glob. Optim. 59, 545–567 (2014)
Deemen, A.V.: On the empirical relevance of condorcet’s paradox. Pub. Choice 158, 311–330 (2014)
Gehrlein, W.V.: Condorcet’s Paradox. Springer, Berlin (2006)
Wolpert, D.H., Macready, W.G.: No free lunch theorems for optimization. IEEE Trans. Evol. Comput. 1, 67–82 (1997)
Diss, M., Gehrlein, W.V.: Borda’s Paradox and weighted scoring rules. Soc. Choice Welf. 38, 121–136 (2012)
Gehrlein, W.V., Lepelley, D.: On the probability of observing Borda’s paradox. Soc. Choice Welf. 35, 1–23 (2015)
Acknowledgment
This work was supported by National Key R&D Program of China (No. 2016YFD0400206), NSF of China (No. 61773119) and NSF of Guangdong Province (No. 2015A030313648).
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Liu, Q., Chen, W., Cao, Y., Li, Y., Wang, L. (2018). Two Possible Paradoxes in Numerical Comparisons of Optimization Algorithms. In: Huang, DS., Jo, KH., Zhang, XL. (eds) Intelligent Computing Theories and Application. ICIC 2018. Lecture Notes in Computer Science(), vol 10955. Springer, Cham. https://doi.org/10.1007/978-3-319-95933-7_77
Download citation
DOI: https://doi.org/10.1007/978-3-319-95933-7_77
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-95932-0
Online ISBN: 978-3-319-95933-7
eBook Packages: Computer ScienceComputer Science (R0)