A reinforcement learning-based communication topology in particle swarm optimization

  • Yue Xu
  • Dechang PiEmail author
Original Article


Recently, a multitude of researchers have considered the fully connected topology (Gbest) as a default communication topology in particle swarm optimization (PSO). Despite many earlier studies of this issue indicating that the Gbest might favor unimodal problems, the topology with fewer connections, e.g., Lbest, might perform better on multimodal problems. It seems that different topologies make PSO a problem-related algorithm, while in this paper a problem-free PSO which integrates a reinforcement learning method has been proposed, referred to as QLPSO. In the new proposed algorithm, each particle acts as an agent independently, selecting the optimal topology under the control of Q-learning (QL) during each iteration. Two variants of QLPSO consider the different dimensions of the communication topology, respectively. In order to investigate the performance of QLPSO, experiments on 28 CEC 2013 benchmark functions are carried out when comparing with static and dynamic topologies. The reported computational results show that the proposed QLPSO is more superior compared with several state-of-the-art methods.


Particle swarm optimization Topology Q-learning CEC 2013 benchmark 



This work was partially supported by National Natural Science Foundation of China (U1433116), the Fundamental Research Funds for the Central Universities (NP2017208), and the Postgraduate Research & Practice Innovation Program of Jiangsu Province (KYCX19_0202).

Compliance with ethical standards

Conflict of interest

The authors declare that there is no conflict of interest with any person(s) or organization(s).


  1. 1.
    Del Ser J, Osaba E, Molina D, Yang X-S, Salcedo-Sanz S, Camacho D, Das S, Suganthan PN, Coello Coello CA, Herrera F (2019) Bio-inspired computation: where we stand and what’s next. Swarm Evolut Comput 48:220–250. CrossRefGoogle Scholar
  2. 2.
    Zhu Z, Zhou J, Zhen J, Shi YH (2011) DNA sequence compression using adaptive particle swarm optimization-based memetic algorithm. IEEE Trans Evolut Comput 15(5):643–658CrossRefGoogle Scholar
  3. 3.
    Arqub OA, Abo-Hammour Z (2014) Numerical solution of systems of second-order boundary value problems using continuous genetic algorithm. Inf Sci 279:396–415CrossRefMathSciNetzbMATHGoogle Scholar
  4. 4.
    Arqub OA (2015) Adaptation of reproducing kernel algorithm for solving fuzzy Fredholm–Volterra integrodifferential equations. Neural Comput Appl 28(7):1–20Google Scholar
  5. 5.
    Arqub OA, Maayah B (2018) Solutions of Bagley–Torvik and Painlevé equations of fractional order using iterative reproducing kernel algorithm with error estimates. Neural Comput Appl 29(5):1465–1479CrossRefGoogle Scholar
  6. 6.
    Zhang H, Llorca J, Davis CC, Milner SD (2012) Nature-inspired self-organization, control, and optimization in heterogeneous wireless networks. IEEE Trans Mob Comput 11(7):1207–1222CrossRefGoogle Scholar
  7. 7.
    Zhang H, Xiong C, Ho JKL, Chow TWS (2017) Object-level video advertising: an optimization framework. IEEE Trans Ind Inf 13(2):520–531CrossRefGoogle Scholar
  8. 8.
    Xu Y, Pi D (2019) A hybrid enhanced bat algorithm for the generalized redundancy allocation problem. Swarm Evolut Comput. CrossRefGoogle Scholar
  9. 9.
    Kennedy J, Eberhart R (2002) Particle swarm optimization. In: Icnn95-international conference on neural networksGoogle Scholar
  10. 10.
    Blackwell T, Kennedy J (2018) Impact of communication topology in particle swarm optimization. IEEE Trans Evolut Comput 23:689–702CrossRefGoogle Scholar
  11. 11.
    Abido MA (2002) Optimal power flow using particle swarm optimization. Int J Electr Power Energy Syst 24(7):563–571CrossRefGoogle Scholar
  12. 12.
    Abido AA (2001) Particle swarm optimization for multimachine power system stabilizer design. In: Power Engineering Society Summer MeetingGoogle Scholar
  13. 13.
    Ozcan E, Cad S, No TS, Mohan CK (2002) Particle swarm optimization: surfing the waves. In: Congress on evolutionary computationGoogle Scholar
  14. 14.
    Clerc M, Kennedy J (2002) The particle swarm: explosion, stability and convergence in multi-dimensional complex space. IEEE Trans Evolut Comput 20(1):1671–1676Google Scholar
  15. 15.
    Liao W, Wang J, Wang J (2006) Nonlinear inertia weight variation for dynamic adaptation in particle swarm optimization. Comput Oper Res 33(3):859–871CrossRefGoogle Scholar
  16. 16.
    Nickabadi A, Ebadzadeh MM, Safabakhsh R (2011) A novel particle swarm optimization algorithm with adaptive inertia weight. Appl Soft Comput J 11(4):3658–3670CrossRefGoogle Scholar
  17. 17.
    Valle YD, Venayagamoorthy GK, Mohagheghi S, Hernandez JC, Harley RG (2008) Particle swarm optimization: basic concepts, variants and applications in power systems. IEEE Trans Evolut Comput 12(2):171–195CrossRefGoogle Scholar
  18. 18.
    Banks A, Vincent J, Anyakoha C (2008) A review of particle swarm optimization. Part II: hybridisation, combinatorial, multicriteria and constrained optimization, and indicative applications. Nat Comput 7(1):109–124CrossRefMathSciNetzbMATHGoogle Scholar
  19. 19.
    Kao YT, Zahara E (2008) A hybrid genetic algorithm and particle swarm optimization for multimodal functions. Appl Soft Comput 8(2):849–857CrossRefGoogle Scholar
  20. 20.
    Gong YJ, Li JJ, Zhou Y, Li Y, Chung HS, Shi YH, Zhang J (2017) Genetic learning particle swarm optimization. IEEE Trans Cybern 46(10):2277–2290CrossRefGoogle Scholar
  21. 21.
    Deng L, Lu G, Shao Y, Fei M, Hu H (2016) A novel camera calibration technique based on differential evolution particle swarm optimization algorithm. Neurocomputing 174:456–465CrossRefGoogle Scholar
  22. 22.
    Li Z, Wang W, Yan Y, Li Z (2015) PS–ABC: a hybrid algorithm based on particle swarm and artificial bee colony for high-dimensional optimization problems. Expert Syst Appl 42(22):8881–8895CrossRefGoogle Scholar
  23. 23.
    Geng J, Li MW, Dong ZH, Liao YS (2015) Port throughput forecasting by MARS- R SVR with chaotic simulated annealing particle swarm optimization algorithm. Neurocomputing 147(1):239–250CrossRefGoogle Scholar
  24. 24.
    Samma H, Lim CP, Saleh JM (2016) A new reinforcement learning-based memetic particle swarm optimizer. Appl Soft Comput 43(C):276–297CrossRefGoogle Scholar
  25. 25.
    Kennedy J, Mendes R (2002) Population structure and particle swarm performance. In: Congress on evolutionary computationGoogle Scholar
  26. 26.
    Watkins CJCH, Dayan P (1992) Q-learning. Mach Learn 8(3–4):279–292zbMATHGoogle Scholar
  27. 27.
    Rakshit P, Konar A, Bhowmik P, Goswami I, Das S, Jain LC, Nagar AK (2013) Realization of an adaptive memetic algorithm using differential evolution and Q-Learning: a case study in multirobot path planning. IEEE Trans Syst Man Cybern Syst 43(4):814–831CrossRefGoogle Scholar
  28. 28.
    Samma H, Mohamad-Saleh J, Suandi SA, Lahasan B (2019) Q-learning-based simulated annealing algorithm for constrained engineering design problems. Neural Comput Appl 1:1–15Google Scholar
  29. 29.
    Feng W, Zhang H, Li K, Lin Z, Yang J, Shen X (2018) A hybrid particle swarm optimization algorithm using adaptive learning strategy. Inf Sci 436:162–177MathSciNetGoogle Scholar
  30. 30.
    Shi YH, Eberhart RC (1998) A modified particle swarm optimizer. In: The 1998 IEEE international conference on evolutionary computation proceedings, 1998. IEEE world congress on computational intelligenceGoogle Scholar
  31. 31.
    Kennedy J (1999) Small worlds and mega-minds: effects of neighborhood topology on particle swarm performance. In: Congress on evolutionary computationGoogle Scholar
  32. 32.
    Eberhart R, Kennedy J (2002) A new optimizer using particle swarm theory. In: Mhs95 sixth international symposium on micro machine & human scienceGoogle Scholar
  33. 33.
    Mendes R, Kennedy J, Neves J (2003) Watch thy neighbor or how the swarm can learn from its environment. In: Swarm intelligence symposiumGoogle Scholar
  34. 34.
    Suganthan PN (1999) Particle swarm optimiser with neighbourhood operator. In: Congress on evolutionary computationGoogle Scholar
  35. 35.
    Bonyadi MR, Li X, Michalewicz Z (2014) A hybrid particle swarm with a time-adaptive topology for constrained optimization. Swarm Evolut Comput 18(1):22–37CrossRefGoogle Scholar
  36. 36.
    Wei HL, Isa NAM (2014) Particle swarm optimization with increasing topology connectivity. Eng Appl Artif Intell 27(27):80–102Google Scholar
  37. 37.
    Marinakis Y, Marinaki M (2013) A hybridized particle swarm optimization with expanding neighborhood topology for the feature selection problem. In: Hybrid metaheuristics. 8th international workshop, HM 2013, pp 37–51Google Scholar
  38. 38.
    Goldberg DE, Richardson J (1987) Genetic algorithms with sharing for multimodal function optimization. In: International conference on genetic algorithms on genetic algorithms & their applicationGoogle Scholar
  39. 39.
    Goudos SK, Zaharis ZD, Kampitaki DG, Rekanos IT, Hilas CS (2009) Pareto optimal design of dual-band base station antenna arrays using multi-objective particle swarm optimization with fitness sharing. IEEE Trans Magn 45(3):1522–1525CrossRefGoogle Scholar
  40. 40.
    Tao L, Wei C, Pei W (2004) PSO with sharing for multimodal function optimization. In: International conference on neural networks & signal processingGoogle Scholar
  41. 41.
    Li X (2010) Niching without niching parameters: particle swarm optimization using a ring topology. IEEE Trans Evolut Comput 14(1):150–169CrossRefGoogle Scholar
  42. 42.
    Parrott D, Li X (2004) A particle swarm model for tracking multiple peaks in a dynamic environment using speciation. In: Congress on evolutionary computationGoogle Scholar
  43. 43.
    Peram T, Veeramachaneni K, Mohan CK (2012) Fitness distance, ratio based particle swarm optimization. In: Swarm intelligence symposiumGoogle Scholar
  44. 44.
    Li X (2007) A multimodal particle swarm optimizer based on fitness Euclidean-distance ratio. In: Conference on genetic & evolutionary computationGoogle Scholar
  45. 45.
    Qu BY, Suganthan PN, Das S (2013) A distance-based locally informed particle swarm model for multimodal optimization. IEEE Trans Evolut Comput 17(3):387–402CrossRefGoogle Scholar
  46. 46.
    Engelbrecht AP (2013) Particle swarm optimization: global best or local best? In: BRICS congress on computational intelligence & Brazilian congress on computational intelligenceGoogle Scholar
  47. 47.
    Shi Y, Liu H, Gao L, Zhang G (2011) Cellular particle swarm optimization. Inf Sci Int J 181(20):4460–4493MathSciNetzbMATHGoogle Scholar
  48. 48.
    Das PK, Behera HS, Panigrahi BK (2016) Intelligent-based multi-robot path planning inspired by improved classical Q-learning and improved particle swarm optimization with perturbed velocity. Eng Sci Technol Int J 19(1):651–669CrossRefGoogle Scholar
  49. 49.
    Wang H, Sun H, Li C, Rahnamayan S (2013) Diversity enhanced particle swarm optimization with neighborhood search. Inf Sci 223(2):119–135CrossRefMathSciNetGoogle Scholar
  50. 50.
    Liang JJ, Qu B-Y, Suganthan PN, Hernández-Díaz AG (2013) Problem definitions and evaluation criteria for the CEC 2013 special session and competition on real-parameter optimization. Technical Report 201212. Computational Intelligence Laboratory, Zhengzhou University, Zhengzhou China and Technical Report, Nanyang Technological University, Singapore, January 2013Google Scholar
  51. 51.
    Montgomery DC (2009) Design and analysis of experiment, 7th edn. Wiley, New YorkGoogle Scholar
  52. 52.
    Shao Z, Pi D, Shao W (2018) A novel discrete water wave optimization algorithm for blocking flow-shop scheduling problem with sequence-dependent setup times. Swarm Evolut Comput 40:53–75. CrossRefGoogle Scholar
  53. 53.
    Derrac J, García S, Molina D, Herrera F (2011) A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms. Swarm Evolut Comput 1(1):3–18CrossRefGoogle Scholar
  54. 54.
    Dor AE, Lemoine D, Clerc M, Siarry P, Deroussi L, Gourgand M (2015) Dynamic cluster in particle swarm optimization algorithm. Nat Comput 14(4):655–672CrossRefMathSciNetzbMATHGoogle Scholar
  55. 55.
    Lad BK, Kulkarni MS, Misra KB (2009) Optimal reliability design of a system. IEEE Trans Reliab R-22(5):255–258Google Scholar
  56. 56.
    Garg H, Rani M, Sharma SP (2013) An efficient two phase approach for solving reliability–redundancy allocation problem using artificial bee colony technique. Comput Oper Res 40(12):2961–2969CrossRefMathSciNetzbMATHGoogle Scholar
  57. 57.
    Hsieh YC, You PS (2011) An effective immune based two-phase approach for the optimal reliability–redundancy allocation problem. Appl Math Comput 218(4):1297–1307MathSciNetGoogle Scholar
  58. 58.
    Ouyang HB, Gao LQ, Li S, Kong XY (2015) Improved novel global harmony search with a new relaxation method for reliability optimization problems. Inf Sci 305(C):14–55CrossRefGoogle Scholar
  59. 59.
    Huang CL (2015) A particle-based simplified swarm optimization algorithm for reliability redundancy allocation problems. Reliability Engineering & System Safety 142:221–230CrossRefGoogle Scholar
  60. 60.
    Ravi VRPJ, Zimmermann HJ (2000) Fuzzy global optimization of complex system reliability. IEEE Trans Fuzzy Syst 8(3):241–248CrossRefGoogle Scholar
  61. 61.
    Ravi V, Murty BSN, Reddy PJ (1997) Nonequilibrium simulated-annealing algorithm applied to reliability optimization of complex systems. IEEE Trans Reliab 46(2):233–239CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2019

Authors and Affiliations

  1. 1.College of Computer Science and TechnologyNanjing University of Aeronautics and AstronauticsNanjingChina

Personalised recommendations