Skip to main content
SpringerLink
Log in

A reinforcement learning-based communication topology in particle swarm optimization

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Recently, a multitude of researchers have considered the fully connected topology (Gbest) as a default communication topology in particle swarm optimization (PSO). Despite many earlier studies of this issue indicating that the Gbest might favor unimodal problems, the topology with fewer connections, e.g., Lbest, might perform better on multimodal problems. It seems that different topologies make PSO a problem-related algorithm, while in this paper a problem-free PSO which integrates a reinforcement learning method has been proposed, referred to as QLPSO. In the new proposed algorithm, each particle acts as an agent independently, selecting the optimal topology under the control of Q-learning (QL) during each iteration. Two variants of QLPSO consider the different dimensions of the communication topology, respectively. In order to investigate the performance of QLPSO, experiments on 28 CEC 2013 benchmark functions are carried out when comparing with static and dynamic topologies. The reported computational results show that the proposed QLPSO is more superior compared with several state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Del Ser J, Osaba E, Molina D, Yang X-S, Salcedo-Sanz S, Camacho D, Das S, Suganthan PN, Coello Coello CA, Herrera F (2019) Bio-inspired computation: where we stand and what’s next. Swarm Evolut Comput 48:220–250. https://doi.org/10.1016/j.swevo.2019.04.008

    Article  Google Scholar 

  2. Zhu Z, Zhou J, Zhen J, Shi YH (2011) DNA sequence compression using adaptive particle swarm optimization-based memetic algorithm. IEEE Trans Evolut Comput 15(5):643–658

    Google Scholar 

  3. Arqub OA, Abo-Hammour Z (2014) Numerical solution of systems of second-order boundary value problems using continuous genetic algorithm. Inf Sci 279:396–415

    MathSciNet  MATH  Google Scholar 

  4. Arqub OA (2015) Adaptation of reproducing kernel algorithm for solving fuzzy Fredholm–Volterra integrodifferential equations. Neural Comput Appl 28(7):1–20

    Google Scholar 

  5. Arqub OA, Maayah B (2018) Solutions of Bagley–Torvik and Painlevé equations of fractional order using iterative reproducing kernel algorithm with error estimates. Neural Comput Appl 29(5):1465–1479

    Google Scholar 

  6. Zhang H, Llorca J, Davis CC, Milner SD (2012) Nature-inspired self-organization, control, and optimization in heterogeneous wireless networks. IEEE Trans Mob Comput 11(7):1207–1222

    Google Scholar 

  7. Zhang H, Xiong C, Ho JKL, Chow TWS (2017) Object-level video advertising: an optimization framework. IEEE Trans Ind Inf 13(2):520–531

    Google Scholar 

  8. Xu Y, Pi D (2019) A hybrid enhanced bat algorithm for the generalized redundancy allocation problem. Swarm Evolut Comput. https://doi.org/10.1016/j.swevo.2019.100562

    Article  Google Scholar 

  9. Kennedy J, Eberhart R (2002) Particle swarm optimization. In: Icnn95-international conference on neural networks

  10. Blackwell T, Kennedy J (2018) Impact of communication topology in particle swarm optimization. IEEE Trans Evolut Comput 23:689–702

    Google Scholar 

  11. Abido MA (2002) Optimal power flow using particle swarm optimization. Int J Electr Power Energy Syst 24(7):563–571

    Google Scholar 

  12. Abido AA (2001) Particle swarm optimization for multimachine power system stabilizer design. In: Power Engineering Society Summer Meeting

  13. Ozcan E, Cad S, No TS, Mohan CK (2002) Particle swarm optimization: surfing the waves. In: Congress on evolutionary computation

  14. Clerc M, Kennedy J (2002) The particle swarm: explosion, stability and convergence in multi-dimensional complex space. IEEE Trans Evolut Comput 20(1):1671–1676

    Google Scholar 

  15. Liao W, Wang J, Wang J (2006) Nonlinear inertia weight variation for dynamic adaptation in particle swarm optimization. Comput Oper Res 33(3):859–871

    Google Scholar 

  16. Nickabadi A, Ebadzadeh MM, Safabakhsh R (2011) A novel particle swarm optimization algorithm with adaptive inertia weight. Appl Soft Comput J 11(4):3658–3670

    Google Scholar 

  17. Valle YD, Venayagamoorthy GK, Mohagheghi S, Hernandez JC, Harley RG (2008) Particle swarm optimization: basic concepts, variants and applications in power systems. IEEE Trans Evolut Comput 12(2):171–195

    Google Scholar 

  18. Banks A, Vincent J, Anyakoha C (2008) A review of particle swarm optimization. Part II: hybridisation, combinatorial, multicriteria and constrained optimization, and indicative applications. Nat Comput 7(1):109–124

    MathSciNet  MATH  Google Scholar 

  19. Kao YT, Zahara E (2008) A hybrid genetic algorithm and particle swarm optimization for multimodal functions. Appl Soft Comput 8(2):849–857

    Google Scholar 

  20. Gong YJ, Li JJ, Zhou Y, Li Y, Chung HS, Shi YH, Zhang J (2017) Genetic learning particle swarm optimization. IEEE Trans Cybern 46(10):2277–2290

    Google Scholar 

  21. Deng L, Lu G, Shao Y, Fei M, Hu H (2016) A novel camera calibration technique based on differential evolution particle swarm optimization algorithm. Neurocomputing 174:456–465

    Google Scholar 

  22. Li Z, Wang W, Yan Y, Li Z (2015) PS–ABC: a hybrid algorithm based on particle swarm and artificial bee colony for high-dimensional optimization problems. Expert Syst Appl 42(22):8881–8895

    Google Scholar 

  23. Geng J, Li MW, Dong ZH, Liao YS (2015) Port throughput forecasting by MARS- R SVR with chaotic simulated annealing particle swarm optimization algorithm. Neurocomputing 147(1):239–250

    Google Scholar 

  24. Samma H, Lim CP, Saleh JM (2016) A new reinforcement learning-based memetic particle swarm optimizer. Appl Soft Comput 43(C):276–297

    Google Scholar 

  25. Kennedy J, Mendes R (2002) Population structure and particle swarm performance. In: Congress on evolutionary computation

  26. Watkins CJCH, Dayan P (1992) Q-learning. Mach Learn 8(3–4):279–292

    MATH  Google Scholar 

  27. Rakshit P, Konar A, Bhowmik P, Goswami I, Das S, Jain LC, Nagar AK (2013) Realization of an adaptive memetic algorithm using differential evolution and Q-Learning: a case study in multirobot path planning. IEEE Trans Syst Man Cybern Syst 43(4):814–831

    Google Scholar 

  28. Samma H, Mohamad-Saleh J, Suandi SA, Lahasan B (2019) Q-learning-based simulated annealing algorithm for constrained engineering design problems. Neural Comput Appl 1:1–15

    Google Scholar 

  29. Feng W, Zhang H, Li K, Lin Z, Yang J, Shen X (2018) A hybrid particle swarm optimization algorithm using adaptive learning strategy. Inf Sci 436:162–177

    MathSciNet  Google Scholar 

  30. Shi YH, Eberhart RC (1998) A modified particle swarm optimizer. In: The 1998 IEEE international conference on evolutionary computation proceedings, 1998. IEEE world congress on computational intelligence

  31. Kennedy J (1999) Small worlds and mega-minds: effects of neighborhood topology on particle swarm performance. In: Congress on evolutionary computation

  32. Eberhart R, Kennedy J (2002) A new optimizer using particle swarm theory. In: Mhs95 sixth international symposium on micro machine & human science

  33. Mendes R, Kennedy J, Neves J (2003) Watch thy neighbor or how the swarm can learn from its environment. In: Swarm intelligence symposium

  34. Suganthan PN (1999) Particle swarm optimiser with neighbourhood operator. In: Congress on evolutionary computation

  35. Bonyadi MR, Li X, Michalewicz Z (2014) A hybrid particle swarm with a time-adaptive topology for constrained optimization. Swarm Evolut Comput 18(1):22–37

    Google Scholar 

  36. Wei HL, Isa NAM (2014) Particle swarm optimization with increasing topology connectivity. Eng Appl Artif Intell 27(27):80–102

    Google Scholar 

  37. Marinakis Y, Marinaki M (2013) A hybridized particle swarm optimization with expanding neighborhood topology for the feature selection problem. In: Hybrid metaheuristics. 8th international workshop, HM 2013, pp 37–51

  38. Goldberg DE, Richardson J (1987) Genetic algorithms with sharing for multimodal function optimization. In: International conference on genetic algorithms on genetic algorithms & their application

  39. Goudos SK, Zaharis ZD, Kampitaki DG, Rekanos IT, Hilas CS (2009) Pareto optimal design of dual-band base station antenna arrays using multi-objective particle swarm optimization with fitness sharing. IEEE Trans Magn 45(3):1522–1525

    Google Scholar 

  40. Tao L, Wei C, Pei W (2004) PSO with sharing for multimodal function optimization. In: International conference on neural networks & signal processing

  41. Li X (2010) Niching without niching parameters: particle swarm optimization using a ring topology. IEEE Trans Evolut Comput 14(1):150–169

    Google Scholar 

  42. Parrott D, Li X (2004) A particle swarm model for tracking multiple peaks in a dynamic environment using speciation. In: Congress on evolutionary computation

  43. Peram T, Veeramachaneni K, Mohan CK (2012) Fitness distance, ratio based particle swarm optimization. In: Swarm intelligence symposium

  44. Li X (2007) A multimodal particle swarm optimizer based on fitness Euclidean-distance ratio. In: Conference on genetic & evolutionary computation

  45. Qu BY, Suganthan PN, Das S (2013) A distance-based locally informed particle swarm model for multimodal optimization. IEEE Trans Evolut Comput 17(3):387–402

    Google Scholar 

  46. Engelbrecht AP (2013) Particle swarm optimization: global best or local best? In: BRICS congress on computational intelligence & Brazilian congress on computational intelligence

  47. Shi Y, Liu H, Gao L, Zhang G (2011) Cellular particle swarm optimization. Inf Sci Int J 181(20):4460–4493

    MathSciNet  MATH  Google Scholar 

  48. Das PK, Behera HS, Panigrahi BK (2016) Intelligent-based multi-robot path planning inspired by improved classical Q-learning and improved particle swarm optimization with perturbed velocity. Eng Sci Technol Int J 19(1):651–669

    Google Scholar 

  49. Wang H, Sun H, Li C, Rahnamayan S (2013) Diversity enhanced particle swarm optimization with neighborhood search. Inf Sci 223(2):119–135

    MathSciNet  Google Scholar 

  50. Liang JJ, Qu B-Y, Suganthan PN, Hernández-Díaz AG (2013) Problem definitions and evaluation criteria for the CEC 2013 special session and competition on real-parameter optimization. Technical Report 201212. Computational Intelligence Laboratory, Zhengzhou University, Zhengzhou China and Technical Report, Nanyang Technological University, Singapore, January 2013

  51. Montgomery DC (2009) Design and analysis of experiment, 7th edn. Wiley, New York

    Google Scholar 

  52. Shao Z, Pi D, Shao W (2018) A novel discrete water wave optimization algorithm for blocking flow-shop scheduling problem with sequence-dependent setup times. Swarm Evolut Comput 40:53–75. https://doi.org/10.1016/j.swevo.2017.12.005

    Article  Google Scholar 

  53. Derrac J, García S, Molina D, Herrera F (2011) A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms. Swarm Evolut Comput 1(1):3–18

    Google Scholar 

  54. Dor AE, Lemoine D, Clerc M, Siarry P, Deroussi L, Gourgand M (2015) Dynamic cluster in particle swarm optimization algorithm. Nat Comput 14(4):655–672

    MathSciNet  MATH  Google Scholar 

  55. Lad BK, Kulkarni MS, Misra KB (2009) Optimal reliability design of a system. IEEE Trans Reliab R-22(5):255–258

    Google Scholar 

  56. Garg H, Rani M, Sharma SP (2013) An efficient two phase approach for solving reliability–redundancy allocation problem using artificial bee colony technique. Comput Oper Res 40(12):2961–2969

    MathSciNet  MATH  Google Scholar 

  57. Hsieh YC, You PS (2011) An effective immune based two-phase approach for the optimal reliability–redundancy allocation problem. Appl Math Comput 218(4):1297–1307

    MathSciNet  Google Scholar 

  58. Ouyang HB, Gao LQ, Li S, Kong XY (2015) Improved novel global harmony search with a new relaxation method for reliability optimization problems. Inf Sci 305(C):14–55

    Google Scholar 

  59. Huang CL (2015) A particle-based simplified swarm optimization algorithm for reliability redundancy allocation problems. Reliability Engineering & System Safety 142:221–230

    Google Scholar 

  60. Ravi VRPJ, Zimmermann HJ (2000) Fuzzy global optimization of complex system reliability. IEEE Trans Fuzzy Syst 8(3):241–248

    Google Scholar 

  61. Ravi V, Murty BSN, Reddy PJ (1997) Nonequilibrium simulated-annealing algorithm applied to reliability optimization of complex systems. IEEE Trans Reliab 46(2):233–239

    Google Scholar 

Download references

Acknowledgements

This work was partially supported by National Natural Science Foundation of China (U1433116), the Fundamental Research Funds for the Central Universities (NP2017208), and the Postgraduate Research & Practice Innovation Program of Jiangsu Province (KYCX19_0202).

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China

    Yue Xu & Dechang Pi

Authors
  1. Yue Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dechang Pi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dechang Pi.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interest with any person(s) or organization(s).

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

To show the stability of the proposed algorithm, convergence curves are depicted in Figs. 16 and 17.

Fig. 16
figure 16figure 16

Convergence curves between QLPSO1D and other static topologies

Full size image
Fig. 17
figure 17figure 17

Convergence curves between QLPSO2D and other static topologies

Full size image

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, Y., Pi, D. A reinforcement learning-based communication topology in particle swarm optimization. Neural Comput & Applic 32, 10007–10032 (2020). https://doi.org/10.1007/s00521-019-04527-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-019-04527-9

Keywords

Search

Navigation