Abstract
Neuroevolution has been used to train Deep Neural Networks on reinforcement learning problems. A few attempts have been made to extend it to address either multi-task or multi-objective optimization problems. This research work presents the Multi-Task Multi-Objective Deep Neuroevolution method, a highly parallelizable algorithm that can be adopted for tackling both multi-task and multi-objective problems. In this method prior knowledge on the tasks is used to explicitly define multiple utility functions, which are optimized simultaneously. Experimental results on some Atari 2600 games, a challenging testbed for deep reinforcement learning algorithms, show that a single neural network with a single set of parameters can outperform previous state of the art techniques. In addition to the standard analysis, all results are also evaluated using the Hypervolume indicator and the Kullback-Leibler divergence to get better insights on the underlying training dynamics. The experimental results show that a neural network trained with the proposed evolution strategy can outperform networks individually trained respectively on each of the tasks.
S. D. Riccio and D. Dyankov—-These authors contributed equally to this work.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Auger, A., Bader, J., Brockhoff, D., Zitzler, E.: Theory of the hypervolume indicator: optimal \(\mu \)-distributions and the choice of the reference point. In: FOGA, pp. 87–102 (2009)
Brockman, G., et al.: OpenAI Gym (2016). https://gym.openai.com
Conti, E., et al.: Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents. In: NeurIPS 2018, Montreal, Canada (2018)
De Jong, K.: Evolutionary Computation - A Unified Approach. The MIT Press, Cambridge (2006)
Dyankov, D., Riccio, S.D., Di Fatta, G., Nicosia, G.: Multi-task learning by pareto optimality. In: Nicosia, G., Pardalos, P., Umeton, R., Giuffrida, G., Sciacca, V. (eds.) LOD 2019. LNCS, vol. 11943, pp. 605–618. Springer, Cham (2019). https://doi.org/10.1007/978-3-03037599-7_50
Espeholt, L., et al.: IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. In: Dy, J., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning, vol. 80, pp. 1407–1416 (2018)
Fonseca, C.M., Paquete, L., López-Ibáñez, M.: An improved dimension-sweep algorithm for the hypervolume indicator. In: 2006 IEEE International Conference on Evolutionary Computation, pp. 1157–1163 (2006)
Hausknecht, M., Lehman, J., Miikkulainen, R., Stone, P.: A neuroevolution approach to general Atari game playing. IEEE Trans. Comput. Intell. AI Games 6, 355–366 (2014)
Jaderberg, M., et al.: Population based training of neural networks (2017). arXiv:1711.09846
Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015). https://doi.org/10.1038/nature14236
Rechenberg, I.: Evolutionsstrategie: optimierung technischer Systeme nach Prinzipien der biologischen Evolution. Ph.D. thesis, Technical University of Berlin, Department of Process Engineering (1971)
Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by back-propagating errors. Nature 323, 533–536 (1986)
Salimans, T., Ho, J., Chen, X., Sidor, S., Sutskever, I.: Evolution Strategies as a Scalable Alternative to Reinforcement Learning. arXiv e-prints arXiv:1703.03864 (2017)
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., Lillicrap, T., Simonyan, K., Hassabis, D.: A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science 362, 1140–1144 (2018)
Stanley, K., Clune, J., Lehman, J., Miikkulainen, R.: Designing neural networks through neuroevolution. Nat. Mach. Intell. 1, 24–35 (2019). https://doi.org/10.1038/s42256-018-0006-z
Stracquadanio, G., Nicosia, G.: Computational energy-based redesign of robust proteins. Comput. Chem. Eng. (2010). https://doi.org/10.1016/j.compchemeng.2010.04.005
Tan, T.G., Teo, J., On, C.: Single- versus multiobjective optimization for evolution of neural controllers in ms. Pac-Man. Int. J. Comput. Games Technol. 2013, 170914 (2013). https://doi.org/10.1155/2013/170914
Vinyals, O., et al.: Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature 575, 350–354 (2019)
Zitzler, E., Thiele, L.: Multiobjective optimization using evolutionary algorithms - a comparative case study. In: A.E., E., T., B., M., S., HP., S. (eds.) Proceedings of the 30th International Conference on Machine Learning, vol. 1498, pp. 292–301 (1998)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Riccio, S.D., Dyankov, D., Jansen, G., Di Fatta, G., Nicosia, G. (2020). Pareto Multi-task Deep Learning. In: Farkaš, I., Masulli, P., Wermter, S. (eds) Artificial Neural Networks and Machine Learning – ICANN 2020. ICANN 2020. Lecture Notes in Computer Science(), vol 12397. Springer, Cham. https://doi.org/10.1007/978-3-030-61616-8_11
Download citation
DOI: https://doi.org/10.1007/978-3-030-61616-8_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-61615-1
Online ISBN: 978-3-030-61616-8
eBook Packages: Computer ScienceComputer Science (R0)