Distributed Evolution of Deep Autoencoders

Conference paper
Part of the Lecture Notes in Networks and Systems book series (LNNS, volume 283)


Autoencoders have seen wide success in domains ranging from feature selection to information retrieval. Despite this success, designing an autoencoder for a given task remains a challenging undertaking due to the lack of firm intuition on how the backing neural network architectures of the encoder and decoder impact the overall performance of the autoencoder. In this work we present a distributed system that uses an efficient evolutionary algorithm to design a modular autoencoder. We demonstrate the effectiveness of this system on the tasks of manifold learning and image denoising. The system beats random search by nearly an order of magnitude on both tasks while achieving near linear horizontal scaling as additional worker nodes are added to the system.


Distributed deep learning Neural architecture search Evolutionary algorithms 


  1. 1.
    Abdi, H., Williams, L.J.: Principal component analysis. WIREs Comput. Stat. 2(4), 433–459 (2010)CrossRefGoogle Scholar
  2. 2.
    Belkin, M., Niyogi, P.: Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput. 15(6), 1373–1396 (2003)CrossRefGoogle Scholar
  3. 3.
    Brock, A., Lim, T., Ritchie, J.M., Weston, N.: SMASH: one-shot model architecture search through hypernetworks. CoRR arXiv:1708.05344 (2017)
  4. 4.
    Charte, F., Rivera, A.J., Martínez, F., del Jesus, M.J.: Automating autoencoder architecture configuration: an evolutionary approach. In: International Work-Conference on the Interplay Between Natural and Artificial Computation, pp. 339–349. Springer, Cham (2019)Google Scholar
  5. 5.
    Coates, A., Ng, A.Y., Lee, H.: An analysis of single-layer networks in unsupervised feature learning. In: Gordon, G.J., Dunson, D.B., Dudík, M. (eds.) Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2011, Fort Lauderdale, USA, 11–13 April 2011, volume 15 of JMLR Proceedings, pp. 215–223. (2011)Google Scholar
  6. 6.
    Google. grpcGoogle Scholar
  7. 7.
    Ha, D., Dai, A.M., Le, Q.V.: Hypernetworks. CoRR arXiv:1609.09106 (2016)
  8. 8.
    Hajewski, J., Oliveira, S.: A scalable system for neural architecture search. In: IEEE Computing and Communication Workshop and Conference, CCWC 2020 (2020)Google Scholar
  9. 9.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778. IEEE Computer Society (2016)Google Scholar
  10. 10.
    Hindman, B., et al.: Mesos: a platform for fine-grained resource sharing in the data center. In: Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation, NSDI 2011, pp. 295–308, Berkeley, CA, USA. USENIX Association (2011)Google Scholar
  11. 11.
    Koutník, J., Cuccu, G., Schmidhuber, J., Gomez, F.: Evolving large-scale neural networks for vision-based reinforcement learning. In: Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation, GECCO 2013, pp. 1061–1068. ACM, New York (2013)Google Scholar
  12. 12.
    Krizhevsky, A., Nair, V., Hinton, G.: CIFAR-10 (Canadian Institute for Advanced Research)Google Scholar
  13. 13.
    Kyriakides, G., Margaritis, K.G.: Neural architecture search with synchronous advantage actor-critic methods and partial training. In: Proceedings of the 10th Hellenic Conference on Artificial Intelligence, SETN 2018, pp. 34:1–34:7. ACM, New York (2018)Google Scholar
  14. 14.
    Lander, S., Shang, Y.: EvoAE - a new evolutionary method for training autoencoders for deep learning networks. In: 2015 IEEE 39th Annual Computer Software and Applications Conference, vol. 2, pp. 790–795, July 2015Google Scholar
  15. 15.
    Lin, T., Zha, H.: Riemannian manifold learning. IEEE Trans. Pattern Anal. Mach. Intell. 30(5), 796–809 (2008)CrossRefGoogle Scholar
  16. 16.
    Liu, H., Simonyan, K., Vinyals, O., Fernando, C., Kavukcuoglu, K.: Hierarchical representations for efficient architecture search. CoRR arXiv:1711.00436 (2017)
  17. 17.
    Liu, H., Simonyan, K., Yang, Y.: DARTS: differentiable architecture search. CoRR arXiv:1806.09055 (2018)
  18. 18.
    Ma, Y., Fu, Y.: Manifold Learning Theory and Applications. CRC Press, Boca Raton (2011)CrossRefGoogle Scholar
  19. 19.
    Miikkulainen, R., et al.: Evolving deep neural networks. CoRR arXiv:1703.00548 (2017)
  20. 20.
    Paszke, A., et al.: Automatic differentiation in PyTorch. In: NIPS Autodiff Workshop (2017)Google Scholar
  21. 21.
    Pham, H., Guan, M., Zoph, B., Le, Q., Dean, J.: Efficient neural architecture search via parameters sharing. In: Dy, J., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4095–4104, Stockholmsmässan, Stockholm Sweden. PMLR, July 2018Google Scholar
  22. 22.
    Real, E., et al.: Large-scale evolution of image classifiers. In: Precup, D., Teh, Y.W. (eds.) Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6–11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 2902–2911. PMLR (2017)Google Scholar
  23. 23.
    Rensin, D.K.: Kubernetes - scheduling the future at cloud scale (2015)Google Scholar
  24. 24.
    Roweis, S.T., Saul, L.K.: Nonlinear dimensionality reduction by locally linear embedding. Science 290, 2323–2326 (2000)CrossRefGoogle Scholar
  25. 25.
    Sciuto, C., Yu, K., Jaggi, M., Musat, C., Salzmann, M.: Evaluating the search phase of neural architecture search. CoRR arXiv:1902.08142 (2019)
  26. 26.
    Suganuma, M., Ozay, M., Okatani, T.: Exploiting the potential of standard convolutional autoencoders for image restoration by evolutionary search. In: Dy, J., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4771–4780, Stockholmsmässan, Stockholm, Sweden. PMLR, July 2018Google Scholar
  27. 27.
    Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. CoRR arXiv:1512.00567 (2015)
  28. 28.
    Talwalkar, A., Kumar, S., Rowley, H.: Large-scale manifold learning. In: 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. IEEE (2008)Google Scholar
  29. 29.
    Tenenbaum, J.B., de Silva, V., Langford, J.C.: A global geometric framework for nonlinear dimensionality reduction. Science 290(5500), 2319–2323 (2000)CrossRefGoogle Scholar
  30. 30.
    Tompson, J., Goroshin, R., Jain, A., LeCun, Y., Bregler, C.: Efficient object localization using convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, 7–12 June 2015, pp. 648–656. IEEE Computer Society (2015)Google Scholar
  31. 31.
    Varda, K.: Protocol buffers: Google’s data interchange format. Technical report, Google, June 2008Google Scholar
  32. 32.
    Vincent, P., Larochelle, H., Bengio, Y., Manzagol, PA.: Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th International Conference on Machine Learning, ICML 2008, pp. 1096–1103. ACM, New York (2008)Google Scholar
  33. 33.
    Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P.A.: Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res. 11, 3371–3408 (2010)MathSciNetzbMATHGoogle Scholar
  34. 34.
    Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 818–833. Springer, Cham (2014). Scholar
  35. 35.
    Zhang, C., Ren, M., Urtasun, R.: Graph hypernetworks for neural architecture search. CoRR arXiv:1810.05749 (2018)
  36. 36.
    Zoph,B., Le, Q.V.: Neural architecture search with reinforcement learning (2017)Google Scholar

Copyright information

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2022

Authors and Affiliations

  1. 1.SalesforceSan FranciscoUSA
  2. 2.Department of Computer ScienceUniversity of IowaIowa CityUSA
  3. 3.AmazonWashington, D.C.USA

Personalised recommendations