Parallel Training of Artificial Neural Networks Using Multithreaded and Multicore CPUs

  • Olena Schuessler
  • Diego Loyola
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6593)


This paper reports on methods for the parallelization of artificial neural networks algorithms using multithreaded and multicore CPUs in order to speed up the training process. The developed algorithms were implemented in two common parallel programming paradigms and their performances are assessed using four datasets with diverse amounts of patterns and with different neural network architectures. All results show a significant increase in computation speed, which is reduced nearly linear with the number of cores for problems with very large training datasets.


Neural network training multithreading and multicore Pthreads and OpenMP parallelization 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Pinkus, A.: Approximation theory of the MLP model in neural networks. Acta Numerica 8, 143–195 (1999)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Sodan, A.C., Machina, J., Deshmeh, A., Macnaughton, K., Esbaugh, B.: Parallelism via Multithreaded and Multicore CPUs. Computer 43(3), 24–32 (2010)CrossRefGoogle Scholar
  3. 3.
    Seiffert, U.: Artificial Neural Networks on Massively Parallel Computer Hardware. In: ESANN 2002 Proceedings - European Symposium on Artificial Neural Networks, April 24-26, pp. 319–330. Bruges, Belgium (2002)Google Scholar
  4. 4.
    Turchenko, V., Grandinetti, L.: Efficiency Analysis of Parallel Batch Pattern NN Training Algorithm on General-Purpose Supercomputer. In: Omatu, S., Rocha, M.P., Bravo, J., Fernández, F., Corchado, E., Bustillo, A., Corchado, J.M. (eds.) IWANN 2009. LNCS, vol. 5518, pp. 223–226. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  5. 5.
    Tsaregorodtsev, V.: Parallel Implementation of back-Propagation Neural Network Software on SMP Computers. In: Malyshkin, V.E. (ed.) PaCT 2005. LNCS, vol. 3606, pp. 186–192. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  6. 6.
    Lotrič, U., Dobnikar, A.: Parallel Implementations of Recurrent Neural Network Learning. In: Kolehmainen, M., Toivanen, P., Beliczynski, B. (eds.) ICANNGA 2009. LNCS, vol. 5495, pp. 99–108. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  7. 7.
    Gallant, S.: Perceptron-based learning algorithms. IEEE Transactions on Neural Networks 1(2), 179–191 (1990)CrossRefGoogle Scholar
  8. 8.
    Rummelhart, D., Hinton, G., Williams, R.: Learning Internal Representations by Error Propagation. In: Rumelhart, D.E., McClelland, J.L. (eds.) Parallel Distributed Processing, vol. I, pp. 318–362. MIT Press, Cambridge (1986)Google Scholar
  9. 9.
    Prechelt, L.: Automatic early stopping using cross validation: quantifying the criteria. Neural Networks 11(4), 761–767 (1998)CrossRefGoogle Scholar
  10. 10.
    Riedmiller, M., Braun, H.: Rprop - A Fast Adaptive Learning Algorithm. In: Proceedings of the International Symposium on Computer and Information Science VII, Technical Report (1992)Google Scholar
  11. 11.
    Fahlman, S.: An Empirical Study of Learning Speed in back-Propagation Networks. Computer Science Technical Report, CMU-CS-88-162 (1988)Google Scholar
  12. 12.
    Butenhof, D.R.: Programming with POSIX Threads. Addison-Wesley, Reading (1997) ISBN 0-201-63392-2 Google Scholar
  13. 13.
    Quinn, M.J.: Parallel Programming in C with MPI and OpenMP. McGraw-Hill Inc., New York (2004) ISBN 0-07-058201-7 Google Scholar
  14. 14.
    Kuhn, B., Petersen, P., O’Toole, E.: OpenMP versus threading in C/C++. Concurrency: Practice and Experience 12, 1165–1176 (2000)CrossRefzbMATHGoogle Scholar
  15. 15.
    Stamatakis, A., Ott, M.: Exploiting Fine-Grained Parallelism in the Phylogenetic Likelihood Function with MPI, Pthreads, and OpenMP: A Performance Study. In: Chetty, M., Ngom, A., Ahmad, S. (eds.) PRIB 2008. LNCS (LNBI), vol. 5265, pp. 424–435. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  16. 16.
    Alpaydin, E., Kaynak, C.: Optical Recognition of Handwritten Digits Data Set,
  17. 17.
    Loyola, D., Coldewey-Egbers, M., Dameris, M., Garny, H., Stenke, A., Van Roozendael, M., Lerot, C., Balis, D., Koukouli, M.: Global long-term monitoring of the ozone layer - a prerequisite for predictions. International Journal of Remote Sensing 30(15), 4295–4318 (2009)CrossRefGoogle Scholar
  18. 18.
    Loyola, D.: Applications of Neural Network Methods to the Processing of Earth Observation Satellite Data. Neural Networks 19(2), 168–177 (2006)CrossRefGoogle Scholar
  19. 19.
    Tang, G., D’Azevedo, E., Zhang, F., Parker, J., Watson, B., Jardine, P.: Application of a hybrid MPI/OpenMP approach for parallel groundwater model calibration using multi-core computers. Computers & Geosciences 36(11), 1451–1460 (2010)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Olena Schuessler
    • 1
  • Diego Loyola
    • 1
  1. 1.German Aerospace CenterInstitute of Remote SensingWeßlingGermany

Personalised recommendations