Amended Cross-Entropy Cost: An Approach for Encouraging Diversity in Classification Ensemble (Brief Announcement)

  • Ron ShohamEmail author
  • Haim Permuter
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11527)


In the field of machine learning, the training of an ensemble of models is a very common method for reducing the variance of the prediction, and yields better results. Many researches indicate that diversity between the predictions of the models is important for the ensemble performance. However, for Deep Learning classification tasks there is no explicit way to encourage diversity. Negative Correlation Learning (NCL) is a method for doing so in regression tasks. In this work we develop a novel algorithm inspired by NCL to explicitly encourage diversity in Deep Neural Networks (DNNs) for classification. In the development of the algorithm we first assume that the same training characteristics that hold in NCL must also hold when training an ensemble for classification. We also suggest the Stacked Diversified Mixture of Classifiers (SDMC), which is based on our outcome. SDMC is a layer that aims to replace the final layer of a DNN classifier. It can be easily applied on any model, while the cost in terms of number of parameters and computational power is relatively low.


  1. 1.
    Brown, G., Wyatt, J.L., Tiňo, P.: Managing diversity in regression ensembles. J. Mach. Learn. Res. 6(9), 1621–1650 (2005)MathSciNetzbMATHGoogle Scholar
  2. 2.
    Carreira-Perpinan, M.A., Raziperchikolaei, R.: An ensemble diversity approach to supervised binary hashing. In: Lee, D.D., Sugiyama, M., Luxburg, U.V., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems 29, pp. 757–765. Curran Associates Inc. (2016)Google Scholar
  3. 3.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)Google Scholar
  4. 4.
    Krizhevsky, A.: Learning multiple layers of features from tiny images. Technical report, Citeseer (2009)Google Scholar
  5. 5.
    Lee, S., Prakash, S.P.S., Cogswell, M., Ranjan, V., Crandall, D., Batra, D.: Stochastic multiple choice learning for training diverse deep ensembles. In: Lee, D.D., Sugiyama, M., Luxburg, U.V., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems 29, pp. 2119–2127. Curran Associates Inc. (2016)Google Scholar
  6. 6.
    Liu, Y., Yao, X.: Ensemble learning via negative correlation. Neural Netw. 12(10), 1399–1404 (1999)CrossRefGoogle Scholar
  7. 7.
    Nielsen, M.A.: Neural Networks and Deep Learning, vol. 25 (2015)Google Scholar
  8. 8.
    Shi, Z., et al.: Crowd counting with deep negative correlation learning (2018)Google Scholar
  9. 9.
    Srivastava, R.K., Greff, K., Schmidhuber, J.: Training very deep networks. In: Advances in Neural Information Processing Systems, pp. 2377–2385 (2015)Google Scholar
  10. 10.
    Zhou, T., Wang, S., Bilmes, J.A.: Diverse ensemble evolution: Curriculum data-model marriage. In: Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R. (eds.) Advances in Neural Information Processing Systems 31, pp. 5909–5920. Curran Associates Inc. (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Ben-Gurion UniversityBeer-ShevaIsrael

Personalised recommendations