Amended Cross-Entropy Cost: An Approach for Encouraging Diversity in Classification Ensemble (Brief Announcement)

Abstract

In the field of machine learning, the training of an ensemble of models is a very common method for reducing the variance of the prediction, and yields better results. Many researches indicate that diversity between the predictions of the models is important for the ensemble performance. However, for Deep Learning classification tasks there is no explicit way to encourage diversity. Negative Correlation Learning (NCL) is a method for doing so in regression tasks. In this work we develop a novel algorithm inspired by NCL to explicitly encourage diversity in Deep Neural Networks (DNNs) for classification. In the development of the algorithm we first assume that the same training characteristics that hold in NCL must also hold when training an ensemble for classification. We also suggest the Stacked Diversified Mixture of Classifiers (SDMC), which is based on our outcome. SDMC is a layer that aims to replace the final layer of a DNN classifier. It can be easily applied on any model, while the cost in terms of number of parameters and computational power is relatively low.