Combination of Classifiers

Abstract

A combination or an ensemble of classifiers is a set of classifiers whose individual decisions are combined to classify new examples. A combination of classifiers is often much more accurate than the individual classifiers that make them up. One reason for this could be that the training data may not provide sufficient information for choosing a single best classifier and a combination is the best compromise. Another reason could be that the learning algorithms used may not be able to solve the difficult search problem posed. Since solving the search problem may be difficult, suitable heuristics may be used in the search. As a consequence of this, even though with the training examples and prior knowledge, a unique best hypothesis exists, we may not be able to find it. A combination of classifiers is a way of compensating for imperfect classifiers. The learning algorithms we use may give us good approximations to the true value but may not be the right hypothesis. By taking a weighted combination of these approximations, we may be able to represent the true hypothesis. In fact, the combination could be equivalent to very complex decision trees.