Naive bayesian classifier committees

  • Zijian Zheng
Multiple Models for Classification
Part of the Lecture Notes in Computer Science book series (LNCS, volume 1398)


The naive Bayesian classifier provides a very simple yet surprisingly accurate technique for machine Some researchers have examined extensions to the naive Bayesian classifier that seek to further improve the accuracy. For example, a naive Bayesian tree approach generates a decision tree with one naive Bayesian classifier at each leaf. Another example is a constructive Bayesian classifier that eliminates attributes and constructs new attributes using Cartesian products of existing attributes. This paper proposes a simple, but effective approach for the same purpose. It generates a naive Bayesian classifier committee for a given classification task. Each member of the committee is a naive Bayesian classifier based on a subset of all the attributes available for the task. During the classification stage, the committee members vote to predict classes. Experiments across a wide variety of natural domains show that this method significantly increases the prediction accuracy of the naive Bayesian classifier on average. It performs better than the two approaches mentioned above in terms of higher prediction accuracy.


Error Rate Committee Member Lower Error Rate Bayesian Classifier Average Error Rate 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. Almuallim, H. and Dietterich, T.G.: Efficient algorithms for identifying relevant features.Proceedings of the gth Canadian Conference on Artificial Intelligence. Vancouver, BC: Morgan Kaufmann (1992) 38–45.Google Scholar
  2. Breiman, L., Friedman, J.H., Olshen, R.A., and Stone, C.J.: Classification And Regression Trees. Belmont, CA: Wadsworth (1984).Google Scholar
  3. Breiman, L.: Bagging predictors. Machine Learning. 24 (1996) 123–140.Google Scholar
  4. Cestnik, B., Kononenko, I., and Bratko, I.: Assistant 86: A knowledge-elicitation tool for sophisticated users. In I. Bratko & N. Lavrač (Eds.), Progress in Machine Learning — Proceedings of the 2nd European Working Session on Learning (EWSL87). Wilmslow, UK: Sigma Press (1987) 31–45.Google Scholar
  5. Chatfield, C.: Statistics for Technology: A Course in Applied Statistics. London: Chapman and Hall (1978).Google Scholar
  6. Domingos, P. and Pazzani, M.: Beyond independence: Conditions for the optimality of the simple Bayesian classifier. Proceedings of the 13th International Conference on Machine Learning. San Francisco, CA: Morgan Kaufmann (1996) 105–112.Google Scholar
  7. Duda, R.O. and Hart, P.E.: Pattern Classification and Scene Analysis. New York: John Wiley (1973).Google Scholar
  8. Fayyad, U.M. and Irani, K.B.: Multi-interval discretization of continuous-valued attributes for classification learning. Proceedings of the 13th International Joint Conference on Artificial Intelligence. Morgan Kaufmann (1993) 1022–1027.Google Scholar
  9. Freund, Y. and Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to boosting. Unpublished manuscript, available from the authors' home pages (“≈yoav,≈schapire”) (1996a).Google Scholar
  10. Freund, Y. and Schapire, R.E.: Experiments with a new boosting algorithm. Proceedings of the 13th International Conference on Machine Learning. Morgan Kaufmann (1996b) 148–156.Google Scholar
  11. John, G.H., Kohavi, R., and Pfleger, K.: Irrelevant features and the subset selection problem. Proceedings of the 11th International Conference on Machine Learning. San Francisco, CA: Morgan Kaufmann (1994) 121–129.Google Scholar
  12. Kira, K. and Rendell, L.A.: The feature selection problem: Traditional methods and a new algorithm. Proceedings of the 10th National Conference on Artificial Intelligence. Menlo Park, CA: AAAI Press/Cambridge, MA: MIT Press (1992) 129–134.Google Scholar
  13. Kohavi, R.: A study of cross-validation and bootstrap for accuracy estimation and model selection. Proceedings of the 14th International Joint Conference on Artificial Intelligence. San Mateo, CA: Morgan Kaufmann (1995) 1137–1143.Google Scholar
  14. Kohavi, R.: Scaling up the accuracy of naive-Bayes classifiers: A decision-tree hybrid. Proceedings of the 2nd International Conference on Knowledge Discovery and Data Mining. Menlo Park, CA: The AAAI Press (1996) 202–207.Google Scholar
  15. Kononenko, I.: Comparison of inductive and naive Bayesian learning approaches to automatic knowledge acquisition. In B. Wielinga et al. (Eds.), Current Trends in Knowledge Acquisition. Amsterdam: IOS Press (1990).Google Scholar
  16. Kononenko, I.: Semi-naive Bayesian classifier. Proceedings of European Conference on Artificial Intelligence (1991) 206–219.Google Scholar
  17. Langley, P., Iba, W.F., and Thompson, K.: An analysis of Bayesian classifiers. Proceedings of the 10th National Conference on Artificial Intelligence. Menlo Park, CA: The AAAI Press (1992) 223–228.Google Scholar
  18. Langley, P.: Selection of relevant features in machine learning. Proceeding of the AAAI Fall Symposium on Relevance, New Orleans, LA: The AAAI Press (1994).Google Scholar
  19. Langley, P. and Sage, S.: Induction of selective Bayesian classifiers. Proceedings of the 10th Conference on Uncertainty in Artificial Intelligence. Seattle, WA: Morgan Kaufmann (1994) 339–406.Google Scholar
  20. Merz, C.J. and Murphy, P.M.: UCI Repository of Machine Learning Databases [≈mleaxn/MLRepository.html]. Irvine, CA: University of California, Department of Information and Computer Science (1997).Google Scholar
  21. Pazzani, M.J.: Constructive induction of Cartesian product attributes. Proceedings of the Conference, ISIS'96: Information, Statistics and Induction in Science. Singapore: World Scientific (1996) 66–77.Google Scholar
  22. Quinlan, J.R.: Bagging, boosting, and C4.5. Proceedings of the 13th National Conference on Artificial Intelligence, Menlo Park: The AAAI Press (1996) 725–730.Google Scholar
  23. Schapire, R.E., Freund, Y., Bartlett, P., and Lee W.S.: Boosting the margin: A new explanation for the effectiveness of voting methods. Proceedings of the 11th International Conference on Machine Learning. Morgan Kaufmann (1997) 322–330.Google Scholar
  24. Ting, K.M.: Discretization of continuous-valued attributes and instance-based learning (Technical Report 491). Sydney, Australia: University of Sydney, Basser Department of Computer Science (1994).Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1998

Authors and Affiliations

  • Zijian Zheng
    • 1
  1. 1.School of Computing and MathematicsDeakin UniversityGeelongAustralia

Personalised recommendations