Abstract
General theories describing the performance of artificial learners are of little help when a user is confronted with a selection of datasets and a given artificial classifier. The objective of this paper is to find out the best description of the learning curves produced by a Naïve Bayes classification. The performance of Naïve Bayes was measured on 121 datasets using k-fold crossvalidation. Power, linear, logarithmic and exponential functions were fit to the data. The exponential function was a better descriptor of the error rate in 44 of 60 useful cases. Average mean squared error is significantly different at P=0,000 from power and linear and at P=0,001 from logarithmic function. The exponential function’s rank is significantly different from the ranks of other models (P=0,000). The results can be used to forecast the future performance of the learner, or to check where on the learning curve the current measurement lies.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Anderson, J.R., Schooler, L.J.: Reflections of the Environment in Memory. Psychological Science 2(6), 396–408 (1991)
Anderson, R.B.: The power law as an emergent property. Memory & Cognition 29(7), 1061–1068 (2001)
Clauset, A., Shalizi, C.R., Newman, M.E.J.: Power-Law Distributions in Empirical Data. SIAM Review 51(4), 661–703 (2009), doi:10.1137/070710111
Heathcote, A., Brown, S., Mewhort, D.J.K.: The power law repealed: The case for an exponential law of practice. Psychonomic Bulletin & Review 7(2), 185–207 (2000), doi:10.3758/bf03212979
Kotsiantis, S.B.: Supervised Machine Learning: A Review of Classification Techniques. Informatica (Ljubljana) 31(3), 249–268 (2007)
Dzemyda, G., Sakalauskas, L.: Large-Scale Data Analysis Using Heuristic Methods. Informatica (Lithuan.) 22(1), 1–10 (2011)
Vapnik, V.N.: Estimation of Dependences Based on Empirical Data. Springer, NY (1982)
Brumen, B., Jurič, M.B., Welzer, T., Rozman, I., Jaakkola, H., Papadopoulos, A.: Assessment of classification models with small amounts of data. Informatica (Lithuan.) 18(3), 343–362 (2007)
Dučinskas, K., Stabingiene, L.: Expected Bayes Error Rate in Supervised Classification of Spatial Gaussian Data. Informatica (Lithuan.) 22(3), 371–381 (2011)
Frey, L.J., Fisher, D.H.: Modeling decision tree performance with the power law. In: Seventh International Workshop on Artificial Intelligence and Statistics. Morgan Kaufmann, Ft. Lauderdale (1999)
Last, M.: Predicting and Optimizing Classifier Utility with the Power Law. In: 7th IEEE International Conference on Data Mining, ICDM Workshops 2007. IEEE, Omaha (2007), doi:10.1109/icdmw.2007.31
Provost, F., Jensen, D., Oates, T.: Efficient progressive sampling. In: Fifth International Conference on Knowledge Discovery and Data Mining. ACM, San Diego (1999)
Singh, S.: Modeling Performance of Different Classification Methods: Deviation from the Power Law. Project Report. Vanderbilt University, Nashville, Tennessee, USA, Department of Computer Science (2005)
Dzemyda, G., Sakalauskas, L.: Optimization and Knowledge-Based Technologies. Informatica (Lithuan.) 20(2), 165–172 (2009)
John, G.H., Langley, P.: Estimating Continuous Distributions in Bayesian Classifiers. In: Eleventh Conference on Uncertainty in Artificial Intelligence, August 18-20. Morgan Kaufmann, San Francisco (1995)
Witten, I.H., Frank, E.: Data Mining: Practical machine learning tools and techniques, 2nd edn. Morgan Kaufmann, San Francisco (2005) ISBN: 0120884070
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The WEKA data mining software: an update. ACM SIGKDD Explorations Newsletter 11(1), 10–18 (2009)
Asuncion, A., Newman, D.: UCI Machine Learning Repository (2010), http://archive.ics.uci.edu/ml/datasets.html (Archived by WebCite® at http://www.webcitation.org/6C2hgsRrX )
TunedIT. TunedIT research repository (2012), http://tunedit.org/search?q=arff&qt=Repository (accessed: December 12, 2012) (Archived by WebCite® at http://www.webcitation.org/6CqplN6Xr )
Kjellerstrand H.: My Weka page (2012), http://www.hakank.org/weka/ (accessed: December 12, 2012) (Archived by WebCite® at http://www.webcitation.org/6Cqq5pQtZ )
Kjellerstrand, H.: My Weka page/DASL (2012), http://www.hakank.org/weka/DASL/ (accessed: December 12, 2012) (Archived by WebCite® at http://www.webcitation.org/6CqqCwPmy )
Chai, K.: Kevin Chai Datasets (2012), http://kevinchai.net/datasets (accessed: December 12, 2012) (Archived by WebCite® at http://www.webcitation.org/6CqqWlQEp )
Brumen, B., Hölbl, M., Harej Pulko, K., Welzer, T., Heričko, M., Jurič, M.B., Jaakkola, H.: Learning Process Termination Criteria. Informatica (Lithuan.) 23(4), 521–536 (2012)
Cohen, P.R.: Empirical methods for artificial intelligence. MIT Press, Cambridge (1995) ISBN: 9780262032254
Weiss, S.M., Kulikowski, C.A.: Computer systems that learn: classification and prediction methods from statistics, neural nets, machine learning, and expert systems. Morgan Kaufmann, San Mateo (1991) ISBN: 978-1558600652
McLachlan, G.J., Do, K.-A., Ambroise, C.: Analyzing microarray gene expression data. Wiley, Hoboken (2004) ISBN: 0471226165
Eaton, J.W.: GNU Octave (2012), http://www.gnu.org/software/octave/ (accessed: December 12, 2012) (Archived by WebCite® at http://www.webcitation.org/6CqyEvDKU )
Marquardt, D.W.: An Algorithm for Least-Squares Estimation of Nonlinear Parameters. Journal of the Society for Industrial and Applied Mathematics 11(2), 431–441 (1963), doi:10.2307/2098941
Levenberg, K.: A Method for the Solution of Certain Non-Linear Problems in Least Squares. Quarterly of Applied Mathematics 2, 164–168 (1944)
Argyrous, G.: Statistics for research: With a guide to SPSS, 3rd edn. SAGE Publications Ltd., Thousand Oaks (2011) ISBN: 1849205957
Medvedev, V., Dzemyda, G., Kurasova, O., Marcinkevicijus, V.: Efficient Data Projection for Visual Analysis of Large Data Sets Using Neural Networks. Informatica (Lithuan.) 22(4), 507–520 (2011)
Abdi, H.: The Bonferonni and Šidák Corrections for Multiple Comparisons. In: Salkind, N.J. (ed.) Encyclopedia of Measurement and Statistics. SAGE Publications, Inc., Thousand Oaks (2007) ISBN: 9781412916110
Pragarauskaite, J., Dzemyda, G.: Markov Models in the Analysis of Frequent Patterns in Financial Data. Informatica (Lithuan.) 24(1), 87–102 (2014)
Pišek, P., Štumberger, B., Marčič, T., Virtič, P.: Design analysis and experimental validation of a double rotor synchronous PM machine used for HEV. IEEE Transactions on Magnetics 49(1), 152–155 (2013), doi:10.1109/TMAG.2012.2220338
Virtič, P.: Determining losses and efficiency of axial flux permanent magnet synchronous motor. Przeglęad Elektrotechniczny 89(2b), 13–16 (2013)
Virtič, P., Pišek, P., Hadžiselimović, M., Marčič, T., Štumberger, B.: Torque analysis of an axial flux permanent magnet synchronous machine by using analytical magnetic field calculation. IEEE Transactions on Magnetics 45(3), 1036–1039 (2009), doi:10.1109/TMAG.2009.2012566
Virtič, P., Pišek, P., Marčič, T., Hadžiselimović, M., Štumberger, B.: Analytical analysis of magnetic field and back electromotive force calculation of an axial-flux permanent magnet synchronous generator with coreless stator. IEEE Transactions on Magnetics 44(11), 4333–4336 (2008)
Hadžiselimović, M., Virtič, P., Štumberger, G., Marčič, T., Štumberger, B.: Determining force characteristics of an electromagnetic brake using co-energy. Journal of Magnetism and Magnetic Materials 320(20), e556-e561 (2008), doi: 10.1016/j.jmmm.2008.04.013
Castillo, G., Gama, J.: Adaptive Bayesian network classifiers. Intelligent Data Analysis 13(1), 39–59 (2009), doi:10.3233/IDA-2009-0355
Castillo, G., Gama, J.: An adaptive prequential learning framework for Bayesian network classifiers. In: Fürnkranz, J., Scheffer, T., Spiliopoulou, M. (eds.) PKDD 2006. LNCS (LNAI), vol. 4213, pp. 67–78. Springer, Heidelberg (2006)
Gama, J., Medas, P., Castillo, G., Rodrigues, P.: Learning with drift detection. In: Bazzan, A.L.C., Labidi, S. (eds.) SBIA 2004. LNCS (LNAI), vol. 3171, pp. 286–295. Springer, Heidelberg (2004)
Cipresso, P., Carelli, L., Solca, F., Meazzi, D., Meriggi, P., Poletti, B., Lulé, D., Ludolph, A.C., Silani, V., Riva, G.: The use of P300-based BCIs in amyotrophic lateral sclerosis: from augmentative and alternative communication to cognitive assessment. Brain and Behavior 2(4), 479–498 (2012), doi:10.1002/brb3.57
Cipresso, P., Paglia, F., Cascia, C., Riva, G., Albani, G., La Barbera, D.: Break in volition: a virtual reality study in patients with obsessive-compulsive disorder. Experimental Brain Research 229(3), 443–449 (2013), doi:10.1007/s00221-013-3471-y
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Brumen, B., Rozman, I., Černezel, A. (2014). Observing a Naïve Bayes Classifier’s Performance on Multiple Datasets. In: Manolopoulos, Y., Trajcevski, G., Kon-Popovska, M. (eds) Advances in Databases and Information Systems. ADBIS 2014. Lecture Notes in Computer Science, vol 8716. Springer, Cham. https://doi.org/10.1007/978-3-319-10933-6_20
Download citation
DOI: https://doi.org/10.1007/978-3-319-10933-6_20
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-10932-9
Online ISBN: 978-3-319-10933-6
eBook Packages: Computer ScienceComputer Science (R0)