On the Statistical Comparison of Inductive Learning Methods
Experimental comparisons between statistical and machine learning methods appear with increasing frequency in the literature. However, there does not seem to be a consensus on how such a comparison is performed in a methodologically sound way. Especially the effect of testing multiple hypotheses on the probability of producing a ”false alarm” is often ignored.
We transfer multiple comparison procedures from the statistical literature to the type of study discussed in this paper. These testing procedures take the number of tests performed into account, thereby controlling the probability of generating ”false alarms”. The multiple comparison procedures selected are illustrated on well-know regression and classification data sets.
Unable to display preview. Download preview PDF.
- [CO94]P. Cheeseman and R.W. Oldford, editors. Selecting models from data: AI and statistics IV. Lecture notes in statistics nr. 89. Springer-Verlag, New York, 1994.Google Scholar
- [DM94]F. Diebold and R. Mariano. Comparing predictive accuracy. Technical re- port, University of Pennsylvania, 1994.Google Scholar
- [Fis35]R. A. Fisher. The design of experiments. Olivier & Boyd, Edinburgh, 1935.Google Scholar
- [Han93]D. J. Hand, editor. Artificial intelligence frontiers in statistics: AI and statis- tics III. Chapman Si Hall, London, 1993.Google Scholar
- [Hay88]W. Hays. Statistics. Holt, Rinehart and Winston, Inc, Fort Worth, 1988.Google Scholar
- [MM77]L. Marascuilo and M. McSweeney. Nonparametric and distribution free methods for the social sciences. Brooks/Cole Publishing Company, Monterey, CA, 1977.Google Scholar
- [Pre95]Lutz Prechelt. A quantitative study of neural network learning algorithm evaluation practices. In Proc Intl. Conf. on Artificial Neural Networks, Cambridge, UK, June 26–28, 1995.Google Scholar
- [Rip93]B.D. Ripley. Flexible non-linear approaches to classification. In V. Cherkassky, J.H. Friedman, and H. C, editors, From Statistics to Neural Networks: Theory and Pattern Recognition Applications. Springer-Verlag, 1993.Google Scholar
- [TK92]Kar Yan Tam and Melody Y. Kiang. Managerial applications of neural net- works: the case of bank failure predictions. Management science,38(7):926–947, 1992.Google Scholar
- [WBM91]B. Winer, D. Brown, and K. Michels. Statistical principles in experimental design. McGraw-Hill, New York, 1991.Google Scholar
- [WG94]A. Weigend and N. Gershenfield. Time series prediction: forecasting the future and understanding the past. Addison-Wesley, Reading, 1994.Google Scholar
- WHR90] A. Weigend, A. Huberman, and D. Rumelhart. Predicting the future: a connectionist approach. International Journal of Neural Systems,1(3):193209, 1990.Google Scholar
- [WY93]P.H. Westfall and S.S. Young. Resampling-Based Multiple Testing. John Wiley & Sons, New York, 1993.Google Scholar