Beyond Traditional Probabilistic Data Processing Techniques: Interval, Fuzzy etc. Methods and Their Applications pp 517-543 | Cite as

# Enhancement of Cross Validation Using Hybrid Visual and Analytical Means with Shannon Function

- 32 Downloads

## Abstract

The algorithm of *k*-fold cross validation is actively used to evaluate and compare machine learning algorithms. However, it has several important deficiencies documented in the literature along with its advantages. The advantages of quick computations are also a source of its major deficiency. It tests only a small fraction of all the possible splits of data, on training and testing data leaving untested many difficult for prediction splits. The associated difficulties include bias in estimated average error rate and its variance, the large variance of the estimated average error, and possible irrelevance of the estimated average error to the problem of the user. The goal of this paper is improving the cross validation approach using the combined visual and analytical means in a hybrid setting. The visual means include both the point-to-point mapping and a new point–to-graph mapping of the n-D data to 2-D data known as General Line Coordinates. The analytical means involve the adaptation of the Shannon function to obtain the worst case error estimate. The method is illustrated by classification tasks with simulated and real data.

## Keywords

*k*-fold cross validation Machine learning Visual analytics Visualization Multidimensional data Shannon function Worst case Error estimate Error rate General line coordinates Linear classifier Hybrid algorithm Interactive algorithm

## References

- 1.D. Anguita, A. Ghio, S. Ridella, D. Sterpi, K-fold cross validation for error rate estimate in support vector machines, in
*DMIN*(2009 Jan), pp. 291–297Google Scholar - 2.S. Arlot, M. Lerasle, Choice of V for V-fold cross-validation in least-squares density estimation. J. Mach. Learn. Res.
**17**(208), 1–50 (2016)MathSciNetzbMATHGoogle Scholar - 3.A. Blum, A. Kalai, J. Langford, Beating the hold-out: bounds for k-fold and progressive cross validation, in
*Proceedings of the Twelfth Annual Conference on Computational Learning Theory*(ACM, 1999 Jul 6), pp. 203–208Google Scholar - 4.K.P. Bennett, C. Campbell, Support vector machines: hype or hallelujah? ACM SIGKDD Explorations Newsl
**2**(2), 1–13 (2000)CrossRefGoogle Scholar - 5.K.P. Bennett, E.J. Bredensteiner, Duality and geometry in SVM classifiers, in
*ICML*(2000 Jun 29), pp. 57–64Google Scholar - 6.Y. Bengio, Y. Grandvalet, No unbiased estimator of the variance of k-fold cross-validation. J. Mach. Learn. Res.
**5**, 1089–1105 (2004)Google Scholar - 7.T.G. Dietterich, Approximate statistical tests for comparing supervised classification learning algorithms. Neural Comput.
**10**(7), 1895–1923 (1998)CrossRefGoogle Scholar - 8.W. Duch, R. Adamczak, K. Grąbczewski, K. Grudziński, N. Jankowski, A. Naud,
*Extraction of Knowledge from Data Using Computational Intelligence Methods*(Copernicus University, Toruń, Poland, 2000). https://www.fizyka.umk.pl/~duch/ref/kdd-tut/Antoine/mds.htm - 9.Y. Grandvalet, Y. Bengio,
*Hypothesis Testing for Cross-Validation*(Montreal Universite de Montreal, Operationnelle DdIeR, 2006 Aug 29), p. 1285Google Scholar - 10.B. Gu, V.S. Sheng, K.Y. Tay, W. Romano, S. Li, Cross validation through two-dimensional solution surface for cost-sensitive SVM. IEEE Trans. Pattern Anal. Mach. Intell.
**39**(6), 1103–1121 (2017)CrossRefGoogle Scholar - 11.G. Hansel, Sur le nombre des functions Booléenes monotones de n variables. C.R. Acad. Sci., Paris,
**262**(20), 1088–1090 (1966)Google Scholar - 12.J. Jolliffe,
*Principal of Component Analysis*(Springer, 1986)Google Scholar - 13.N.A. Karpova, Some properties of Shannon functions. Math. Notes Acad. Sci. USSR
**8**(5), 843–849 (1970)MathSciNetzbMATHGoogle Scholar - 14.T. Kohonen,
*Self-Organizing Maps*(Springer, Berlin, Germany, 1995)CrossRefGoogle Scholar - 15.A. Inselberg,
*Parallel Coordinates: Visual Multidimensional Geometry and Its Applications*(Springer, 2009)Google Scholar - 16.B. Kovalerchuk, E. Triantaphyllou, A. Despande, E. Vityaev, Interactive learning of monotone Boolean functions. Inf. Sci.
**94**(1–4), 87–118 (1996)CrossRefGoogle Scholar - 17.B. Kovalerchuk, Quest for rigorous combining probabilistic and fuzzy logic approaches for computing with words, in
*On Fuzziness. A Homage to Lotfi A. Zadeh*, vol. 1 (Studies in Fuzziness and Soft Computing Vol. 216), edited by R. Seising, E. Trillas, C. Moraga, S. Termini (Springer, Berlin, New York, 2013), pp. 333–344Google Scholar - 18.B. Kovalerchuk, Visualization of multidimensional data with collocated paired coordinates and general line coordinates, in
*Proceedings of SPIE 2014*, vol. 9017. https://doi.org/10.1117/12.2042427 - 19.B. Kovalerchuk, Visual cognitive algorithms for high-dimensional data and super-intelligence challenges. J.: Cogn. Syst. Res.
**45**, 95–108 (2017)Google Scholar - 20.B. Kovalerchuk, V. Grishin, Adjustable general line coordinates for visual knowledge discovery in n-D data. Inf. Vis. (2017). https://doi.org/10.1177/1473871617715860CrossRefGoogle Scholar
- 21.B. Kovalerchuk, M. Kovalerchuk, Toward virtual data scientist, in
*Proceedings of the 2017 International Joint Conference on Neural Networks*(Anchorage, AK, USA, 14–19 May 2017), pp. 3073–3080Google Scholar - 22.B. Kovalerchuk, D. Dovhalets, Constructing interactive visual classification, clustering and dimension reduction models for n-D data. Informatics
**4**(23) (2017). http://www.mdpi.com/2227–9709/4/3/23 - 23.V. Kreinovich (ed.),
*Uncertainty Modeling, Studies in Computational Intelligence*, vol. 683 (Springer, 2017)Google Scholar - 24.M. Lichman,
*UCI Machine Learning Repository*(University of California, School of Information and Computer Science, Irvine, CA, 2013). https://archive.ics.uci.edu/ml. Accessed on 15 June 2017 - 25.J. Kruskal, M. Wish,
*Multidimensional Scaling*(Sage Publications, 1978)Google Scholar - 26.M. Lichman,
*UCI Machine Learning Repository*(University of California, School of Information and Computer Science, Irvine, CA, 2013). Parkinson’s, https://archive.ics.uci.edu/ml/datasets/. Accessed on 15 June 2017 - 27.T. Mitchell, Introduction to machine learning, in
*Machine Learning*(McGraw-Hill, Columbus, 1997)Google Scholar - 28.T.J. Mitchell, S.Y. Chen, R.D. Macredie, Hypermedia learning and prior knowledge: domain expertise vs. system expertise. J. Comput. Assist. Learn.
**21**(1), 53–64 (2005)Google Scholar - 29.J.G. Moreno-Torres, J.A. Sáez, F. Herrera, Study on the impact of partition-induced dataset shift on
*k*-fold cross validation. IEEE Trans. Neural Netw. Learn. Syst.**23**(8), 1304–1312 (2012)CrossRefGoogle Scholar - 30.C.E. Shannon, The synthesis of two-terminal switching circuits. Bell Syst. Tech. J.
**28**, 59–98 (1949)MathSciNetCrossRefGoogle Scholar - 31.J. Sharko, G. Grinstein, K. Marx, Vectorized Radviz and its application to multiple cluster datasets. IEEE Trans. Vis. Comput. Graph.
**14**(6), 1427–1444 (2008)Google Scholar - 32.H. Shimodaira, Improving predictive inference under covariate shift by weighting the log-likelihood function. J. Stat. Plan. Inference
**90**(2), 227–244 (2000)MathSciNetCrossRefGoogle Scholar - 33.G. Turán, F. Vatan, On the computation of Boolean functions by analog circuits of bounded fan-in, in
*Proceedings of the 35th Annual Symposium on Foundations of Computer Science, 1994*(IEEE, 1994 Nov 20), pp. 553–564Google Scholar - 34.V. Vapnik, A. Vashist, A new learning paradigm: learning using privileged information. Neural Netw.
**22**(5), 544–557 (2009)CrossRefGoogle Scholar - 35.T.T. Wong, Parametric methods for comparing the performance of two classification algorithms evaluated by k-fold cross validation on multiple data sets. Pattern Recogn.
**65**, 97–107 (2017)CrossRefGoogle Scholar - 36.H. Yin, ViSOM-a novel method for multivariate data projection and structure visualization. IEEE Trans. Neural Netw.
**13**(1), 237–243 (2002)MathSciNetCrossRefGoogle Scholar - 37.L.A. Zadeh (ed.), Computing with words in Information/Intelligent systems 1: Foundations. Physica (2013)Google Scholar
- 38.P. Zhang, Model selection via multifold cross validation. Ann. Stat.
**1**, 299–313 (1993)MathSciNetCrossRefGoogle Scholar