Enhancement of Cross Validation Using Hybrid Visual and Analytical Means with Shannon Function

  • Boris KovalerchukEmail author
Part of the Studies in Computational Intelligence book series (SCI, volume 835)


The algorithm of k-fold cross validation is actively used to evaluate and compare machine learning algorithms. However, it has several important deficiencies documented in the literature along with its advantages. The advantages of quick computations are also a source of its major deficiency. It tests only a small fraction of all the possible splits of data, on training and testing data leaving untested many difficult for prediction splits. The associated difficulties include bias in estimated average error rate and its variance, the large variance of the estimated average error, and possible irrelevance of the estimated average error to the problem of the user. The goal of this paper is improving the cross validation approach using the combined visual and analytical means in a hybrid setting. The visual means include both the point-to-point mapping and a new point–to-graph mapping of the n-D data to 2-D data known as General Line Coordinates. The analytical means involve the adaptation of the Shannon function to obtain the worst case error estimate. The method is illustrated by classification tasks with simulated and real data.


k-fold cross validation Machine learning Visual analytics Visualization Multidimensional data Shannon function Worst case Error estimate Error rate General line coordinates Linear classifier Hybrid algorithm Interactive algorithm 


  1. 1.
    D. Anguita, A. Ghio, S. Ridella, D. Sterpi, K-fold cross validation for error rate estimate in support vector machines, in DMIN (2009 Jan), pp. 291–297Google Scholar
  2. 2.
    S. Arlot, M. Lerasle, Choice of V for V-fold cross-validation in least-squares density estimation. J. Mach. Learn. Res. 17(208), 1–50 (2016)MathSciNetzbMATHGoogle Scholar
  3. 3.
    A. Blum, A. Kalai, J. Langford, Beating the hold-out: bounds for k-fold and progressive cross validation, in Proceedings of the Twelfth Annual Conference on Computational Learning Theory (ACM, 1999 Jul 6), pp. 203–208Google Scholar
  4. 4.
    K.P. Bennett, C. Campbell, Support vector machines: hype or hallelujah? ACM SIGKDD Explorations Newsl 2(2), 1–13 (2000)CrossRefGoogle Scholar
  5. 5.
    K.P. Bennett, E.J. Bredensteiner, Duality and geometry in SVM classifiers, in ICML (2000 Jun 29), pp. 57–64Google Scholar
  6. 6.
    Y. Bengio, Y. Grandvalet, No unbiased estimator of the variance of k-fold cross-validation. J. Mach. Learn. Res. 5, 1089–1105 (2004)Google Scholar
  7. 7.
    T.G. Dietterich, Approximate statistical tests for comparing supervised classification learning algorithms. Neural Comput. 10(7), 1895–1923 (1998)CrossRefGoogle Scholar
  8. 8.
    W. Duch, R. Adamczak, K. Grąbczewski, K. Grudziński, N. Jankowski, A. Naud, Extraction of Knowledge from Data Using Computational Intelligence Methods (Copernicus University, Toruń, Poland, 2000).
  9. 9.
    Y. Grandvalet, Y. Bengio, Hypothesis Testing for Cross-Validation (Montreal Universite de Montreal, Operationnelle DdIeR, 2006 Aug 29), p. 1285Google Scholar
  10. 10.
    B. Gu, V.S. Sheng, K.Y. Tay, W. Romano, S. Li, Cross validation through two-dimensional solution surface for cost-sensitive SVM. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1103–1121 (2017)CrossRefGoogle Scholar
  11. 11.
    G. Hansel, Sur le nombre des functions Booléenes monotones de n variables. C.R. Acad. Sci., Paris, 262(20), 1088–1090 (1966)Google Scholar
  12. 12.
    J. Jolliffe, Principal of Component Analysis (Springer, 1986)Google Scholar
  13. 13.
    N.A. Karpova, Some properties of Shannon functions. Math. Notes Acad. Sci. USSR 8(5), 843–849 (1970)MathSciNetzbMATHGoogle Scholar
  14. 14.
    T. Kohonen, Self-Organizing Maps (Springer, Berlin, Germany, 1995)CrossRefGoogle Scholar
  15. 15.
    A. Inselberg, Parallel Coordinates: Visual Multidimensional Geometry and Its Applications (Springer, 2009)Google Scholar
  16. 16.
    B. Kovalerchuk, E. Triantaphyllou, A. Despande, E. Vityaev, Interactive learning of monotone Boolean functions. Inf. Sci. 94(1–4), 87–118 (1996)CrossRefGoogle Scholar
  17. 17.
    B. Kovalerchuk, Quest for rigorous combining probabilistic and fuzzy logic approaches for computing with words, in On Fuzziness. A Homage to Lotfi A. Zadeh, vol. 1 (Studies in Fuzziness and Soft Computing Vol. 216), edited by R. Seising, E. Trillas, C. Moraga, S. Termini (Springer, Berlin, New York, 2013), pp. 333–344Google Scholar
  18. 18.
    B. Kovalerchuk, Visualization of multidimensional data with collocated paired coordinates and general line coordinates, in Proceedings of SPIE 2014, vol. 9017.
  19. 19.
    B. Kovalerchuk, Visual cognitive algorithms for high-dimensional data and super-intelligence challenges. J.: Cogn. Syst. Res. 45, 95–108 (2017)Google Scholar
  20. 20.
    B. Kovalerchuk, V. Grishin, Adjustable general line coordinates for visual knowledge discovery in n-D data. Inf. Vis. (2017). Scholar
  21. 21.
    B. Kovalerchuk, M. Kovalerchuk, Toward virtual data scientist, in Proceedings of the 2017 International Joint Conference on Neural Networks (Anchorage, AK, USA, 14–19 May 2017), pp. 3073–3080Google Scholar
  22. 22.
    B. Kovalerchuk, D. Dovhalets, Constructing interactive visual classification, clustering and dimension reduction models for n-D data. Informatics 4(23) (2017).–9709/4/3/23
  23. 23.
    V. Kreinovich (ed.), Uncertainty Modeling, Studies in Computational Intelligence, vol. 683 (Springer, 2017)Google Scholar
  24. 24.
    M. Lichman, UCI Machine Learning Repository (University of California, School of Information and Computer Science, Irvine, CA, 2013). Accessed on 15 June 2017
  25. 25.
    J. Kruskal, M. Wish, Multidimensional Scaling (Sage Publications, 1978)Google Scholar
  26. 26.
    M. Lichman, UCI Machine Learning Repository (University of California, School of Information and Computer Science, Irvine, CA, 2013). Parkinson’s, Accessed on 15 June 2017
  27. 27.
    T. Mitchell, Introduction to machine learning, in Machine Learning (McGraw-Hill, Columbus, 1997)Google Scholar
  28. 28.
    T.J. Mitchell, S.Y. Chen, R.D. Macredie, Hypermedia learning and prior knowledge: domain expertise vs. system expertise. J. Comput. Assist. Learn. 21(1), 53–64 (2005)Google Scholar
  29. 29.
    J.G. Moreno-Torres, J.A. Sáez, F. Herrera, Study on the impact of partition-induced dataset shift on k-fold cross validation. IEEE Trans. Neural Netw. Learn. Syst. 23(8), 1304–1312 (2012)CrossRefGoogle Scholar
  30. 30.
    C.E. Shannon, The synthesis of two-terminal switching circuits. Bell Syst. Tech. J. 28, 59–98 (1949)MathSciNetCrossRefGoogle Scholar
  31. 31.
    J. Sharko, G. Grinstein, K. Marx, Vectorized Radviz and its application to multiple cluster datasets. IEEE Trans. Vis. Comput. Graph. 14(6), 1427–1444 (2008)Google Scholar
  32. 32.
    H. Shimodaira, Improving predictive inference under covariate shift by weighting the log-likelihood function. J. Stat. Plan. Inference 90(2), 227–244 (2000)MathSciNetCrossRefGoogle Scholar
  33. 33.
    G. Turán, F. Vatan, On the computation of Boolean functions by analog circuits of bounded fan-in, in Proceedings of the 35th Annual Symposium on Foundations of Computer Science, 1994 (IEEE, 1994 Nov 20), pp. 553–564Google Scholar
  34. 34.
    V. Vapnik, A. Vashist, A new learning paradigm: learning using privileged information. Neural Netw. 22(5), 544–557 (2009)CrossRefGoogle Scholar
  35. 35.
    T.T. Wong, Parametric methods for comparing the performance of two classification algorithms evaluated by k-fold cross validation on multiple data sets. Pattern Recogn. 65, 97–107 (2017)CrossRefGoogle Scholar
  36. 36.
    H. Yin, ViSOM-a novel method for multivariate data projection and structure visualization. IEEE Trans. Neural Netw. 13(1), 237–243 (2002)MathSciNetCrossRefGoogle Scholar
  37. 37.
    L.A. Zadeh (ed.), Computing with words in Information/Intelligent systems 1: Foundations. Physica (2013)Google Scholar
  38. 38.
    P. Zhang, Model selection via multifold cross validation. Ann. Stat. 1, 299–313 (1993)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Department of Computer ScienceCentral Washington UniversityEllensburgUSA

Personalised recommendations