The Empirical Variance Estimator for Computer Aided Diagnosis: Lessons for Algorithm Validation

  • Alex F. Mendelson
  • Maria A. Zuluaga
  • Lennart Thurfjell
  • Brian F. Hutton
  • Sébastien Ourselin
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8674)


Computer aided diagnosis is an established field in medical image analysis; a great deal of effort goes into the development and refinement of pipelines to achieve greater performance. This improvement is dependent on reliable comparison, which is intimately related to variance estimation. For supervised methods, this can be confounded by statistical issues at the comparatively small sample sizes typical of the field. Given the importance of reliable comparison to pipeline development, this issue has received relatively little attention. As a solution, we advocate an empirical variance estimator based on validation within disjoint subsets of the available data. Using Alzheimer’s disease classification in the ADNI dataset as an examplar, we investigate the behaviour of different variance estimators in a series of resampling experiments. We show that the proposed estimator is unbiased, and that it exceeds the estimates of naive approaches, which are biased down. Because the estimator avoids independence assumptions, it is able to accommodate arbitrary validation strategies and performance metrics. As it is unbiased, it is able to provide statistically convincing comparison and confidence intervals for algorithm performance. Finally, we show how the estimator can be used to compare different validation strategies, and make some recommendations about which should be used.


Cross Validation Variance Estimator Unbiased Estimator Validation Strategy Medical Image Analysis 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Cuingnet, R., Gerardin, E., Tessieras, J., Auzias, G., Lehéricy, S., Habert, M.O., Chupin, M., Benali, H., Colliot, O.: Automatic classification of patients with alzheimer’s disease from structural MRI: a comparison of ten methods using the adni database. Neuroimage 56(2), 766–781 (2011)CrossRefGoogle Scholar
  2. 2.
    Dietterich, T.G.: Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation 10(7), 1895–1923 (1998)CrossRefGoogle Scholar
  3. 3.
    Bengio, Y., Grandvalet, Y.: No unbiased estimator of the variance of k-fold cross-validation. J. Mach. Learn. Res. 5, 1089–1105 (2004)MathSciNetGoogle Scholar
  4. 4.
    Noirhomme, Q., Lesenfants, D., Gomez, F., Soddu, A., Schrouff, J., Garraux, G., Luxen, A., Phillips, C., Laureys, S.: Biased binomial assessment of cross-validated estimation of classification accuracies illustrated in diagnosis predictions. NeuroImage: Clinical 4, 687–694 (2014)CrossRefGoogle Scholar
  5. 5.
    Nadeau, C., Bengio, Y.: Inference for the generalization error. Machine Learning 52(3), 239–281 (2003)CrossRefGoogle Scholar
  6. 6.
    Grandvalet, Y., Bengio, Y.: Hypothesis testing for cross-validation. Montreal Universite de Montreal, Operationnelle DdIeR (2006)Google Scholar
  7. 7.
    Gousias, I.S., Rueckert, D., Heckemann, R.A., Dyet, L.E., Boardman, J.P., Edwards, A.D., Hammers, A.: Automatic segmentation of brain MRIs of 2-year-olds into 83 regions of interest. NeuroImage 40(2), 672–684 (2008)CrossRefGoogle Scholar
  8. 8.
    Cardoso, M.J., Leung, K., Modat, M., Keihaninejad, S., Cash, D., Barnes, J., Fox, N.C., Ourselin, S.: STEPS: Similarity and truth estimation for propagated segmentations and its application to hippocampal segmentation and brain parcelation. Medical Image Analysis 17(6), 671–684 (2013)CrossRefGoogle Scholar
  9. 9.
    Modat, M., Ridgway, G.R., Taylor, Z.A., Lehmann, M., Barnes, J., Hawkes, D.J., Fox, N.C., Ourselin, S.: Fast free-form deformation using graphics processing units. Computer Methods and Programs in Biomedicine 98(3), 278–284 (2010)CrossRefzbMATHGoogle Scholar
  10. 10.
    Chang, C.C., Lin, C.J.: LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology 2, 27:1–27:27 (2011), Software available at Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Alex F. Mendelson
    • 1
  • Maria A. Zuluaga
    • 1
  • Lennart Thurfjell
    • 2
  • Brian F. Hutton
    • 3
    • 4
  • Sébastien Ourselin
    • 1
    • 5
  1. 1.Translational Imaging Group, Centre for Medical Image ComputingUniversity College LondonLondonUK
  2. 2.GE HealthcareUppsalaSweden
  3. 3.Institute of Nuclear MedicineUniversity College LondonLondonUK
  4. 4.Centre for Medical Radiation PhysicsUniversity of WollongongAustralia
  5. 5.Dementia Research CentreUniversity College LondonUK

Personalised recommendations