Skip to main content
SpringerLink
Log in

Calibration of Deep Medical Image Classifiers: An Empirical Comparison Using Dermatology and Histopathology Datasets

  • Conference paper
  • First Online:
Uncertainty for Safe Utilization of Machine Learning in Medical Imaging (UNSURE 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13563))

  • 471 Accesses

Abstract

As deep learning classifiers become ever more widely deployed for medical image analysis tasks, issues of predictive calibration need to be addressed. Mis-calibration is the deviation between predictive probability (confidence) and classification correctness. Well-calibrated classifiers enable cost-sensitive and selective decision-making. This paper presents an empirical investigation of calibration methods on two medical image datasets (multi-class dermatology and binary histopathology image classification). We show the effect of temperature scaling with temperature optimized using various measures of calibration replacing the standard negative log-likelihood. We do so not only for networks trained using one-hot encoding and cross-entropy loss, but also using focal loss and label smoothing. We compare these with two Bayesian methods. Results suggest little or no advantage to the use of alternative calibration metrics for tuning temperature. Temperature scaling of networks trained using focal loss (with appropriate hyperparameters) provided strong results in terms of both calibration and accuracy across both datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 44.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 59.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    GitHub Repository: https://github.com/UoD-CVIP/Medical_Calibration.

References

  1. Bejnordi, B.E., et al.: Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. J. Am. Med. Assoc. 318(22), 2199–2210 (2017)

    Article  Google Scholar 

  2. Blundell, C., Cornebise, J., Kavukcuoglu, K., Wierstra, D.: Weight uncertainty in neural networks. In: Proceedings of the 32nd International Conference on Machine Learning, pp. 1613–1622. PMLR (2015)

    Google Scholar 

  3. Botev, A., Ritter, H., Barber, D.: Practical Gauss-Newton optimisation for deep learning. In: Proceedings of the 34th International Conference on Machine Learning, pp. 557–565. PMLR (2017)

    Google Scholar 

  4. Carse, J., McKenna, S.: Active learning for patch-based digital pathology using convolutional neural networks to reduce annotation costs. In: Reyes-Aldasoro, C.C., Janowczyk, A., Veta, M., Bankhead, P., Sirinukunwattana, K. (eds.) ECDP 2019. LNCS, vol. 11435, pp. 20–27. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-23937-4_3

    Chapter  Google Scholar 

  5. Carse, J., et al.: Robust selective classification of skin lesions with asymmetric costs. In: Sudre, C.H., Licandro, R., Baumgartner, C., Melbourne, A., Dalca, A., Hutter, J., Tanno, R., Abaci Turk, E., Van Leemput, K., Torrents Barrena, J., Wells, W.M., Macgowan, C. (eds.) UNSURE/PIPPI -2021. LNCS, vol. 12959, pp. 112–121. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87735-4_11

    Chapter  Google Scholar 

  6. Codella, N.C., et al.: Skin lesion analysis toward melanoma detection: a challenge at the 2017 international symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC). In: 2018 IEEE 15th International Symposium on Biomedical Imaging, pp. 168–172. IEEE (2018)

    Google Scholar 

  7. Combalia, M., et al.: BCN20000: dermoscopic lesions in the wild. arXiv preprint arXiv:1908.02288 (2019)

  8. Dai, Z., Low, B.K.H., Jaillet, P.: Federated Bayesian optimization via Thompson sampling. Adv. Neural Inf. Process. Syst. 33, 9687–9699 (2020)

    Google Scholar 

  9. Daxberger, E., Kristiadi, A., Immer, A., Eschenhagen, R., Bauer, M., Hennig, P.: Laplace redux-effortless Bayesian deep learning. Adv. Neural Inf. Process. Syst. 34, 20089–20103 (2021)

    Google Scholar 

  10. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)

    Google Scholar 

  11. Frenkel, L., Goldberger, J.: Network calibration by class-based temperature scaling. In: 2021 29th European Signal Processing Conference, pp. 1486–1490. IEEE (2021)

    Google Scholar 

  12. Gawlikowski, J., et al.: A survey of uncertainty in deep neural networks. arXiv preprint arXiv:2107.03342 (2021)

  13. Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: Proceedings of the 34th International Conference on Machine Learning, pp. 1321–1330. PMLR (2017)

    Google Scholar 

  14. Hendrycks, D., Mazeika, M., Dietterich, T.: Deep anomaly detection with outlier exposure. arXiv preprint arXiv:1812.04606 (2018)

  15. Hendrycks, D., Mu, N., Cubuk, E.D., Zoph, B., Gilmer, J., Lakshminarayanan, B.: AugMix: a simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv:1912.02781 (2019)

  16. Islam, M., Glocker, B.: Spatially varying label smoothing: capturing uncertainty from expert annotations. In: Feragen, A., Sommer, S., Schnabel, J., Nielsen, M. (eds.) IPMI 2021. LNCS, vol. 12729, pp. 677–688. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-78191-0_52

    Chapter  Google Scholar 

  17. Kwon, Y., Won, J.H., Kim, B.J., Paik, M.C.: Uncertainty quantification using Bayesian neural networks in classification: application to biomedical image segmentation. Comput. Stat. Data Anal. 142, 106816 (2020)

    Article  MathSciNet  Google Scholar 

  18. Liang, G., Zhang, Y., Wang, X., Jacobs, N.: Improved trainable calibration method for neural networks on medical imaging classification. arXiv preprint arXiv:2009.04057 (2020)

  19. Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2980–2988 (2017)

    Google Scholar 

  20. Liu, D.C., Nocedal, J.: On the limited memory BFGS method for large scale optimization. Math. Program. 45, 503–528 (1989)

    Article  MathSciNet  Google Scholar 

  21. MacKay, D.J.: Bayesian interpolation. Neural Comput. 4(3), 415–447 (1992)

    Article  Google Scholar 

  22. Maron, R.C., et al.: Systematic outperformance of 112 dermatologists in multiclass skin cancer image classification by convolutional neural networks. Eur. J. Cancer 119, 57–65 (2019)

    Article  Google Scholar 

  23. Mukhoti, J., Kulharia, V., Sanyal, A., Golodetz, S., Torr, P., Dokania, P.: Calibrating deep neural networks using focal loss. Adv. Neural Inf. Process. Syst. 33, 15288–15299 (2020)

    Google Scholar 

  24. Müller, R., Kornblith, S., Hinton, G.E.: When does label smoothing help? In: Advances in Neural Information Processing Systems, vol. 32 (2019)

    Google Scholar 

  25. Parzen, E.: On estimation of a probability density function and mode. Ann. Math. Stat. 33(3), 1065–1076 (1962)

    Article  MathSciNet  Google Scholar 

  26. Platt, J., et al.: Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Adv. Large Margin Classifiers 10(3), 61–74 (1999)

    Google Scholar 

  27. Roelofs, R., Cain, N., Shlens, J., Mozer, M.C.: Mitigating bias in calibration error estimation. In: International Conference on Artificial Intelligence and Statistics, pp. 4036–4054. PMLR (2022)

    Google Scholar 

  28. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826 (2016)

    Google Scholar 

  29. Tan, M., Le, Q.: EfficientNet: rethinking model scaling for convolutional neural networks. In: Proceedings of the 36th International Conference on Machine Learning International Conference on Machine Learning, pp. 6105–6114. PMLR (2019)

    Google Scholar 

  30. Tschandl, P., Rosendahl, C., Kittler, H.: The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Sci. Data 5(1), 1–9 (2018)

    Article  Google Scholar 

  31. Ulmer, D., Meijerink, L., Cinà, G.: Trust issues: uncertainty estimation does not enable reliable OOD detection on medical tabular data. In: Machine Learning for Health, pp. 341–354. PMLR (2020)

    Google Scholar 

  32. Veeling, B.S., Linmans, J., Winkens, J., Cohen, T., Welling, M.: Rotation equivariant CNNs for digital pathology. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11071, pp. 210–218. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00934-2_24

    Chapter  Google Scholar 

  33. Zhang, J., Kailkhura, B., Han, T.Y.J.: Mix-n-Match: ensemble and compositional methods for uncertainty calibration in deep learning. In: Proceedings of the 37th International Conference on Machine Learning, pp. 11117–11128. PMLR (2020)

    Google Scholar 

Download references

Acknowledgments

J. Carse was supported by the UK Engineering and Physical Sciences Research Council (EPSRC Training Grant EP/N509632/1). This paper also reports independent research partly funded by the National Institute for Health Research (Artificial Intelligence, Deep learning for effective triaging of skin disease in the NHS, AI AWARD01901) and NHSX. The views expressed in this publication are those of the authors and not necessarily those of the National Institute for Health Research, NHSX or the Department of Health and Social Care.

Author information

Authors and Affiliations

  1. CVIP, School of Science and Engineering, University of Dundee, Scotland, UK

    Jacob Carse, Andres Alvarez Olmo & Stephen McKenna

Authors
  1. Jacob Carse
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andres Alvarez Olmo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stephen McKenna
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stephen McKenna .

Editor information

Editors and Affiliations

  1. University College London, London, UK

    Carole H. Sudre

  2. University of Tübingen, Tübingen, Germany

    Christian F. Baumgartner

  3. Massachusetts General Hospital, Charlestown, MA, USA

    Adrian Dalca

  4. Imperial College London, London, UK

    Chen Qin

  5. Google DeepMind, London, UK

    Ryutaro Tanno

  6. Technical University of Denmark, Kongens Lyngby, Denmark

    Koen Van Leemput

  7. Harvard Medical School, Boston, MA, USA

    William M. Wells III

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Carse, J., Alvarez Olmo, A., McKenna, S. (2022). Calibration of Deep Medical Image Classifiers: An Empirical Comparison Using Dermatology and Histopathology Datasets. In: Sudre, C.H., et al. Uncertainty for Safe Utilization of Machine Learning in Medical Imaging. UNSURE 2022. Lecture Notes in Computer Science, vol 13563. Springer, Cham. https://doi.org/10.1007/978-3-031-16749-2_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-16749-2_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-16748-5

  • Online ISBN: 978-3-031-16749-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation