Abstract
Conformal prediction (CP) generates a set of predictions for a given test sample such that the prediction set almost always contains the true label (e.g., 99.5% of the time). CP provides comprehensive predictions on possible labels of a given test sample, and the size of the set indicates how certain the predictions are (e.g., a set larger than one is ‘uncertain’). Such distinct properties of CP enable effective collaborations between human experts and medical AI models, allowing efficient intervention and quality check in clinical decision-making. In this paper, we propose a new method called Reliable-Region-Based Conformal Prediction (RR-CP), which aims to impose a stronger statistical guarantee so that an extremely low error rate (e.g., 0.5%) can be achieved in the test time, and under this constraint, the size of the prediction set is optimized to be small. We consider a small prediction set size an important measure only when the low error rate is achieved. Experiments on five public datasets show that our RR-CP performs well: with a reasonably small-sized prediction set, it achieves the user-specified low error rate significantly more frequently than exiting CP methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
The user-specified error rate \(\alpha \) should be very small for medical application cases.
- 2.
References
Acevedo, A., Merino, A., Alférez, S., Molina, Á., Boldú, L., Rodellar, J.: A dataset of microscopic peripheral blood cell images for development of automatic recognition systems. Data Brief 30 (2020)
Angelopoulos, A.N., Bates, S.: A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511 (2021)
Angelopoulos, A.N., Bates, S., Fisch, A., Lei, L., Schuster, T.: Conformal risk control. arXiv preprint arXiv:2208.02814 (2022)
Angelopoulos, A.N., Bates, S., Jordan, M. Malik, J.: Uncertainty sets for image classifiers using conformal prediction. In: International Conference on Learning Representations (2021)
Babbar, V., Bhatt, U., Weller, A.: On the utility of prediction sets in human-AI teams. arXiv preprint arXiv:2205.01411 (2022)
Bilic, P., et al.: The liver tumor segmentation benchmark (LiTS). Med. Image Anal. 84, 102680 (2023)
Efron, B., Tibshirani, R.J.: An Introduction to the Bootstrap. CRC Press (1994)
Fisch, A., Schuster, T., Jaakkola, T., Barzilay, R.: Conformal prediction sets with limited false positives. In: International Conference on Machine Learning, pp. 6514–6532. PMLR (2022)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Lu, C., Angelopoulos, A.N., Pomerantz, S.: Improving trustworthiness of AI disease severity rating in medical imaging with ordinal conformal prediction sets. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2022: 25th International Conference, Singapore, 18–22 September 2022, Proceedings, Part VIII, pp. 545–554. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-16452-1_52
Charles, L., Lemay, A., Chang, K., Höbel, K., Kalpathy-Cramer, J.: Fair conformal predictors for applications in medical imaging. Proc. AAAI Conf. Artif. Intell. 36, 12008–12016 (2022)
Romano, Y., Sesia, M., Candes, E.: Classification with valid and adaptive coverage. Adv. Neural. Inf. Process. Syst. 33, 3581–3591 (2020)
Tschandl, P., Rosendahl, C., Kittler, H.: The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Sci. Data 5(1), 1–9 (2018)
Vovk, V., Gammerman, A., Saunders, C.: Machine-learning applications of algorithmic randomness. In: International Conference on Machine Learning, pp. 444–453 (1999)
Xu, C., Xie, Y.: Conformal prediction set for time-series. arXiv preprint arXiv:2206.07851 (2022)
Yang, J., et al.: MedMNIST v2 - a large-scale lightweight benchmark for 2D and 3D biomedical image classification. Sci. Data 10(1), 41 (2023)
Acknowledgement
This work was supported in part by National Natural Science Foundation of China (62201263) and Natural Science Foundation of Jiangsu Province (BK20220949). S.W. is supported by Shanghai Sailing Programs of Shanghai Municipal Science and Technology Committee (22YF1409300).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, Y., Wang, S., Zhang, Y., Chen, D.Z. (2023). RR-CP: Reliable-Region-Based Conformal Prediction for Trustworthy Medical Image Classification. In: Sudre, C.H., Baumgartner, C.F., Dalca, A., Mehta, R., Qin, C., Wells, W.M. (eds) Uncertainty for Safe Utilization of Machine Learning in Medical Imaging. UNSURE 2023. Lecture Notes in Computer Science, vol 14291. Springer, Cham. https://doi.org/10.1007/978-3-031-44336-7_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-44336-7_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-44335-0
Online ISBN: 978-3-031-44336-7
eBook Packages: Computer ScienceComputer Science (R0)
