Abstract
Deep learning methods have reached state-of-the-art performance in cardiac image segmentation. Currently, the main bottleneck towards their effective translation into clinics requires assuring continuous high model performance and segmentation results. In this work, we present a novel learning framework to monitor the performance of heart segmentation models in the absence of ground truth. Formulated as an anomaly detection problem, the monitoring framework allows deriving surrogate quality measures for a segmentation and allows flagging suspicious results. We propose two different types of quality measures, a global score and a pixel-wise map. We demonstrate their use by reproducing the final rankings of a cardiac segmentation challenge in the absence of ground truth. Results show that our framework is accurate, fast, and scalable, confirming it is a viable option for quality control monitoring in clinical practice and large population studies.
This work was supported through funding from the Monaco Government.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Audibert, J., Michiardi, P., Guyard, F., Marti, S., Zuluaga, M.A.: USAD: unsupervised anomaly detection on multivariate time series. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3395–3404 (2020)
Bai, W., et al.: Automated cardiovascular magnetic resonance image analysis with fully convolutional networks. J. Cardiovasc. Magn. Reson. 20(1), 65 (2018)
Baumgartner, C.F., Koch, L.M., Pollefeys, M., Konukoglu, E.: An exploration of 2D and 3D deep learning techniques for cardiac MR image segmentation. In: Pop, M., et al. (eds.) STACOM 2017. LNCS, vol. 10663, pp. 111–119. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-75541-0_12
Bergmann, P., Löwe, S., Fauser, M., Sattlegger, D., Steger, C.: Improving unsupervised defect segmentation by applying structural similarity to autoencoders. In: 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, pp. 372–380 (2019)
Bernard, O., et al.: Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: is the problem solved? IEEE Trans. Med. Imaging 37(11), 2514–2525 (2018)
Isensee, F., Jaeger, P.F., Full, P.M., Wolf, I., Engelhardt, S., Maier-Hein, K.H.: Automatic cardiac disease assessment on cine-MRI via time-series segmentation and domain specific features. In: Pop, M., et al. (eds.) STACOM 2017. LNCS, vol. 10663, pp. 120–129. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-75541-0_13
Khened, M., Alex, V., Krishnamurthi, G.: Densely connected fully convolutional network for short-axis cardiac cine MR image segmentation and heart diagnosis using random forest. In: Pop, M., et al. (eds.) STACOM 2017. LNCS, vol. 10663, pp. 140–151. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-75541-0_15
Kohlberger, T., Singh, V., Alvino, C., Bahlmann, C., Grady, L.: Evaluating segmentation error without ground truth. In: Ayache, N., Delingette, H., Golland, P., Mori, K. (eds.) MICCAI 2012, Part I. LNCS, vol. 7510, pp. 528–536. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33415-3_65
Oktay, O., et al.: Anatomically constrained neural networks (ACNNs): application to cardiac image enhancement and segmentation. IEEE Trans. Med. Imaging 37(2), 384–395 (2017)
Painchaud, N., Skandarani, Y., Judge, T., Bernard, O., Lalande, A., Jodoin, P.M.: Cardiac segmentation with strong anatomical guarantees. IEEE Trans. Med. Imaging 39(11), 3703–3713 (2020)
Puyol-Antón, E., et al.: Automated quantification of myocardial tissue characteristics from native t1 mapping using neural networks with uncertainty-based quality-control. J. Cardiovasc. Magn. Reson. 22(1), 1–15 (2020)
Robinson, R., et al.: Real-time prediction of segmentation quality. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018, Part IV. LNCS, vol. 11073, pp. 578–585. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00937-3_66
Robinson, R., et al.: Automated quality control in image segmentation: application to the UK biobank cardiovascular magnetic resonance imaging study. J. Cardiovasc. Magn. Reson. 21(1), 1–14 (2019)
Ruijsink, B., et al.: Fully automated, quality-controlled cardiac analysis from CMR. JACC Cardiovasc. Imaging 13(3), 684–695 (2020)
Sculley, D., et al.: Hidden technical debt in machine learning systems. Adv. Neural Inf. Process. Syst. 28, 2503–2511 (2015)
Sudre, C.H., Li, W., Vercauteren, T., Ourselin, S., Jorge Cardoso, M.: Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations. In: Cardoso, M.J., et al. (eds.) DLMIA/ML-CDS -2017. LNCS, vol. 10553, pp. 240–248. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-67558-9_28
Grinias, E., Tziritas, G.: Fast fully-automatic cardiac segmentation in MRI using MRF model optimization, substructures tracking and B-spline smoothing. In: Pop, M., et al. (eds.) STACOM 2017. LNCS, vol. 10663, pp. 91–100. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-75541-0_10
Valindria, V.V., et al.: Reverse classification accuracy: predicting segmentation performance in the absence of ground truth. IEEE Trans. Med. Imaging 36(8), 1597–1606 (2017)
Yang, X., Bian, C., Yu, L., Ni, D., Heng, P.-A.: Class-balanced deep neural network for automatic ventricular structure segmentation. In: Pop, M., et al. (eds.) STACOM 2017. LNCS, vol. 10663, pp. 152–160. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-75541-0_16
Yue, Q., Luo, X., Ye, Q., Xu, L., Zhuang, X.: Cardiac segmentation from LGE MRI using deep neural network incorporating shape and spatial priors. In: Shen, D., et al. (eds.) MICCAI 2019, Part II. LNCS, vol. 11765, pp. 559–567. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32245-8_62
Zuluaga, M.A., Burgos, N., Mendelson, A.F., Taylor, A.M., Ourselin, S.: Voxelwise atlas rating for computer assisted diagnosis: application to congenital heart diseases of the great arteries. Med. Image Anal. 26(1), 185–194 (2015)
Zuluaga, M.A., Cardoso, M.J., Modat, M., Ourselin, S.: Multi-atlas propagation whole heart segmentation from MRI and CTA using a local normalised correlation coefficient criterion. In: Ourselin, S., Rueckert, D., Smith, N. (eds.) FIMH 2013. LNCS, vol. 7945, pp. 174–181. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-38899-6_21
Acknowledgments
The authors would like to thank Christian Baumgartner, Elios Grinias, Jelmer M. Wolterink, and Clement Zotti for their help in the reproduction of the ACDC Challenge rankings by sharing their code or results and, overall, through their valuable advice.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Galati, F., Zuluaga, M.A. (2021). Efficient Model Monitoring for Quality Control in Cardiac Image Segmentation. In: Ennis, D.B., Perotti, L.E., Wang, V.Y. (eds) Functional Imaging and Modeling of the Heart. FIMH 2021. Lecture Notes in Computer Science(), vol 12738. Springer, Cham. https://doi.org/10.1007/978-3-030-78710-3_11
Download citation
DOI: https://doi.org/10.1007/978-3-030-78710-3_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-78709-7
Online ISBN: 978-3-030-78710-3
eBook Packages: Computer ScienceComputer Science (R0)