Attaining Human-Level Performance with Atlas Location Autocontext for Anatomical Landmark Detection in 3D CT Data

  • Alison Q. O’NeilEmail author
  • Antanas Kascenas
  • Joseph Henry
  • Daniel Wyeth
  • Matthew Shepherd
  • Erin Beveridge
  • Lauren Clunie
  • Carrie Sansom
  • Evelina Šeduikytė
  • Keith Muir
  • Ian Poole
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11131)


We present an efficient neural network method for locating anatomical landmarks in 3D medical CT scans, using atlas location autocontext in order to learn long-range spatial context. Location predictions are made by regression to Gaussian heatmaps, one heatmap per landmark. This system allows patchwise application of a shallow network, thus enabling multiple volumetric heatmaps to be predicted concurrently without prohibitive GPU memory requirements. Further, the system allows inter-landmark spatial relationships to be exploited using a simple overdetermined affine mapping that is robust to detection failures and occlusion or partial views. Evaluation is performed for 22 landmarks defined on a range of structures in head CT scans. Models are trained and validated on 201 scans. Over the final test set of 20 scans which was independently annotated by 2 human annotators, the neural network reaches an accuracy which matches the annotator variability, with similar human and machine patterns of variability across landmark classes.



Many thanks to Queen Elizabeth University Hospital, University of Glasgow, who provided many of the medical scans used for this study, including those shown in the images.


  1. 1.
    Abadi, M., et al.: Tensorflow: large-scale machine learning heterogeneous distributed systems. arXiv preprint arXiv:1603.04467 (2016)
  2. 2.
    Chen, C., Xie, W., Franke, J., Grutzner, P.A., Nolte, L.P., Zheng, G.: Automatic X-ray landmark detection and shape segmentation via data-driven joint estimation of image displacements. Med. Image Anal. 18, 487–499 (2014)CrossRefGoogle Scholar
  3. 3.
    Chollet, F.: Keras (2015)Google Scholar
  4. 4.
    Dabbah, M.A., et al.: Detection and location of 127 anatomical landmarks in diverse CT datasets. In: SPIE Medical Imaging, vol. 9034, p. 903415 (2014)Google Scholar
  5. 5.
    Dai, X., Gao, Y., Shen, D.: Online updating of context-aware landmark detectors for prostate localization in daily treatment CT images. Med. Phys. 42(5), 2594–2606 (2015)CrossRefGoogle Scholar
  6. 6.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. CVPR 1, 886–893 (2005)Google Scholar
  7. 7.
    Dong, C., Chen, Y.W., Lin, C.L.: Non-rigid registration with constraint of anatomical landmarks for assessment of locoregional therapy. In: IEEE International Conference on Information and Automation (2015)Google Scholar
  8. 8.
    Fan, X., Zheng, K., Lin, Y., Wang, S.: Combining local appearance and holistic view: dual-source deep neural networks for human pose estimation. In: CVPR (2015)Google Scholar
  9. 9.
    Gao, Y., Shen, D.: Context-aware anatomical landmark detection: application to deformable model initialization in prostate CT images. In: Wu, G., Zhang, D., Zhou, L. (eds.) MLMI 2014. LNCS, vol. 8679, pp. 165–173. Springer, Cham (2014). Scholar
  10. 10.
    Gao, Y., Shen, D.: Collaborative regression-based anatomical landmark detection. Phys. Med. Biol. 60(24), 9377–9401 (2016)CrossRefGoogle Scholar
  11. 11.
    Ghesu, F.C., Georgescu, B., Grbic, S., Maier, A.K., Hornegger, J., Comaniciu, D.: Robust multi-scale anatomical landmark detection in incomplete 3D-CT data. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10433, pp. 194–202. Springer, Cham (2017). Scholar
  12. 12.
    Han, D., Gao, Y., Yaozong, G., Yap, P.T., Shen, D.: Robust anatomical landmark detection with application to MR brain image registration. Comput. Med. Imaging Graph. 46(3), 277–290 (2015)CrossRefGoogle Scholar
  13. 13.
    He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: ICCV, pp. 1026–1034 (2015)Google Scholar
  14. 14.
    Hellier, P., Barillot, C.: Coupling dense and landmark-based approaches for nonrigid registration. IEEE Trans. Med. Imaging 22(2), 217–227 (2003)CrossRefGoogle Scholar
  15. 15.
    Ho, T.K.: The random subspace method for constructing decision forests. IEEE Trans. Pattern Anal. Mach. Intell. 20(8), 832–844 (1998). Scholar
  16. 16.
    Ibragimov, B., Likar, B., Pernuš, F., Vrtovec, T.: Shape representation for efficient landmark-based segmentation in 3-D. IEEE Trans. Med. Imaging 33(4), 861–874 (2014)CrossRefGoogle Scholar
  17. 17.
    Ibragimov, B., Likar, B., Pernuš, F., Vrtovec, T.: A game-theoretic framework for landmark-based image segmentation. IEEE Trans. Med. Imaging 31(9), 1761–1776 (2012)CrossRefGoogle Scholar
  18. 18.
    Jimenez-Del-Toro, O., et al.: Cloud-based evaluation of anatomical structure segmentation and landmark detection algorithms: VISCERAL anatomy benchmarks. IEEE Trans. Med. Imaging 35(11), 2459–2475 (2016)CrossRefGoogle Scholar
  19. 19.
    Kamnitsas, K., et al.: Efficient multi-scale CNN with fully connected CRF for accurate brain lesion segmentation. Med. Image Anal. 36, 61–78 (2017)CrossRefGoogle Scholar
  20. 20.
    Kingma, D.P., Ba, J.L.: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  21. 21.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)Google Scholar
  22. 22.
    Lay, N., Birkbeck, N., Zhang, J., Zhou, S.K.: Rapid multi-organ segmentation using context integration and discriminative models. In: Gee, J.C., Joshi, S., Pohl, K.M., Wells, W.M., Zöllei, L. (eds.) IPMI 2013. LNCS, vol. 7917, pp. 450–462. Springer, Heidelberg (2013). Scholar
  23. 23.
    Leavens, C., et al.: Validation of automatic landmark identification for atlas-based segmentation for radiation treatment planning of the head-and-neck region. In: SPIE Medical Imaging, vol. 6914 (2008)Google Scholar
  24. 24.
    Lisowska, A., et al.: Context-aware convolutional neural networks for stroke sign detection in non-contract CT scans. In: Annual Conference on Medical Image Understanding and Analysis, pp. 494–505 (2017)Google Scholar
  25. 25.
    Lu, X., Xu, D., Liu, D.: Robust 3D organ localization with dual learning architectures and fusion. In: Carneiro, G., et al. (eds.) LABELS/DLMIA -2016. LNCS, vol. 10008, pp. 12–20. Springer, Cham (2016). Scholar
  26. 26.
    Milletari, F., Navab, N., Ahmadi, S.A.: V-Net: full convolutional neural networks for volumetric medical image segmentation. In: IEEE Fourth International Conference on 3D Vision (3DV), pp. 565–571 (2016)Google Scholar
  27. 27.
    Oktay, O., et al.: Stratified decision forests for accurate anatomical landmark localization. IEEE Trans. Med. Imaging 36(1), 332–342 (2017)CrossRefGoogle Scholar
  28. 28.
    O’Neil, A.: Detection of Anatomical Structures in Medical Datasets. EngD Thesis, September 2016Google Scholar
  29. 29.
    O’Neil, A., Beveridge, E., Houston, G., McCormick, L., Poole, I.: Arterial tree tracking from anatomical landmarks in magnetic resonance angiography scans. In: SPIE Medical Imaging, vol. 9034 (2014)Google Scholar
  30. 30.
    O’Neil, A., Murphy, S., Poole, I.: Anatomical landmark detection in CT data by learned atlas location autocontext. In: MIUA (2015)Google Scholar
  31. 31.
    Payer, C., Štern, D., Bischof, H., Urschler, M.: Regressing heatmaps for multiple landmark localization using CNNs. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 230–238. Springer, Cham (2016). Scholar
  32. 32.
    Pfister, T., Charles, J., Zisserman, A.: Flowing convnets for human pose estimation in videos. In: IEEE ICCV, pp. 1913–1921 (2015)Google Scholar
  33. 33.
    Polzin, T., Rühaak, J., Wernera, R., Handels, H., Modersitzki, J.: Lung registration using automatically detected landmarks. Methods Inf. Med. 53(4), 250–256 (2014)CrossRefGoogle Scholar
  34. 34.
    Riegler, G., Fersti, D., Ruther, M., Bischof, H.: Hough networks for head pose estimation and facial feature localization. In: BMVC (2014)Google Scholar
  35. 35.
    Rohr, K., Stiehl, H.S., Sprengel, R., Buzug, T.M., Weese, J., Kuhn, M.H.: Landmark-based elastic registration using approximating thin-plate splines. IEEE Trans. Med. Imaging 20(6), 526–534 (2001)CrossRefGoogle Scholar
  36. 36.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). Scholar
  37. 37.
    Toshev, A., Szegedy, C.: DeepPose: Human pose estimation via deep neural networks. In: CVPR (2014)Google Scholar
  38. 38.
    Tu, Z., Bai, X.: Auto-context and its application to high-level vision tasks and 3D brain segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 32(10), 1744–1757 (2010)CrossRefGoogle Scholar
  39. 39.
    Wang, C.W., et al.: Evaluation and comparison of anatomical landmark detection methods for cephalometric x-ray images: a grand challenge. IEEE Trans. Med. Imaging 34(9), 1890–1900 (2015)CrossRefGoogle Scholar
  40. 40.
    Zhang, J., Gao, Y., Gao, Y., Munsell, B.C., Shen, D.: Detecting anatomical landmarks for fast Alzheimer’s disease diagnosis. IEEE Trans. Med. Imaging 35(12), 2524–2533 (2016)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Alison Q. O’Neil
    • 1
    Email author
  • Antanas Kascenas
    • 1
  • Joseph Henry
    • 1
  • Daniel Wyeth
    • 1
  • Matthew Shepherd
    • 1
  • Erin Beveridge
    • 1
  • Lauren Clunie
    • 1
  • Carrie Sansom
    • 1
  • Evelina Šeduikytė
    • 1
  • Keith Muir
    • 2
  • Ian Poole
    • 1
  1. 1.Canon Medical Research EuropeEdinburghUK
  2. 2.Queen Elizabeth University HospitalUniversity of GlasgowGlasgowUK

Personalised recommendations