Multimodal Image Registration with Deep Context Reinforcement Learning

  • Kai MaEmail author
  • Jiangping Wang
  • Vivek Singh
  • Birgi Tamersoy
  • Yao-Jen Chang
  • Andreas Wimmer
  • Terrence Chen
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10433)


Automatic and robust registration between real-time patient imaging and pre-operative data (e.g. CT and MRI) is crucial for computer-aided interventions and AR-based navigation guidance. In this paper, we present a novel approach to automatically align range image of the patient with pre-operative CT images. Unlike existing approaches based on the surface similarity optimization process, our algorithm leverages the contextual information of medical images to resolve data ambiguities and improve robustness. The proposed algorithm is derived from deep reinforcement learning algorithm that automatically learns to extract optimal feature representation to reduce the appearance discrepancy between these two modalities. Quantitative evaluations on 1788 pairs of CT and depth images from real clinical setting demonstrate that the proposed method achieves the state-of-the-art performance.

Supplementary material

455905_1_En_28_MOESM1_ESM.avi (384 kb)
Supplementary material 1 (avi 384 KB)
455905_1_En_28_MOESM2_ESM.avi (384 kb)
Supplementary material 2 (avi 384 KB)
455905_1_En_28_MOESM3_ESM.avi (412 kb)
Supplementary material 3 (avi 412 KB)


  1. 1.
    Achilles, F., Ichim, A.-E., Coskun, H., Tombari, F., Noachtar, S., Navab, N.: Patient MoCap: human pose estimation under blanket occlusion for hospital monitoring applications. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9900, pp. 491–499. Springer, Cham (2016). doi: 10.1007/978-3-319-46720-7_57 CrossRefGoogle Scholar
  2. 2.
    Bauer, S., Wasza, J., Haase, S., Marosi, N., Hornegger, J.: Multi-modal surface registration for markerless initial patient setup in radiation therapy using Microsoft’s Kinect sensor. In: ICCV Workshops (2011)Google Scholar
  3. 3.
    Bellman, R.: A Markovian decision process. Indiana Univ. Math. J. 6, 679–684 (1957)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Cao, X., Gao, Y., Yang, J., Wu, G., Shen, D.: Learning-based multimodal image registration for prostate cancer radiation therapy. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9902, pp. 1–9. Springer, Cham (2016). doi: 10.1007/978-3-319-46726-9_1 CrossRefGoogle Scholar
  5. 5.
    Elmi-Terander, A., Skulason, H., Söderman, M., et al.: Surgical navigation technology based on augmented reality and integrated 3D intraoperative imaging: a spine cadaveric feasibility and accuracy study. Spine 41, 303–311 (2016)CrossRefGoogle Scholar
  6. 6.
    Ghesu, F.C., Georgescu, B., Mansi, T., Neumann, D., Hornegger, J., Comaniciu, D.: An artificial agent for anatomical landmark detection in medical images. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9902, pp. 229–237. Springer, Cham (2016). doi: 10.1007/978-3-319-46726-9_27 CrossRefGoogle Scholar
  7. 7.
    Gutiérrez-Becker, B., Mateus, D., Peter, L., Navab, N.: Learning optimization updates for multimodal registration. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9902, pp. 19–27. Springer, Cham (2016). doi: 10.1007/978-3-319-46726-9_3 CrossRefGoogle Scholar
  8. 8.
    Hasselt, H., Guez, A., Silver, D.: Deep reinforcement learning with double Q-learning. In: AAAI (2016)Google Scholar
  9. 9.
    Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: arXiv (2015)Google Scholar
  10. 10.
    Levine, S., Pastor, P., Krizhevsky, A., Quillen, D.: Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. In: ISER (2016)Google Scholar
  11. 11.
    Liao, R., Miao, S., de Tournemire, P., Grbic, S., Kamen, A., Mansi, T., Comaniciu, D.: An artificial agent for robust image registration. In: AAAI (2017)Google Scholar
  12. 12.
    Ma, K., Chang, Y.J., Singh, V.K., O’donnell, T., Wels, M., Betz, T., Wimmer, A., Chen, T.: Calibrating RGB-D sensors to medical image scanners. US Patent 9,633,435Google Scholar
  13. 13.
    Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., Hassabis, D.: Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015)CrossRefGoogle Scholar
  14. 14.
    Nutti, B., Kronander, S., Nilsing, M., Maad, K., Svensson, C., Li, H.: Depth sensor-based realtime tumor tracking for accurate radiation therapy. In: Eurographics (2014)Google Scholar
  15. 15.
    Simonovsky, M., Gutiérrez-Becker, B., Mateus, D., Navab, N., Komodakis, N.: A deep metric for multimodal registration. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9902, pp. 10–18. Springer, Cham (2016). doi: 10.1007/978-3-319-46726-9_2 CrossRefGoogle Scholar
  16. 16.
    Singh, V., Chang, Y., Ma, K., Wels, M., Soza, G., Chen, T.: Estimating a patient surface model for optimizing the medical scanning workflow. In: Golland, P., Hata, N., Barillot, C., Hornegger, J., Howe, R. (eds.) MICCAI 2014. LNCS, vol. 8673, pp. 472–479. Springer, Cham (2014). doi: 10.1007/978-3-319-10404-1_59 Google Scholar
  17. 17.
    Toews, M., Zöllei, L., Wells, W.M.: Feature-based alignment of volumetric multi-modal images. Inf. Process. Med. Imaging 23, 25–36 (2013)CrossRefGoogle Scholar
  18. 18.
    Wang, Z., de Freitas, N., Lanctot, M.: Dueling network architectures for deep reinforcement learning. In: ICML (2016)Google Scholar
  19. 19.
    Xiao, D., Luo, H., Jia, F., Zhang, Y., Li, Y., Guo, X., Cai, W., Fang, C., Fan, Y., Zheng, H., Hu, Q.: A Kinect camera based navigation system for percutaneous abdominal puncture. Phys. Med. Biol. 61, 5687–5705 (2016)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Kai Ma
    • 1
    Email author
  • Jiangping Wang
    • 1
  • Vivek Singh
    • 1
  • Birgi Tamersoy
    • 2
  • Yao-Jen Chang
    • 1
  • Andreas Wimmer
    • 2
  • Terrence Chen
    • 1
  1. 1.Medical Imaging TechnologiesSiemens Medical Solutions USA, Inc.PrincetonUSA
  2. 2.Siemens Healthcare GmbHForchheimGermany

Personalised recommendations