Endoscopic Depth Measurement and Super-Spectral-Resolution Imaging

  • Jianyu LinEmail author
  • Neil T. Clancy
  • Yang Hu
  • Ji Qi
  • Taran Tatla
  • Danail Stoyanov
  • Lena Maier-Hein
  • Daniel S. Elson
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10434)


Intra-operative measurements of tissue shape and multi/hyperspectral information have the potential to provide surgical guidance and decision making support. We report an optical probe based system to combine sparse hyperspectral measurements and spectrally-encoded structured lighting (SL) for surface measurements. The system provides informative signals for navigation with a surgical interface. By rapidly switching between SL and white light (WL) modes, SL information is combined with structure-from-motion (SfM) from white light images, based on SURF feature detection and Lucas-Kanade (LK) optical flow to provide quasi-dense surface shape reconstruction with known scale in real-time. Furthermore, “super-spectral-resolution” was realized, whereby the RGB images and sparse hyperspectral data were integrated to recover dense pixel-level hyperspectral stacks, by using convolutional neural networks to upscale the wavelength dimension. Validation and demonstration of this system is reported on ex vivo/in vivo animal/human experiments.


SURF Feature Detection Surgical Interface WL Images Tissue Shape Narrow Band Imaging 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Ethics Statement

The ethics approval for human study was covered by Central London Research Ethics Committee (reference No. 10/H0718/55), animal study was conducted under UK Home Office license (reference No. 70/24843, 70/7508, 70/6927, 8012639).

Supplementary material

451304_1_En_5_MOESM1_ESM.mp4 (972 kb)
Supplementary material 1 (MP4 972 kb)
451304_1_En_5_MOESM2_ESM.mp4 (2.7 mb)
Supplementary material 2 (MP4 2723 kb)
451304_1_En_5_MOESM3_ESM.mp4 (151 kb)
Supplementary material 3 (MP4 150 kb)
451304_1_En_5_MOESM4_ESM.mp4 (2.5 mb)
Supplementary material 4 (MP4 2556 kb)
451304_1_En_5_MOESM5_ESM.mp4 (6.3 mb)
Supplementary material 5 (MP4 6476 kb)
451304_1_En_5_MOESM6_ESM.mp4 (771 kb)
Supplementary material 6 (MP4 770 kb)


  1. 1.
    Maier-Hein, L., Mountney, P., Bartoli, A., Elhawary, H., Elson, D., Groch, A., Kolb, A., Rodrigues, M., Sorger, J., Speidel, S., Stoyanov, D.: Optical techniques for 3D surface reconstruction in computer-assisted laparoscopic surgery. Med. Image Anal. 17, 974–996 (2013)CrossRefGoogle Scholar
  2. 2.
    Lu, G., Fei, B.: Medical hyperspectral imaging: a review. J. Biomed. Opt. 19(1), 010901 (2014)CrossRefGoogle Scholar
  3. 3.
    Clancy, N.T., Arya, S., Stoyanov, D., Singh, M., Hanna, G.B., Elson, D.S.: Intraoperative measurement of bowel oxygen saturation using a multispectral imaging laparoscope. Biomed. Opt. Exp. 6, 4179–4190 (2015)CrossRefGoogle Scholar
  4. 4.
    Lin, J., Clancy, N.T., Sun, X., Qi, J., Janatka, M., Stoyanov, D., Elson, D.S.: Probe-based rapid hybrid hyperspectral and tissue surface imaging aided by fully convolutional networks. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9902, pp. 414–422. Springer, Cham (2016). doi: 10.1007/978-3-319-46726-9_48CrossRefGoogle Scholar
  5. 5.
    Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2003)zbMATHGoogle Scholar
  6. 6.
    Du, X., Clancy, N., Arya, S., Hanna, G.B., Kelly, J., Elson, D.S., Stoyanov, D.: Robust surface tracking combining features, intensity and illumination compensation. Int. J. Comput. Assist. Radiol. Surg. 10, 1915–1926 (2015)CrossRefGoogle Scholar
  7. 7.
    Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1874–1883 (2016)Google Scholar
  8. 8.
    Oktay, O., et al.: Multi-input cardiac image super-resolution using convolutional neural networks. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9902, pp. 246–254. Springer, Cham (2016). doi: 10.1007/978-3-319-46726-9_29CrossRefGoogle Scholar
  9. 9.
    Clancy, N.T., Stoyanov, D., Yang, G.-Z., Elson, D.S.: Stroboscopic illumination scheme for seamless 3D endoscopy, p. 82140M-82146 (2012)Google Scholar
  10. 10.
    Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., Kudlur, M., Levenberg, J., Monga, R., Moore, S., Murray, D.G., Steiner, B., Tucker, P., Vasudevan, V., Warden, P., Wicke, M., Yu, Y., Zheng, X.: TensorFlow: a system for large-scale machine learning. ArXiv e-prints 1605 (2016)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Jianyu Lin
    • 1
    • 2
    Email author
  • Neil T. Clancy
    • 1
    • 3
  • Yang Hu
    • 1
    • 2
  • Ji Qi
    • 1
    • 3
  • Taran Tatla
    • 6
  • Danail Stoyanov
    • 4
    • 5
  • Lena Maier-Hein
    • 7
  • Daniel S. Elson
    • 1
    • 3
  1. 1.Hamlyn Centre for Robotic SurgeryImperial College LondonLondonUK
  2. 2.Department of ComputingImperial College LondonLondonUK
  3. 3.Department of Surgery and CancerImperial College LondonLondonUK
  4. 4.Department of Otolaryngology - Head and Neck SurgeryNorthwick Park HospitalHarrowUK
  5. 5.Centre for Medical Image ComputingUniversity College LondonLondonUK
  6. 6.Department of Computer ScienceUniversity College LondonLondonUK
  7. 7.Division of Medical and Biological InformaticsGerman Cancer Research CenterHeidelbergGermany

Personalised recommendations