An Application-Dependent Framework for the Recognition of High-Level Surgical Tasks in the OR

  • Florent Lalys
  • Laurent Riffaud
  • David Bouget
  • Pierre Jannin
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6891)


Surgical process analysis and modeling is a recent and important topic aiming at introducing a new generation of computer-assisted surgical systems. Among all of the techniques already in use for extracting data from the Operating Room, the use of image videos allows automating the surgeons’ assistance without altering the surgical routine. We proposed in this paper an application-dependent framework able to automatically extract the phases of the surgery only by using microscope videos as input data and that can be adaptable to different surgical specialties. First, four distinct types of classifiers based on image processing were implemented to extract visual cues from video frames. Each of these classifiers was related to one kind of visual cue: visual cues recognizable through color were detected with a color histogram approach, for shape-oriented visual cues we trained a Haar classifier, for texture-oriented visual cues we used a bag-of-word approach with SIFT descriptors, and for all other visual cues we used a classical image classification approach including a feature extraction, selection, and a supervised classification. The extraction of this semantic vector for each video frame then permitted to classify time series using either Hidden Markov Model or Dynamic Time Warping algorithms. The framework was validated on cataract surgeries, obtaining accuracies of 95%.


Surgical phase digital microscope cataract surgeries DTW 


  1. 1.
    Speidel, S., Sudra, G., Senemaud, J., Drentschew, M., Müller-stich, B., Gun, C., Dillmann, R.: Situation modeling and situation recognition for a context-aware augmented reality system. In: Progression in Biomed. Optics and Imaging, vol. 9(1), p. 35 (2008)Google Scholar
  2. 2.
    Lo, B., Darzi, A., Yang, G.: Episode Classification for the Analysis of Tissue/Instrument Interaction with Multiple Visual Cues. In: Ellis, R.E., Peters, T.M. (eds.) MICCAI 2003. LNCS, vol. 2878, pp. 230–237. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  3. 3.
    Klank, U., Padoy, N., Feussner, H., Navab, N.: Automatic feature generation in endoscopic images. Int. J. Comput. Assist. Radiol. Surg. 3(3-4), 331–339 (2008)CrossRefGoogle Scholar
  4. 4.
    Blum, T., Feußner, H., Navab, N.: Modeling and Segmentation of Surgical Workflow from Laparoscopic Video. In: Jiang, T., Navab, N., Pluim, J.P.W., Viergever, M.A., et al. (eds.) MICCAI 2010. LNCS, vol. 6363, pp. 400–407. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  5. 5.
    Bhatia, B., Oates, T., Xiao, Y., Hu, P.: Real-time identification of operating room state from video. In: AAAI, pp. 1761–1766 (2007)Google Scholar
  6. 6.
    Padoy, N., Blum, T., Feuner, H., Berger, M.O., Navab, N.: On-line recognition of surgical activity for monitoring in the operating room. In: Proc’s of the 20th Conference on Innovative Applications of Artificial Intelligence (2008)Google Scholar
  7. 7.
    Voros, S., Hager, G.: Towards “real-time” tool-tissue interaction detection in robotically assisted laparoscopy. Biomed. Robotics and Biomechatronics, 562–567 (2008)Google Scholar
  8. 8.
    Reiley, C., Hager, G.: Decomposition of robotic surgical tasks: an analysis of subtasks and their correlation to skill. In: M2CAI Workshop, MICCAI 2009 (2009)Google Scholar
  9. 9.
    Lalys, F., Riffaud, L., Morandi, X., Jannin, P.: Surgical phases detection from microscope videos by combining SVM and HMM. In: Menze, B., Langs, G., Tu, Z., Criminisi, A. (eds.) MICCAI 2010. LNCS, vol. 6533, pp. 54–62. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  10. 10.
    Viola, P. and Jones, M.: Rapid real-time face detection. IJCV, 137-154 (2004)Google Scholar
  11. 11.
    Lowe, D.G.: Object recognition from scale-invariant features. In: ICCV 1999, vol. 2, pp. 1150–1157 (1999)Google Scholar
  12. 12.
    Lalys, F., Riffaud, L., Morandi, X., Jannin, P.: Automatic phases recognition in pituitary surgeries by microscope images classification. In: Navab, N., Jannin, P. (eds.) IPCAI 2010. LNCS, vol. 6135, pp. 34–44. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  13. 13.
    Haralick, R., Shanmugam, K., Dinstein, I.: Textural features for image classification. IEEE Trans. on Systems, Man, and Cybernetics 3(6), 61–621 (1973)CrossRefGoogle Scholar
  14. 14.
    Hu, M.: Visual pattern recognition by moment invariants. Trans. Inf. Theory 8(2), 79–87 (1962)zbMATHGoogle Scholar
  15. 15.
    Ahmed, N., Natarajan, T., Rao, R.: Discrete Cosine Transform. IEEE Trans. Comp., 90–93 (1974)Google Scholar
  16. 16.
    Guyon, I., Weston, J., Barhill, S., Vapnik, V.: Gene selection for cancer classification using support vector machine. Machine Learning 46, 389–422 (2002)CrossRefzbMATHGoogle Scholar
  17. 17.
    Hamming, R.W.: Coding and Information Theory. Prentice-Hall Inc., Englewood Cliffs (1980)zbMATHGoogle Scholar
  18. 18.
    Sakoe, H., Chiba, S.: Dynamic programming algorithm optimization for spoken work recognition. In: IEEE Trans. Acoustic, Speech and Signal Proc., vol. 26(1), pp. 43–49 (1978)Google Scholar
  19. 19.
    Ahmadi, S.-A., Sielhorst, T., Stauder, R., Horn, M., Feussner, H., Navab, N.: Recovery of surgical workflow without explicit models. In: Larsen, R., Nielsen, M., Sporring, J. (eds.) MICCAI 2006. LNCS, vol. 4190, pp. 420–428. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  20. 20.
    Bouarfa, L., Jonker, P., Dankelman, J.: Discovery of high-level tasks in the operating room. J. Biomedical Informatics 44(3), 455–462 (2010)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Florent Lalys
    • 1
    • 2
    • 3
  • Laurent Riffaud
    • 1
    • 2
    • 3
    • 4
  • David Bouget
    • 1
    • 2
    • 3
  • Pierre Jannin
    • 1
    • 2
    • 3
  1. 1.Faculté de Médecine CS 34317INSERM, U746Rennes CedexFrance
  2. 2.INRIA, VisAGeS Unité/ProjetRennesFrance
  3. 3.University of Rennes I, CNRS, UMR 6074, IRISARennesFrance
  4. 4.Department of NeurosurgeryPontchaillou University HospitalRennesFrance

Personalised recommendations