Classification approach for automatic laparoscopic video database organization

  • Andru Putra TwinandaEmail author
  • Jacques Marescaux
  • Michel de Mathelin
  • Nicolas Padoy
Original Article



One of the advantages of minimally invasive surgery (MIS) is that the underlying digitization provides invaluable information regarding the execution of procedures in various patient-specific conditions. However, such information can only be obtained conveniently if the laparoscopic video database comes with semantic annotations, which are typically provided manually by experts. Considering the growing popularity of MIS, manual annotation becomes a laborious and costly task. In this paper, we tackle the problem of laparoscopic video classification, which consists of automatically identifying the type of abdominal surgery performed in a video. In addition to performing classifications on the full recordings of the procedures, we also carry out sub-video and video clip classifications. These classifications are carried out to investigate how many frames from a video are needed to get a good classification performance and which parts of the procedures contain more discriminative features.


Our classification pipeline is as follows. First, we reject the irrelevant frames from the videos using the color properties of the video frames. Second, we extract visual features from the relevant frames. Third, we quantize the features using several feature encoding methods, i.e., vector quantization, sparse coding (SC), and Fisher encoding. Fourth, we carry out the classification using support vector machines. While the sub-video classification is carried out by uniformly downsampling the video frames, the video clip classification is carried out by taking three parts of the videos (i.e., beginning, middle, and end) and running the classification pipeline separately for every video part. Ultimately, we build our final classification model by combining the features using a multiple kernel learning (MKL) approach.


To carry out the experiments, we use a dataset containing 208 videos of eight different surgeries performed by 10 different surgeons. The results show that SC with \(K\)-singular value decomposition (K-SVD) yields the best classification accuracy. The results also demonstrate that the classification accuracy only decreases by 3 % when solely 60 % of the video frames are utilized. Furthermore, it is also shown that the end part of the procedures is the most discriminative part of the surgery. Specifically, by using only the last 20 % of the video frames, a classification accuracy greater than 70 % can be achieved. Finally, the combination of all features yields the best performance of 90.38 % accuracy.


The SC with K-SVD provides the best representation of our videos, yielding the best accuracies for all features. In terms of information, the end part of the laparoscopic videos is the most discriminative compared to the other parts of the videos. In addition to their good performance individually, the features yield even better classification results when all of them are combined using the MKL approach.


Minimally invasive surgery Laparoscopic videos Classification Database organization Support vector machine Feature encoding Multiple kernel learning 



This work was supported by French state funds managed by the ANR within the Investissements d’Avenir program under references ANR-11-LABX-0004 (Labex CAMI), ANR-10-IDEX-0002-02 (IdEx Unistra), and ANR-10-IAHU-02 (IHU Strasbourg). The authors would like to thank the IRCAD audiovisual team for their help in generating the dataset.

Conflict of interest

Andru P. Twinanda, Jacques Marescaux, Michel de Mathelin, and Nicolas Padoy declare that they have no conflict of interest.

Supplementary material

Supplementary material 1 (mp4 58397 KB)


  1. 1.
    Aharon M, Elad M, Bruckstein A (2006) K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation. Signal Process IEEE Trans 54(11):4311–4322CrossRefGoogle Scholar
  2. 2.
    Akata Z, Perronnin F, Harchaoui Z, Schmid C (2014) Good practice in large-scale learning for image classification. IEEE Trans Pattern Anal Mach Intell 36(3):507–520CrossRefPubMedGoogle Scholar
  3. 3.
    Allan M, Thompson S, Clarkson MJ, Ourselin S, Hawkes D, Kelly J, Stoyanov D (2014) 2d-3d pose tracking of rigid instruments in minimally invasive surgery. In: IPCAI, Springer International Publishing, pp 1–10Google Scholar
  4. 4.
    Atasoy S, Mateus D, Meining A, Yang GZ, Navab N (2012) Endoscopic video manifolds for targeted optical biopsy. IEEE Trans Med Imaging 31(3):637–653CrossRefPubMedGoogle Scholar
  5. 5.
    Bay H, Tuytelaars T, Gool LV (2006) Surf: speeded up robust features. In: In ECCV, pp 404–417Google Scholar
  6. 6.
    Blum T, Feussner H, Navab N (2010) Modeling and segmentation of surgical workflow from laparoscopic video. In: MICCAI (3), pp 400–407Google Scholar
  7. 7.
    Cabras P, Goyard D, Nageotte F, Zanne P, Doignon C (2014) Comparison of methods for estimating the position of actuated instruments in flexible endoscopic surgery. In: IROS, pp 3522–3528Google Scholar
  8. 8.
    Chatfield K, Lempitsky V, Vedaldi A, Zisserman A (2011) The devil is in the details: an evaluation of recent feature encoding methods. In: BMVA, pp 76.1–76.12Google Scholar
  9. 9.
    Chu WS, Zhou F, De la Torre F (2012) Unsupervised temporal commonality discovery. In: ECCVGoogle Scholar
  10. 10.
    Coates A, Ng A (2011) The importance of encoding versus training with sparse coding and vector quantization. In: ICML, pp 921–928Google Scholar
  11. 11.
    Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: CVPR, pp 886–893Google Scholar
  12. 12.
    Dollár P, Rabaud V, Cottrell G, Belongie S (2005) Behavior recognition via sparse spatio-temporal features. In: VS-PETSGoogle Scholar
  13. 13.
    Harris C, Stephens M (1988) A combined corner and edge detector. In: Proceedings of AVC, pp 23.1–23.6Google Scholar
  14. 14.
    Lalys F, Riffaud L, Bouget D, Jannin P (2012) A framework for the recognition of high-level surgical tasks from video images for cataract surgeries. IEEE Trans Biomed Eng 59(4):966–976PubMedCentralCrossRefPubMedGoogle Scholar
  15. 15.
    Laptev I, Marszałek M, Schmid C, Rozenfeld B (2008) Learning realistic human actions from movies. In: CVPRGoogle Scholar
  16. 16.
    Lowe D (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60:91–110CrossRefGoogle Scholar
  17. 17.
    Muenzer B, Schoeffmann K, Boszormenyi L (2013) Relevance segmentation of laparoscopic videos. In: IEEE International Symposium on Multimedia, pp 84–91Google Scholar
  18. 18.
    Padoy N, Mateus D, Weinland D, Berger MO, Navab N (2009) Workflow monitoring based on 3D motion features. In: Workshop on video-oriented object and event classification in conjunction with ICCV 2009, pp 585–592Google Scholar
  19. 19.
    Perronnin F, Sánchez J, Mensink T (2010) Improving the fisher kernel for large-scale image classification. In: ECCV, pp 143–156Google Scholar
  20. 20.
    Reiter A, Allen PK, Zhao T (2012) Feature classification for tracking articulated surgical tools. In: MICCAI, vol 7511, pp 592–600Google Scholar
  21. 21.
    Twinanda AP, Marescaux J, Mathelin MD, Padoy N (2014a) Towards better laparoscopic video database organization by automatic surgery classification. In: IPCAI, pp 186–194Google Scholar
  22. 22.
    Twinanda AP, Mathelin MD, Padoy N (2014b) Fisher kernel based task boundary retrieval in laparoscopic database with single video query. In: MICCAI, pp 409–416Google Scholar
  23. 23.
    Varma M, Babu RB (2009) More generality in efficient multiple kernel learning. In: ICML, ACM, pp 1065–1072Google Scholar
  24. 24.
    Vedaldi A, Fulkerson B (2010) Vlfeat: an open and portable library of computer vision algorithms. In: ICM, ACM, pp 1469–1472Google Scholar
  25. 25.
    Xia L, Aggarwal J (2013) Spatio-temporal depth cuboid similarity feature for activity recognition using depth camera. In: CVPRGoogle Scholar
  26. 26.
    Zappella L, Bejar B, Hager G, Vidal R (2013) Surgical gesture classification from video and kinematic data. Med Image Anal 17(7):732–745CrossRefPubMedGoogle Scholar

Copyright information

© CARS 2015

Authors and Affiliations

  • Andru Putra Twinanda
    • 1
    Email author
  • Jacques Marescaux
    • 2
  • Michel de Mathelin
    • 1
  • Nicolas Padoy
    • 1
  1. 1.ICube LaboratoryUniversity of Strasbourg, CNRS, IHUStrasbourgFrance
  2. 2.IRCADUniversity Hospital of StrasbourgStrasbourgFrance

Personalised recommendations