Classification approach for automatic laparoscopic video database organization
- 300 Downloads
One of the advantages of minimally invasive surgery (MIS) is that the underlying digitization provides invaluable information regarding the execution of procedures in various patient-specific conditions. However, such information can only be obtained conveniently if the laparoscopic video database comes with semantic annotations, which are typically provided manually by experts. Considering the growing popularity of MIS, manual annotation becomes a laborious and costly task. In this paper, we tackle the problem of laparoscopic video classification, which consists of automatically identifying the type of abdominal surgery performed in a video. In addition to performing classifications on the full recordings of the procedures, we also carry out sub-video and video clip classifications. These classifications are carried out to investigate how many frames from a video are needed to get a good classification performance and which parts of the procedures contain more discriminative features.
Our classification pipeline is as follows. First, we reject the irrelevant frames from the videos using the color properties of the video frames. Second, we extract visual features from the relevant frames. Third, we quantize the features using several feature encoding methods, i.e., vector quantization, sparse coding (SC), and Fisher encoding. Fourth, we carry out the classification using support vector machines. While the sub-video classification is carried out by uniformly downsampling the video frames, the video clip classification is carried out by taking three parts of the videos (i.e., beginning, middle, and end) and running the classification pipeline separately for every video part. Ultimately, we build our final classification model by combining the features using a multiple kernel learning (MKL) approach.
To carry out the experiments, we use a dataset containing 208 videos of eight different surgeries performed by 10 different surgeons. The results show that SC with \(K\)-singular value decomposition (K-SVD) yields the best classification accuracy. The results also demonstrate that the classification accuracy only decreases by 3 % when solely 60 % of the video frames are utilized. Furthermore, it is also shown that the end part of the procedures is the most discriminative part of the surgery. Specifically, by using only the last 20 % of the video frames, a classification accuracy greater than 70 % can be achieved. Finally, the combination of all features yields the best performance of 90.38 % accuracy.
The SC with K-SVD provides the best representation of our videos, yielding the best accuracies for all features. In terms of information, the end part of the laparoscopic videos is the most discriminative compared to the other parts of the videos. In addition to their good performance individually, the features yield even better classification results when all of them are combined using the MKL approach.
KeywordsMinimally invasive surgery Laparoscopic videos Classification Database organization Support vector machine Feature encoding Multiple kernel learning
This work was supported by French state funds managed by the ANR within the Investissements d’Avenir program under references ANR-11-LABX-0004 (Labex CAMI), ANR-10-IDEX-0002-02 (IdEx Unistra), and ANR-10-IAHU-02 (IHU Strasbourg). The authors would like to thank the IRCAD audiovisual team for their help in generating the dataset.
Conflict of interest
Andru P. Twinanda, Jacques Marescaux, Michel de Mathelin, and Nicolas Padoy declare that they have no conflict of interest.
Supplementary material 1 (mp4 58397 KB)
- 3.Allan M, Thompson S, Clarkson MJ, Ourselin S, Hawkes D, Kelly J, Stoyanov D (2014) 2d-3d pose tracking of rigid instruments in minimally invasive surgery. In: IPCAI, Springer International Publishing, pp 1–10Google Scholar
- 5.Bay H, Tuytelaars T, Gool LV (2006) Surf: speeded up robust features. In: In ECCV, pp 404–417Google Scholar
- 6.Blum T, Feussner H, Navab N (2010) Modeling and segmentation of surgical workflow from laparoscopic video. In: MICCAI (3), pp 400–407Google Scholar
- 7.Cabras P, Goyard D, Nageotte F, Zanne P, Doignon C (2014) Comparison of methods for estimating the position of actuated instruments in flexible endoscopic surgery. In: IROS, pp 3522–3528Google Scholar
- 8.Chatfield K, Lempitsky V, Vedaldi A, Zisserman A (2011) The devil is in the details: an evaluation of recent feature encoding methods. In: BMVA, pp 76.1–76.12Google Scholar
- 9.Chu WS, Zhou F, De la Torre F (2012) Unsupervised temporal commonality discovery. In: ECCVGoogle Scholar
- 10.Coates A, Ng A (2011) The importance of encoding versus training with sparse coding and vector quantization. In: ICML, pp 921–928Google Scholar
- 11.Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: CVPR, pp 886–893Google Scholar
- 12.Dollár P, Rabaud V, Cottrell G, Belongie S (2005) Behavior recognition via sparse spatio-temporal features. In: VS-PETSGoogle Scholar
- 13.Harris C, Stephens M (1988) A combined corner and edge detector. In: Proceedings of AVC, pp 23.1–23.6Google Scholar
- 15.Laptev I, Marszałek M, Schmid C, Rozenfeld B (2008) Learning realistic human actions from movies. In: CVPRGoogle Scholar
- 17.Muenzer B, Schoeffmann K, Boszormenyi L (2013) Relevance segmentation of laparoscopic videos. In: IEEE International Symposium on Multimedia, pp 84–91Google Scholar
- 18.Padoy N, Mateus D, Weinland D, Berger MO, Navab N (2009) Workflow monitoring based on 3D motion features. In: Workshop on video-oriented object and event classification in conjunction with ICCV 2009, pp 585–592Google Scholar
- 19.Perronnin F, Sánchez J, Mensink T (2010) Improving the fisher kernel for large-scale image classification. In: ECCV, pp 143–156Google Scholar
- 20.Reiter A, Allen PK, Zhao T (2012) Feature classification for tracking articulated surgical tools. In: MICCAI, vol 7511, pp 592–600Google Scholar
- 21.Twinanda AP, Marescaux J, Mathelin MD, Padoy N (2014a) Towards better laparoscopic video database organization by automatic surgery classification. In: IPCAI, pp 186–194Google Scholar
- 22.Twinanda AP, Mathelin MD, Padoy N (2014b) Fisher kernel based task boundary retrieval in laparoscopic database with single video query. In: MICCAI, pp 409–416Google Scholar
- 23.Varma M, Babu RB (2009) More generality in efficient multiple kernel learning. In: ICML, ACM, pp 1065–1072Google Scholar
- 24.Vedaldi A, Fulkerson B (2010) Vlfeat: an open and portable library of computer vision algorithms. In: ICM, ACM, pp 1469–1472Google Scholar
- 25.Xia L, Aggarwal J (2013) Spatio-temporal depth cuboid similarity feature for activity recognition using depth camera. In: CVPRGoogle Scholar