Hard Frame Detection and Online Mapping for Surgical Phase Recognition

  • Fangqiu Yi
  • Tingting JiangEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11768)


Surgical phase recognition is an important topic of Computer Assisted Surgery (CAS) systems. In the complicated surgical procedures, there are lots of hard frames that have indistinguishable visual features but are assigned with different labels. Prior works try to classify hard frames along with other simple frames indiscriminately, which causes various problems. Different from previous approaches, we take hard frames as mislabeled samples and find them in the training set via data cleansing strategy. Then, we propose an Online Hard Frame Mapper (OHFM) to handle the detected hard frames separately. We evaluate our solution on the M2CAI16 Workflow Challenge dataset and the Cholec80 dataset and achieve superior results. (The code is available at


Surgical phase recognition Data cleansing Deep learning 



This work was partially supported by the National Basic Research Program of China (973 Program) under contract 2015CB351803, the Natural Science Foundation of China under contracts 61572042 and 61527804. We also acknowledge the Clinical Medicine Plus X-Young Scholars Project, and High-Performance Computing Platform of Peking University for providing computational resources.

Supplementary material (11.2 mb)
Supplementary material 1 (zip 11481 KB)


  1. 1.
    Beenish, B., Tim, O., Yan, X., Peter, H.: Real-time identification of operating room state from video. In: Proceedings of the 19th Conference on Innovative Applications of Artificial Intelligence, vol. 2, pp. 1761–1766 (2007)Google Scholar
  2. 2.
    Blum, T., Feußner, H., Navab, N.: Modeling and segmentation of surgical workflow from laparoscopic video. In: Jiang, T., Navab, N., Pluim, J.P.W., Viergever, M.A. (eds.) MICCAI 2010. LNCS, vol. 6363, pp. 400–407. Springer, Heidelberg (2010). Scholar
  3. 3.
    Bricon-Souf, N., Newman, C.R.: Context awareness in health care: a review. Int. J. Med. Inform. 76(1), 2–12 (2007)CrossRefGoogle Scholar
  4. 4.
    Cadène, R., Robert, T., Thome, N., Cord, M.: MICCAI workflow challenge: convolutional neural networks with time smoothing and Hidden Markov Model for video frames classification. arxiv abs/1610.05541 (2016)Google Scholar
  5. 5.
    Frenay, B., Verleysen, M.: Classification in the presence of label noise: a survey. IEEE Trans. Neural Netw. Learn. Syst. 25(5), 845–869 (2014)CrossRefGoogle Scholar
  6. 6.
    Gamberger, D., Lavrac, N., Dzeroski, S.: Noise detection and elimination in data preprocessing: experiments in medical domains. Appl. Artif. Intell. 14(2), 205–223 (2000)CrossRefGoogle Scholar
  7. 7.
    Jin, Y., et al.: SV-RCNet: workflow recognition from surgical videos using recurrent convolutional network. IEEE Trans. Med. Imaging 37(5), 1114–1126 (2018)CrossRefGoogle Scholar
  8. 8.
    Lalys, F., Riffaud, L., Bouget, D., Jannin, P.: A framework for the recognition of high-level surgical tasks from video images for cataract surgeries. IEEE Trans. Biomed. Eng. 59(4), 966–976 (2012)CrossRefGoogle Scholar
  9. 9.
    Lalys, F., Riffaud, L., Morandi, X., Jannin, P.: Surgical phases detection from microscope videos by combining SVM and HMM. In: Menze, B., Langs, G., Tu, Z., Criminisi, A. (eds.) MCV 2010. LNCS, vol. 6533, pp. 54–62. Springer, Heidelberg (2011). Scholar
  10. 10.
    Lin, H.C., Shafran, I., Murphy, T.E., Okamura, A.M., Yuh, D.D., Hager, G.D.: Automatic detection and segmentation of robot-assisted surgical motions. In: Duncan, J.S., Gerig, G. (eds.) MICCAI 2005. LNCS, vol. 3749, pp. 802–810. Springer, Heidelberg (2005). Scholar
  11. 11.
    Miranda, A.L.B., Garcia, L.P.F., Carvalho, A.C.P.L.F., Lorena, A.C.: Use of classification algorithms in noise detection and elimination. In: Corchado, E., Wu, X., Oja, E., Herrero, Á., Baruque, B. (eds.) HAIS 2009. LNCS (LNAI), vol. 5572, pp. 417–424. Springer, Heidelberg (2009). Scholar
  12. 12.
    Padoy, N., Blum, T., Feussner, H., Berger, M.O., Navab, N.: On-line recognition of surgical activity for monitoring in the operating room. In: Proceedings of the 20th Conference on Innovative Applications of Artificial Intelligence, vol. 3, pp. 1718–1724 (2008)Google Scholar
  13. 13.
    Padoy, N., Blum, T., Ahmadi, S.A., Feussner, H., Berger, M.O., Navab, N.: Statistical modeling and recognition of surgical workflow. Med. Image Anal. 16(3), 632–641 (2012)CrossRefGoogle Scholar
  14. 14.
    Sun, J., Zhao, F., Wang, C., Chen, S.: Identifying and correcting mislabeled training instances. In: Future Generation Communication and Networking (FGCN 2007), vol. 1, pp. 244–250 (2007)Google Scholar
  15. 15.
    Tao, L., Zappella, L., Hager, G.D., Vidal, R.: Surgical gesture segmentation and recognition. In: Mori, K., Sakuma, I., Sato, Y., Barillot, C., Navab, N. (eds.) MICCAI 2013. LNCS, vol. 8151, pp. 339–346. Springer, Heidelberg (2013). Scholar
  16. 16.
    Twinanda, A.P., Shehata, S., et al.: EndoNet: a deep architecture for recognition tasks on laparoscopic videos. IEEE Trans. Med. Imaging 36(1), 86–97 (2017)CrossRefGoogle Scholar
  17. 17.
    Twinanda, A.P., Mutter, D., et al.: Single- and multi-task architectures for surgical workflow challenge at M2CAI 2016. arxiv abs/1610.08844 (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.NELVT, Department of Computer SciencePeking UniversityBeijingChina

Personalised recommendations