A novel image-based retrieval system for characterization of maxillofacial lesions in cone beam CT images

  • Fatemeh AbdolaliEmail author
  • Reza Aghaeizadeh Zoroofi
  • Yoshito Otake
  • Yoshinobu Sato
Original Article



The objective of medical content-based image retrieval (CBIR) is to assist clinicians in decision making by retrieving the most similar cases to a given query image from a large database. Herein, a new method for content-based image retrieval of cone beam CT (CBCT) scans is presented.


The introduced framework consists of two main phases: training database construction and querying. The goal of the training phase is database construction, which consists of three main steps. First, automatic segmentation of lesions using 3D symmetry analysis is performed. Embedding the prior shape knowledge of the 3D symmetry characteristics of the healthy human head structure increases the accuracy of automatic segmentation. Then, spatial pyramid matching is used for feature extraction, and the relative importance of each feature is learned using classifiers.


The method was applied to a dataset of 1145 volumetric CBCT images with four classes of maxillofacial lesions. A symmetry-based analysis model for automatic lesion segmentation was evaluated using similarity measures. Mean Dice coefficients of 0.89, 0.85, 0.92, and 0.87 were achieved for maxillary sinus perforation, radiolucent lesion, unerupted tooth, and root fracture classes, respectively. Moreover, the execution time of automatic segmentation was reduced to 3 min per case. The performance of the proposed search engine was evaluated using mean average precision and normalized discounted cumulative gain. A mean average retrieval accuracy and normalized discounted cumulative gain of 0.90 and 0.92, respectively, were achieved.


Quantitative results show that the proposed approach is more effective than previous methods in the literature, and it can facilitate the introduction of CBIR in clinical CBCT applications.


Cone beam CT Content-based medical image retrieval Maxillofacial lesions Spatial pyramid matching Symmetry analysis 



This work is partly supported by MEXT Grant-in-Aid for Scientific Research No. 26108004. The authors would like to extend thanks to the clinical staffs of Taleghani Educational Hospital, Imam Hossein Educational Hospital, Guilan University of Medical Sciences, and Farzaneh Momeni Dental Imaging Center for clinical assistance and reviewing the cases.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

Human and animal rights

For this type of study, formal consent is not required because this study is a retrospective study.

Informed consent

Written informed consent was not required for this study because this study is a retrospective study.


  1. 1.
    Müller H, Michoux N, Bandon D, Geissbuhler A (2004) A review of content-based image retrieval systems in medical applications—clinical benefits and future directions. Int J Med Inf 73:1–23CrossRefGoogle Scholar
  2. 2.
    Tyagi V (2017) Content-based image retrieval techniques: a review. In: Content-based image retrieval. Springer, pp 29–48Google Scholar
  3. 3.
    Datta R, Joshi D, Li J, Wang JZ (2008) Image retrieval: ideas, influences, and trends of the new age. ACM Comput Surv Csur 40:5Google Scholar
  4. 4.
    Müller H, de Herrera AGS, Kalpathy-Cramer J, Demner-Fushman D, Antani SK, Eggel I (2012) Overview of the ImageCLEF 2012 medical image retrieval and classification tasks. In: CLEF online work. Noteslabsworkshop, pp 1–16Google Scholar
  5. 5.
    Müller H, Rosset A, Garcia A, Vallee JP, Geissbuhler A (2005) Benefits of content-based visual data access in radiology. Radiographics 25(3):849–858CrossRefPubMedGoogle Scholar
  6. 6.
    Korn P, Sidiropoulos N, Faloutsos C, Siegel E, Protopapas Z (1998) Fast and effective retrieval of medical tumor shapes. IEEE Trans Knowl Data Eng 10:889–904CrossRefGoogle Scholar
  7. 7.
    Long LR, Antani S, Lee D-J, Krainak DM, Thoma GR (2003) Biomedical information from a national collection of spine x-rays: film to content-based retrieval. In: Medical imaging 2003 PACS integrated medical information systems design evaluation. International Society for Optics and Photonics, pp 70–85Google Scholar
  8. 8.
    Gundreddy RR, Tan M, Qiu Y, Cheng S, Liu H, Zheng B (2015) Assessment of performance and reproducibility of applying a content-based image retrieval scheme for classification of breast lesions. Med Phys 42:4241–4249CrossRefPubMedPubMedCentralGoogle Scholar
  9. 9.
    Ramos J, Kockelkorn TT, Ramos I, Ramos R, Grutters J, Viergever MA, van Ginneken B, Campilho A (2016) Content-based image retrieval by metric learning from radiology reports: application to interstitial lung diseases. IEEE J Biomed Health Inform 20:281–292CrossRefPubMedGoogle Scholar
  10. 10.
    Kurtz C, Depeursinge A, Napel S, Beaulieu CF, Rubin DL (2014) On combining image-based and ontological semantic dissimilarities for medical image retrieval applications. Med Image Anal 18:1082–1100CrossRefPubMedPubMedCentralGoogle Scholar
  11. 11.
    Tzelepi M, Tefas A (2018) Deep convolutional learning for content based image retrieval. Neurocomputing 275:2467–2478CrossRefGoogle Scholar
  12. 12.
    Langs G, Hanbury A, Menze B, Müller H (2012) VISCERAL: towards large data in medical imaging-challenges and directions. In: MICCAI international workshop on medical content-based retrieval for clinical decision support, pp 92–98Google Scholar
  13. 13.
    Carson C, Belongie S, Greenspan H, Malik J (2002) Blobworld: image segmentation using expectation-maximization and its application to image querying. IEEE Trans Pattern Anal Mach Intell 24(8):1026–1038CrossRefGoogle Scholar
  14. 14.
    Bordes F, Berthier T, Di Jorio L, Vincent P, Bengio Y (2018) Iteratively unveiling new regions of interest in Deep Learning models. In: International conference on medical imaging with deep learning. Retrieved from Accessed 13 Mar 2019
  15. 15.
    Penatti OA, Valle E, da Torres RS (2012) Comparative study of global color and texture descriptors for web image retrieval. J Vis Commun Image Represent 23:359–380CrossRefGoogle Scholar
  16. 16.
    Abdolali F, Zoroofi RA, Otake Y, Sato Y (2016) Automatic segmentation of maxillofacial cysts in cone beam CT images. Comput Biol Med 72:108–119CrossRefPubMedGoogle Scholar
  17. 17.
    Loy G, Eklundh J-O (2006) Detecting symmetry and symmetric constellations of features. In: European conference on computer vision. Springer, pp 508–521Google Scholar
  18. 18.
    Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60:91–110CrossRefGoogle Scholar
  19. 19.
    Cheung W, Hamarneh G (2009) n-SIFT: n-dimensional scale invariant feature transform. IEEE Trans Image Process 18:2012–2021CrossRefPubMedGoogle Scholar
  20. 20.
    Ni D, Chui YP, Qu Y, Yang X, Qin J, Wong T-T, Ho SS, Heng PA (2009) Reconstruction of volumetric ultrasound panorama based on improved 3D SIFT. Comput Med Imaging Graph 33:559–566CrossRefPubMedGoogle Scholar
  21. 21.
    Comaniciu D, Meer P (2002) Mean shift: a robust approach toward feature space analysis. IEEE Trans Pattern Anal Mach Intell 24:603–619CrossRefGoogle Scholar
  22. 22.
    Cui Y, Feng J (2013) Real-time B-spline free-form deformation via GPU acceleration. Comput Gr 37:1–11CrossRefGoogle Scholar
  23. 23.
    Yang J, Yu K, Gong Y, Huang T (2009) Linear spatial pyramid matching using sparse coding for image classification. In: IEEE conference on computer vision pattern recognition, CVPR 2009. IEEE, pp 1794–1801Google Scholar
  24. 24.
    Csurka G, Dance C, Fan L, Willamowski J, Bray C (2004) Visual categorization with bags of keypoints. In: Workshop on statistical learning in computer vision, ECCV, Prague. pp 1–2Google Scholar
  25. 25.
    Griffin G, Holub A, Perona P (2007) Caltech-256 object category dataset. Technical report 7694, California Institute of Technology.
  26. 26.
    Fei-Fei L, Fergus R, Perona P (2007) Learning generative visual models from few training examples: an incremental Bayesian approach tested on 101 object categories. Comput Vis Image Underst 106:59–70CrossRefGoogle Scholar
  27. 27.
    Atkinson KE (2008) An introduction to numerical analysis. Wiley, HobokenGoogle Scholar
  28. 28.
    Dice LR (1945) Measures of the amount of ecologic association between species. Ecology 26:297–302CrossRefGoogle Scholar
  29. 29.
    Cignoni P, Rocchini C, Scopigno R (1998) Metro: measuring error on simplified surfaces. In: Computer graphics forum. Wiley, pp 167–174Google Scholar
  30. 30.
    Clemmensen L, Hastie T, Witten D, Ersbøll B (2011) Sparse discriminant analysis. Technometrics 53:406–413CrossRefGoogle Scholar
  31. 31.
    Schütze H, Manning CD, Raghavan P (2008) Introduction to information retrieval. Cambridge University Press, CambridgeGoogle Scholar
  32. 32.
    Järvelin K, Kekäläinen J (2002) Cumulated gain-based evaluation of IR techniques. ACM Trans Inf Syst TOIS 20:422–446CrossRefGoogle Scholar
  33. 33.
    Stoetzer M, Nickel F, Rana M, Lemound J, Wenzel D, von See C, Gellrich N-C (2013) Advances in assessing the volume of odontogenic cysts and tumors in the mandible: a retrospective clinical trial. Head Face Med 9:14CrossRefPubMedPubMedCentralGoogle Scholar
  34. 34.
    Abdolali F, Zoroofi RA, Abdolali M (2015) Content based image retrieval for maxillofacial lesions. In: 9th Iranian conference on machine vision and image processing (MVIP). IEEE, pp 5–8Google Scholar
  35. 35.
    Deserno TM, Molander B, Guld MO, Thies C, Grondahl HG (2007) Content-based access to oral and maxillofacial radiographs. Dentomaxillofacial Radiol 36:328–335CrossRefGoogle Scholar

Copyright information

© CARS 2019

Authors and Affiliations

  1. 1.Control and Intelligent Processing Center of Excellence, School of Electrical and Computer Engineering, University College of EngineeringUniversity of TehranTehranIran
  2. 2.Graduate School of Information ScienceNara Institute of Science and Technology (NAIST)IkomaJapan

Personalised recommendations