Generate Structured Radiology Report from CT Images Using Image Annotation Techniques: Preliminary Results with Liver CT

  • Samira Loveymi
  • Mir Hossein Dezfoulian
  • Muharram MansoorizadehEmail author
Original Paper


A medical annotation system for radiology images extracts clinically useful information from the images, allowing the machines to infer useful abstract semantics and become capable of automatic reasoning and making diagnostic decision. It also supplies human-interpretable explanation for the images. We have implemented a computerized framework that, given a liver CT image, predicts radiological annotations with high accuracy, in order to generate a structured report, which includes predicting very specific high-level semantic content. Each report of a liver CT image is related to different inhomogeneous parts like the liver, lesion, and vessel. We put forward a claim that gathering all kinds of features is not suitable for filling all parts of the report. As a matter of fact, for each group of annotations, one should find and extract the best feature that results in the best answers for that specific annotation. To this end, the main challenge is discovering the relationships between these specific semantic concepts and their association with the low-level image features. Our framework was implemented by combining a set of the state-of-the-art low-level imaging features. In addition, we propose a novel feature (DLBP (deep local binary pattern)) based on LBP that incorporates multi-slice analysis in CT images and further improves the performance. In order to model our annotation system, two methods were used, namely multi-class support vector machine (SVM) and random subspace (RS) which is an ensemble learning method. Applying this representation leads to a high prediction accuracy of 93.1% despite its relatively low dimension in comparison with the existing works.


Deep local binary pattern Medical image annotation Computed tomography Liver CT images 



  1. 1.
    Atam P Dhawan. Medical image analysis, volume 31. John Wiley & Sons, 2011.Google Scholar
  2. 2.
    Loveymi S, Shadgar B, Osareh A: An efficient approach to automated medical image annotation. International Review on Computers and Software 6(5):749–759, 2011Google Scholar
  3. 3.
    Tommasi T, Orabona F, Caputo B: Discriminative cue integration for medical image annotation. Pattern Recognition Letters 29(15):1996–2002, 2008CrossRefGoogle Scholar
  4. 4.
    Tatiana Tommasi, Francesco Orabona, and Barbara Caputo. An svm confidence-based approach to medical image annotation. In Workshop of the Cross-Language Evaluation Forum for European Languages, pages 696–703. Springer, 2008.Google Scholar
  5. 5.
    Tatiana Tommasi, Barbara Caputo, Petra Welter, Mark Oliver Guldld, and Thomas M Deserno. Overview of the clef 2009 medical image annotation track. In Workshop of the Cross-Language Evaluation Forum for European Languages, pages 85–93. Springer, 2009.Google Scholar
  6. 6.
    Dimitrovski I, Kocev D, Loskovska S, Dzeroski S: Hierarchical annotation of medical images. Pattern Recognition 44(10-11):2436–2449, 2011CrossRefGoogle Scholar
  7. 7.
    Demner-Fushman D, Antani S, Simpson M, Thoma GR: Annotation and retrieval of clinically relevant images. International Journal of Medical Informatics 78(12):e59–e67, 2009CrossRefGoogle Scholar
  8. 8.
    Zehra Camlica, Hamid R Tizhoosh, and Farzad Khalvati. Medical image classification via SVM using LBP features from saliency-based folded data. In 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA), pages 128–132. IEEE, 2015.Google Scholar
  9. 9.
    Riadh Bouslimi, Abir Messaoudi, and Jalel Akaichi. Using a bag of words for automatic medical image annotation with a latent semantic. arXiv preprint arXiv:1306.0178, 2013.Google Scholar
  10. 10.
    Alaidine Ben Ayed, Mustapha Kardouchi, and Sid-Ahmed Selouani. Rotation invariant fuzzy shape contexts based on eigenshapes and Fourier transforms for efficient radiological image retrieval. In 2012 International Conference on Multimedia Computing and Systems, pages 266–271. IEEE, 2012.Google Scholar
  11. 11.
    Zare MR, Mueen A, Seng WC: Automatic medical x-ray image classification using annotation. Journal of Digital Imaging 27(1):77–89, 2014CrossRefGoogle Scholar
  12. 12.
    Mueen A, Zainuddin R, Baba MS: Automatic multilevel medical image annotation and retrieval. Journal of digital imaging 21(3):290–295, 2008CrossRefGoogle Scholar
  13. 13.
    Wang Y, Wang L, Rastegar-Mojarad M, Moon S, Shen F, NaveedAfzal SL, Zeng Y, Mehrabi S, Sohn S et al.: Clinical information extraction applications: a literature review. Journal of biomedical informatics 77:34–49, 2018CrossRefGoogle Scholar
  14. 14.
    Kokciyan N, Turkay R, Uskudarli S, Yolum P, Bakir B, Acar B: Semantic description of liver ct images: an ontological approach. IEEE journal of biomedical and health informatics 18(4):1363–1369, 2014CrossRefGoogle Scholar
  15. 15.
    Gao H, Aiello Bowles EJ, Carrell D, Buist DSM: Using natural language processing to extract mammographic findings. Journal of biomedical informatics 54:77–84, 2015CrossRefGoogle Scholar
  16. 16.
    Castro SM, Tseytlin E, Medvedeva O, Mitchell K, Visweswaran S, Bekhuis T, Jacobson RS: Automated annotation and classification of bi-rads assessment from radiology reports. Journal of biomedical informatics 69:177–187, 2017CrossRefGoogle Scholar
  17. 17.
    Imane Nedjar, Said Mahmoudi, Mohammed Amine Chikh, Khadidja Abi-Yad, and ZouheyrBouafia. Automatic annotation of liver ct image: ImageCLEFmed 2015. In CLEF (Working Notes), 2015.Google Scholar
  18. 18.
    Neda Barzegar Marvasti, Maria del Mar Roldan Garcia, Suzan Uskudarli, Jose Francisco Aldana Montes, and Burak Acar. Overview of the ImageCLEF 2015 liver ct annotation task. In CLEF (Working Notes), 2015.Google Scholar
  19. 19.
    Kumar A, Dyer S, Kim J, Li C, Leong PHW, Fulham M, Feng D: Adapting content-based image retrieval techniques for the semantic annotation of medical images. Computerized Medical Imaging and Graphics 49:37–45, 2016CrossRefGoogle Scholar
  20. 20.
    Zhiyun Xue, Sameer Antani, L Rodney Long, and George R Thoma. Automatic multi-label annotation of abdominal ct images using CBIR. In Medical Imaging 2017: Imaging Informatics for Healthcare, Research, and Applications, volume 10138, page 1013807. International Society for Optics and PhotonicsGoogle Scholar
  21. 21.
    Depeursinge A, Kurtz C, Beaulieu C, Napel S, Rubin D: Predicting visual semantic descriptive terms from radiological image data: preliminary results with liver lesions in ct. IEEE transactions on medical imaging 33(8):1669–1676, 2014CrossRefGoogle Scholar
  22. 22.
    Xu Y, Lin L, Hu H, Wang D, Zhu W, Wang J, Han X-H, Chen Y-W: Texture-specific bag of visual words model and spatial cone matching based method for the retrieval of focal liver lesions using multiphase contrast-enhanced ct images. International journal of computer-assisted radiology and surgery 13(1):151–164, 2018CrossRefGoogle Scholar
  23. 23.
    Pol Cirujeda, Henning Muller, Daniel Rubin, Todd A Aguilera, Billy W Loo, MaximilianDiehn, Xavier Binefa, and Adrien Depeursinge. 3d Riesz-wavelet-based covariance descriptors for texture classification of lung nodule tissue in ct. In 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pages 7909– 7912. IEEE, 2015.Google Scholar
  24. 24.
    Arai K, Herdiyeni Y, Okumura H: Comparison of 2d and 3d local binary pattern in lung cancer diagnosis. Int J Adv Comput Sci Appl 3(4):89–95, 2012Google Scholar
  25. 25.
    Zhang J, Xia Y, Xie Y, Fulham M, Feng DD: Classification of medical images in the biomedical literature by jointly using deep and handcrafted visual features. IEEE Journal of biomedical and health informatics 22(5):1521–1530, 2018CrossRefGoogle Scholar
  26. 26.
    Nanni L, Paci M, Brahnam S, Ghidoni S: An ensemble of visual features for Gaussians of local descriptors and non-binary coding for texture descriptors. Expert Systems with Applications 82:27–39, 2017CrossRefGoogle Scholar
  27. 27.
    Gang Wang, David Forsyth, and Derek Hoiem. Comparative object similarity for improved recognition with few or no examples. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 3525–3532. IEEE, 2010.Google Scholar
  28. 28.
    Carlotta Domeniconi and Bojun Yan. Nearest neighbor ensemble. In Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004., volume 1, pages 228–231. IEEE, 2004.Google Scholar
  29. 29.
    Castellano G, Fanelli AM, Sforza G, Torsello MA: Shape annotation for intelligent image retrieval. Applied Intelligence 44(1):179–195, 2016CrossRefGoogle Scholar
  30. 30.
    Varish N, Pal AK: A novel image retrieval scheme using gray level co-occurrence matrix descriptors of discrete cosine transform based residual image. Applied Intelligence 48(9):2930–2953, 2018CrossRefGoogle Scholar
  31. 31.
    Soh L-K, Tsatsoulis C: Texture analysis of SAR sea ice imagery using gray level co-occurrence matrices. IEEE Transactions on geoscience and remote sensing 37(2):780–795, 1999CrossRefGoogle Scholar
  32. 32.
    Janis Fehr and Hans Burkhardt. 3d rotation invariant local binary patterns. In 2008 19th International Conference on Pattern Recognition, pages 1–4. IEEE, 2008.Google Scholar
  33. 33.
    Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 2013.Google Scholar
  34. 34.
    Wu T-F, Lin C-J, Weng RC: Probability estimates for multi-class classification by pairwise coupling. Journal of Machine Learning Research 5(Aug):975–1005, 2004Google Scholar
  35. 35.
    Ludmila I Kuncheva. Combining pattern classifiers: methods and algorithms, 2nd Ed. John Wiley & Sons, 2014.Google Scholar
  36. 36.
    Image Nedjar, Said Mahmoudi, and Mohammed Amine Chikh. Content-based medical image retrieval for liver ct annotation. Transactions on Machine Learning and Artificial Intelligence, 5(4), 2017.Google Scholar
  37. 37.
    Assaf B, Spanier NC, Sosna J, Acar B, Joskowicz L: A fully automatic end-to-end method for content-based image retrieval of ct scans with similar liver lesion annotations. International journal of computer-assisted radiology and surgery 13(1):165–174, 2018CrossRefGoogle Scholar
  38. 38.
    Kurtz C, Depeursinge A, Napel S, Beaulieu CF, Rubin DL: On combining image-based and ontological semantic dissimilarities for medical image retrieval applications. Medical image analysis 18(7):1082–1100, 2014CrossRefGoogle Scholar
  39. 39.
    Spanier AB, Cohen D, Joskowicz L: A new method for the automatic retrieval of medical cases based on the RadLex ontology. International journal of computer assisted radiology and surgery 12(3):471–484, 2017CrossRefGoogle Scholar

Copyright information

© Society for Imaging Informatics in Medicine 2019

Authors and Affiliations

  1. 1.Computer Engineering DepartmentBu-Ali Sina UniversityHamedanIran

Personalised recommendations