Performance evaluation of breast lesion detection systems with expert delineations: a comparative investigation on mammographic images

  • Bikesh K. SinghEmail author
  • Pankaj Jain
  • Sumit K. Banchhor
  • Kesari Verma


Performance of computerized diagnostic systems yearning to be approved by medical regulatory bodies must meet the expectations of human experts. Highly accurate lesion segmentation techniques have thus turned out to be an essential part for clinical acceptability of mammography based computer-aided diagnosis systems. The objective of this study is to evaluate the performance of six popular breast tumor detection techniques with manual delineations provided by two experienced radiologists on the mammographic images. In our study, 20 mammographic images from the mini-MIAS database are utilized. For the analysis, input mammographic images are first manually cropped to generate the region of interest (ROI). The ROI images are then pre-processed and segmentation is performed using different techniques, namely: expected maximization, K-means, Fuzzy c-Means (FCM), multilevel thresholding, region growing, and particle swarm optimization. The results were compared against the manual tracings. Among the other five segmentation techniques, FCM achieves the highest Jaccard Index (0.73 ± 0.06) and Dice Similarity Coefficient (0.82 ± 0.08) values. Statistical analysis (t-test, Mann Whitney U test, Wilcoxon test, Chi-Square test, and Kolmogorov–Smirnov test) and graphical analysis (Bland Altman and Regression plots) further prove the stability and reliability of the segmentation methods. Segmentation using FCM demonstrates the most accurate results and can be employed for the detection of breast cancer in the mammographic images. Further, it is concluded that computer-aided lesion detection systems can be used to assist Radiologists in routine clinical practice for the detection of breast tumors in mammographic images.


Mammogram Lesion detection Breast cancer Performance evaluation Manual delineations 



This work was supported by the Chhatisgarh Council of Science & Technology, Raipur, India (grant number 2481/CCOST/MRP/2016). The authors of this article are extremely grateful to them.


  1. 1.
    Al-Faris AQ, Umi Kalthum N, MatIsa NA, Shuaib IL (2014) Computer-aided segmentation system for breast MRI tumour using modified automatic seeded region growing (BMRI-MASRG). J Digit Imaging 27:133–144CrossRefGoogle Scholar
  2. 2.
    Arodź T, Kurdziel M, Popiela TJ, Sevre EO, Yuen DA (2006) Detection of clustered microcalcifications in small field digital mammography. Comput Methods Prog Biomed 81(1):56–65CrossRefGoogle Scholar
  3. 3.
    Banchhor SK, Londhe ND, Saba L, Radeva P, Laird JR, Suri JS (2017) Relationship between automated coronary calcium volumes and a set of manual coronary lumen volume, vessel volume and atheroma volume in Japanese diabetic cohort. J Clin Diagn Res 11(6):TC09Google Scholar
  4. 4.
    Breast cancer statistics, How common is breast cancer? American cancer society. Online document available at Accessed 4 Nov 2019
  5. 5.
    Chen Z, Zwiggelaar R A modified fuzzy c-means algorithm for breast tissue density segmentation in mammograms. In: Information technology and applications in biomedicine (ITAB), 2010 10th IEEE international conference on 2010 Nov 3. IEEE, pp 1–4Google Scholar
  6. 6.
    Choi JY, Kim DH, Plataniotis KN, Ro YM (2016) Classifier ensemble generation and selection with multiple feature representations for classification applications in computer-aided detection and diagnosis on mammography. Expert Syst Appl 46:106–121CrossRefGoogle Scholar
  7. 7.
    Ciecholewski M (2017) Malignant and benign mass segmentation in mammograms using active contour methods. Symmetry 9(11):277CrossRefGoogle Scholar
  8. 8.
    Costa DD, Campos LF, Barros AK (2011) Classification of breast tissue in mammograms using efficient coding. Biomed Eng Online 10:55CrossRefGoogle Scholar
  9. 9.
    Dheeba J, Singh NA, Selvi ST (2014) Computer-aided detection of breast cancer on mammograms: a swarm intelligence optimized wavelet neural network approach. J Biomed Inform 49:45–52CrossRefGoogle Scholar
  10. 10.
    Duijm LE, Louwman MW, Groenewoud JH, Van de Poll-Franse LV, Fracheboud J, Coebergh JW (2009) Inter-observer variability in mammography screening and effect of type and number of readers on screening outcome. Br J Cancer 100(6):901CrossRefGoogle Scholar
  11. 11.
    DuPrel JB, Röhrig B, Hommel G, Blettner M (2010) Choosing statistical tests: part 12 of a series on evaluation of scientific publications. Dtsch Arztebl Int 107(19):343Google Scholar
  12. 12.
    Elmore JG, Wells CK, Lee CH, Howard DH, Feinstein AR (1994) Variability in Radiologists' interpretations of mammograms. N Engl J Med 331(22):1493–1499CrossRefGoogle Scholar
  13. 13.
    Elmoufidi A, El Fahssi K, Jai-Andaloussi S, Sekkaki A (2015) Automatically density based breast segmentation for mammograms by using dynamic K-means algorithm and seed based region growing. In: Instrumentation and measurement technology conference (I2MTC), 2015 IEEE international. IEEE, pp 533–538Google Scholar
  14. 14.
    Fan J, Zeng G, Body M, Hacid MS (2005) Seeded region growing: an extensive and comparative study. Pattern Recogn Lett 26(8):1139–1156CrossRefGoogle Scholar
  15. 15.
    GokilaDeepa G (2012) Mammogram image segmentation using fuzzy hybrid with particle swarm optimization (PSO). International Journal of Engineering and Innovative Technology (IJEIT) 2(6)Google Scholar
  16. 16.
    Guliato D, Rangayyan RM, Carnielli WA, Zuffo JA, Desautels JL (2003) Segmentation of breast tumors in mammograms using fuzzy sets. J Electronic Imaging 12(3):369–379CrossRefGoogle Scholar
  17. 17.
    Gumaei A, El-Zaart A, Hussien M, Berbar M Breast segmentation using k-means algorithm with a mixture of gamma distributions. In: Broadband networks and fast internet (RELABIRA), 2012 symposium on 2012 May 28. IEEE, pp 97–102Google Scholar
  18. 18.
    Harrabi R, Braiek EB (2012) Color image segmentation using multi-level thresholding approach and data fusion techniques: application in the breast cancer cells images. E J Image Video Proc 2012(1):11CrossRefGoogle Scholar
  19. 19.
    He W, Hogg P, Juette A, Denton ER, Zwiggelaar R (2015) Breast image pre-processing for mammographic tissue segmentation. Comput Biol Med 67:61–73CrossRefGoogle Scholar
  20. 20.
    Jalalian A, Mashohor S, Mahmud R, Karasfi B, Saripan MI, Ramli AR (2017) Foundation and methodologies in computer-aided diagnosis systems for breast cancer detection. EXCLI J 16:113Google Scholar
  21. 21.
    Jen CC, Yu SS (2015) Automatic detection of abnormal mammograms in mammographic images. Expert Syst Appl 42(6):3048–3055CrossRefGoogle Scholar
  22. 22.
    Karmilasari SW, Hermita M, Agustiyani NP, Hanum Y, Lussiana ET (2014) Sample k-means clustering method for determining the stage of breast cancer malignancy based on cancer size on mammogram image basis. IJACSA. Int J Adv Comput Sci Appl 5(3):86–90Google Scholar
  23. 23.
    Keller B, Nathan D, Wang Y, Zheng Y, Gee J, Conant E, Kontos D (2011) Adaptive multi-cluster fuzzy C-means segmentation of breast parenchymal tissue in digital mammography. In: International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 562–569Google Scholar
  24. 24.
    Keller BM, Nathan DL, Gavenonis SC, Chen J, Conant EF, Kontos D (2013) Reader variability in breast density estimation from full-field digital mammograms: the effect of image postprocessing on relative and absolute measures. Acad Radiol 20(5):560–568CrossRefGoogle Scholar
  25. 25.
    Li Y, Brennan PC, Lee W, Nickson C, Pietrzyk MW, Ryan EA (2015) An investigation into the consistency in mammographic density identification by radiologists: effect of radiologist expertise and mammographic appearance. J Digit Imaging 28(5):626–632CrossRefGoogle Scholar
  26. 26.
    Malvia S, Bagadi SA, Dubey US, Saxena S (2017) Epidemiology of breast cancer in Indian women. Asia Pac J Clin Oncol 13(4):289–295CrossRefGoogle Scholar
  27. 27.
    MedCalc:- Accessed 3 July 2019
  28. 28.
    Melouah A (2015) Comparison of automatic seed generation methods for breast tumor detection using region growing technique. In: IFIP international conference on computer science and its Applications_x000D_. Springer, Cham, pp 119–128Google Scholar
  29. 29.
    Neto OP, Carvalho O, Sampaio W, Corrêa A, Paiva A Automatic segmentation of masses in digital mammograms using particle swarm optimization and graph clustering. In: Systems, signals and image processing (IWSSIP), 2015 international conference on 2015 Sep 10. IEEE, pp 109–112Google Scholar
  30. 30.
    Ng KH, Muttarak M (2003) Advances in mammography have improved early detection of breast cancer. J HK Coll Radiol 6:126–131Google Scholar
  31. 31.
    Nurhasanah, Sampurno J, Faryuni ID, Ivansyah O (2016) Automated analysis of image mammogram for breast cancer diagnosis. In: AIP conference proceedings, vol. 1719, no. 1. AIP Publishing, pp 030036Google Scholar
  32. 32.
    Oliver A, Freixenet J, Marti J, Perez E, Pont J, Denton ER, Zwiggelaar R (2010) A review of automatic mass detection and segmentation in mammographic images. Med Image Anal 14(2):87–110CrossRefGoogle Scholar
  33. 33.
    Pompe E, de Jong PA, De Jong WU, Takx RA, Eikendal AL, Willemink MJ, Oudkerk M, Budde RP, Lammers JW, Hoesein FA (2016) Inter-observer and inter-examination variability of manual vertebral bone attenuation measurements on computed tomography. Eur Radiol 26(9):3046–3053CrossRefGoogle Scholar
  34. 34.
    Raja NS, Sukanya SA, Nikita Y (2015) Improved PSO based multi-level thresholding for cancer infected breast thermal images using Otsu. Procedia Computer Science 48:524–529CrossRefGoogle Scholar
  35. 35.
    Raju NG, Rao NC (2013) Particle swarm optimization methods for image segmentation applied in mammography. Journal of Engineering Research and Applications 3(6):1572–1579. ISSN: 2248-9622Google Scholar
  36. 36.
    Ramani R, Valarmathy S, Vanitha NS (2013) Breast cancer detection in mammograms based on clustering techniques-a survey. Int J Comput Appl 62(11)CrossRefGoogle Scholar
  37. 37.
    Redondo A, Comas M, Macia F, Ferrer F, Murta-Nascimento C, Maristany MT, Molins E, Sala M, Castells X (2012) Inter-and intra radiologist variability in the BI-RADS assessment and breast density categories for screening mammograms. Br J Radiol 85(1019):1465–1470CrossRefGoogle Scholar
  38. 38.
    Rejani Y, Selvi ST (2009) Breast Cancer detection using multilevel thresholding. arXiv preprint arXiv:0911.0490Google Scholar
  39. 39.
    Saba L, Than JC, Noor NM, Rijal OM, Kassim RM, Yunus A, Ng CR, Suri JS (2016) Inter-observer variability analysis of automatic lung delineation in normal and disease patients. J Med Syst 40(6):142CrossRefGoogle Scholar
  40. 40.
    Saba L, Banchhor SK, Araki T, Suri HS, Londhe ND, Laird JR, Viskovic K, Suri JS (2018) Intra-and inter-operator reproducibility analysis of automated cloud-based carotid intima media thickness ultrasound measurement. J Clin Diagn Res 12(2):KC01-KC11Google Scholar
  41. 41.
    Saba L, Banchhor SK, Araki T, Viskovic K, Londhe ND, Laird JR, Suri HS, Suri JS (2018) Intra-and inter-operator reproducibility of automated cloud-based carotid lumen diameter ultrasound measurement. Indian Heart J 70:649–664CrossRefGoogle Scholar
  42. 42.
    Sadad T, Munir A, Saba T, Hussain A (2018) Fuzzy C-means and region growing based classification of tumor from mammograms using hybrid texture feature. J Comput Sci 29:34–45MathSciNetCrossRefGoogle Scholar
  43. 43.
    Saha A, Grimm LJ, Harowicz M, Ghate SV, Kim C, Walsh R, Mazurowski MA (2016) Interobserver variability in identification of breast tumors in MRI and its implications for prognostic biomarkers and radiogenomics. Med Phys 43(8 Part1):4558–4564CrossRefGoogle Scholar
  44. 44.
    Sathish A, Sundaram JM (2004) A comparative study on K-means and fuzzy C-means algorithm for breast cancer analysis. International Journal of Computational Intelligence and Informatics. H. Simpson, Dumb robots, 3rd edn. UOS Press, Springfield, pp 6–9Google Scholar
  45. 45.
    Satyendra SA, Pawar MM (2017) Segmentation of breast images using Gaussian mixture models. Int J Adv Res Ideas Innov Technol 3(3):437–441. ISSN: 2454-132XGoogle Scholar
  46. 46.
    Senthilkumar B, Umamaheswari G, Karthik J A novel region growing segmentation algorithm for the detection of breast cancer. In: Computational intelligence and computing research (ICCIC), 2010 IEEE international conference on 2010 Dec 28. IEEE, pp 1–4Google Scholar
  47. 47.
    Sheshadri HS, Kandaswamy A (2005) Detection of breast cancer by mammogram image segmentation. J Cancer Res Ther 1(4):232CrossRefGoogle Scholar
  48. 48.
    Siegel R, Naishadham D, Jemal A (2013a) Cancer statistics. CA Cancer J Clin 63:11–30CrossRefGoogle Scholar
  49. 49.
    Spandana P, Rao KM (2013) Novel image processing techniques for early detection of breast cancer, mat lab and lab view implementation. In: Point-of-Care Healthcare Technologies (PHT), pp 105–108Google Scholar
  50. 50.
    Suckling J et al (1994) The Mammographic Image Analysis Society Digital Mammogram Database Exerpta Medica. Int Congr Ser 1069:375–378Google Scholar
  51. 51.
    Sujata RB, Dhiman R, Chugh TS (2012) An evaluation of two mammography segmentation techniques. International Journal of Advanced Computer Research 2 Number-4(7):2277–7970. (ISSN (print): 2249-7277 ISSN (online)Google Scholar
  52. 52.
    Survey by Indian cancer society (2013) Indian cancer societyGoogle Scholar
  53. 53.
    Thyagarajan R, Murugavalli S (2012) Segmentation of digital breast tomograms using clustering techniques. In: India conference (INDICON), 2012 annual IEEE. IEEE, pp 1090–1094Google Scholar
  54. 54.
    Valarmathie P, Sivakrithika V, Dinakaran K (2016) Classification of mammogram masses using selected texture, shape and margin features with multilayer perceptron classifier. Biomedical research India, Special Issue page no. s310–s313Google Scholar
  55. 55.
    Vedanarayanan V (2017) Advanced image segmentation techniques for accurate isolation of abnormality to enhance breast cancer detection in digital mammographs. Biomed Res 28(6):2753–2757Google Scholar
  56. 56.
    Vesal S, Ravikumar N, Ellman S, Maier A (2018) Comparative analysis of unsupervised algorithms for breast MRI lesion segmentation. In: Bildverarbeitung für die Medizin 2018. Springer Vieweg, Berlin, pp 257–262CrossRefGoogle Scholar
  57. 57.
    Wang Y, Li J, Gao X (2014) Latent feature mining of spatial and marginal characteristics for mammographic mass classification. Neurocomputing 144:107–118CrossRefGoogle Scholar
  58. 58.
    Wang J, Jing H, Wernick MN, Nishikawa RM, Yang Y (2014) Analysis of perceived similarity betweenpairs of microcalcification clusters in mammograms. Med Phys 41(5):051904CrossRefGoogle Scholar
  59. 59.
    Yuvaraj K, Ragupathy US (2013) Automatic mammographic mass segmentation based on region growing technique. In: 3rd international conference on electronics, biomedical engineering and its applications (ICEBEA'2013), pp 29–30Google Scholar
  60. 60.
    Zheng Y, Keller BM, Ray S, Wang Y, Conant EF, Gee JC, Kontos D (2015) Parenchymal texture analysis in digital mammography: a fully automated pipeline for breast cancer risk assessment. Med Phys 42(7):4149–4160CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  • Bikesh K. Singh
    • 1
    Email author
  • Pankaj Jain
    • 1
  • Sumit K. Banchhor
    • 1
  • Kesari Verma
    • 2
  1. 1.Department of Biomedical EngineeringNational Institute of Technology RaipurRaipurIndia
  2. 2.Department of Computer ApplicationsNational Institute of Technology RaipurRaipurIndia

Personalised recommendations