Effect of training on ultrasonography (US) BI-RADS features for radiology residents: a multicenter study comparing performances after training

  • Jung Hyun Yoon
  • Hye Sun Lee
  • You Me Kim
  • Ji Hyun Youk
  • Sung Hun Kim
  • Sun Hye Jeong
  • Ji Young Hwang
  • Jin Hee Moon
  • Young Mi Park
  • Min Jung KimEmail author



To evaluate the effect of training radiology residents on breast ultrasonography (US) according to the Breast Imaging Reporting And Data System (BI-RADS) and the factors that influence the training effect.


This multicenter, prospective study was approved by eight institutional review boards. From September 2013 to July 2014, 248 breast masses in 227 women were included for US image acquisition. Representative B-mode and video images of the breast masses were recorded, among which 54 cases were included in the education set and 66 in the test set. Sixty-one radiology residents scheduled for breast imaging training individually reviewed the test set, immediately before, 1 month after, and 6 months after training. Diagnostic performances and US descriptors of the residents were evaluated and compared against those of expert radiologists.


Agreements between residents and experienced radiologists showed improvement after training, while agreements between post-training and post-6-month training descriptors did not show significant differences (all p > 0.05, respectively). Sensitivity, negative predictive value (NPV), and AUC were significantly improved for residents post-training and post-6-month training (all p < 0.05), while approximating the performances of expert radiologists except for AUC (0.836, 0.840, and 0.908, respectively, p < 0.05). Low levels of pre-training AUC, total number of breast US examinations, and the number of sessions per week that residents were involved in were factors influencing the improvement of AUC.


Training using education material dedicated for breast US imaging effectively improved the diagnostic performances of radiology residents and agreements with experienced radiologists on US BI-RADS features.

Key Points

• Agreements on lesion descriptors between residents and experienced radiologists showed improvement after training, regardless of test point.

• Sensitivity, NPV, and AUC were significantly improved for residents in post-training and post-6-month training (all p < 0.05).

• Low levels of pre-training AUC, total number of breast US examinations, and the number of sessions per week that residents were involved in were factors influencing the improvement of AUC.


Breast Ultrasound Education Residency Training 



American College of Radiology


Area under the receiver operating characteristic curve


Breast Imaging Reporting And Data System


Negative predictive value


Positive predictive value





This study has received funding by the Korean Society of Breast Imaging & Korean Society for Breast Screening (KSBI & KSFBS-2013-No.001).

Compliance with ethical standards


The scientific guarantor of this publication is Min Jung Kim.

Conflict of interest

The authors of this manuscript declare no relationships with any companies, whose products or services may be related to the subject matter of the article.

Statistics and biometry

One of the authors has significant statistical expertise, Hye Sun Lee.

Informed consent

Written informed consent was obtained from all subjects (patients) in this study.

Ethical approval

Institutional Review Board approval was obtained.

Study subjects or cohorts overlap

Some study subjects or cohorts have been previously reported in Ultrasound Med Biol 2016;42(9):2083–2088.

The performances of the experienced radiologists in reviewing static and real-time video clips obtained from the patients included in this study have been published. While our study focuses on comparing the performances and agreements between residents and experienced radiologists on breast US examinations, the prior study focuses on comparing the diagnostic performances between static and video clip images.


• prospective

• cross-sectional study/observational

• multicenter study

Supplementary material

330_2018_5934_MOESM1_ESM.mp4 (9.9 mb)
ESM 1 (MP4 10,174 kb)
330_2018_5934_MOESM2_ESM.docx (11.5 mb)
ESM 2 (DOCX 11,742 kb)


  1. 1.
    Hooley RJ, Scoutt LM, Philpotts LE (2013) Breast ultrasonography: state of the art. Radiology 268:642–659CrossRefGoogle Scholar
  2. 2.
    Berg WA, Blume JD, Cormack JB et al (2008) Combined screening with ultrasound and mammography vs mammography alone in women at elevated risk of breast cancer. JAMA 299:2151–2163CrossRefPubMedCentralGoogle Scholar
  3. 3.
    Berg WA, Zhang Z, Lehrer D et al (2012) Detection of breast cancer with addition of annual screening ultrasound or a single screening MRI to mammography in women with elevated breast cancer risk. JAMA 307:1394–1404CrossRefPubMedCentralGoogle Scholar
  4. 4.
    American College of Radiology (2013) Breast imaging reporting and data system, 5th edn. American College of Radiology, Reston, VAGoogle Scholar
  5. 5.
    Lee HJ, Kim EK, Kim MJ et al (2008) Observer variability of Breast Imaging Reporting and Data System (BI-RADS) for breast ultrasound. Eur J Radiol 65:293–298CrossRefGoogle Scholar
  6. 6.
    Lee YJ, Choi SY, Kim KS, Yang PS (2016) Variability in observer performance between faculty members and residents using Breast Imaging Reporting and Data System (BI-RADS)-ultrasound, fifth edition (2013). Iran J Radiol 13:e28281PubMedCentralGoogle Scholar
  7. 7.
    Park CS, Lee JH, Yim HW et al (2007) Observer agreement using the ACR Breast Imaging Reporting and Data System (BI-RADS)-ultrasound, First Edition (2003). Korean J Radiol 8:397–402CrossRefPubMedCentralGoogle Scholar
  8. 8.
    Schwab F, Redling K, Siebert M, Schotzau A, Schoenenberger CA, Zanetti-Dällenbach R (2016) Inter- and intra-observer agreement in ultrasound BI-RADS classification and real-time elastography Tsukuba score assessment of breast lesions. Ultrasound Med Biol 42:2622–2629CrossRefGoogle Scholar
  9. 9.
    Berg WA, Campassi C, Langenberg P, Sexton MJ (2000) Breast Imaging Reporting and Data System: inter- and intraobserver variability in feature analysis and final assessment. AJR Am J Roentgenol 174:1769–1777CrossRefGoogle Scholar
  10. 10.
    Ortiz-Perez T, Trevino EJ, Sepulveda KA, Hilsenbeck SG, Wang T, Sedgwick EL (2013) Does formal instruction about the BI-RADS ultrasound lexicon result in improved appropriate use of the lexicon? AJR Am J Roentgenol 201:456–461CrossRefGoogle Scholar
  11. 11.
    Scheel JR, Peacock S, Orem J et al (2016) Improving breast ultrasound interpretation in Uganda using a condensed Breast Imaging Reporting and Data System. Acad Radiol 23:1271–1277CrossRefGoogle Scholar
  12. 12.
    Berg WA, Blume JD, Cormack JB, Mendelson EB (2012) Training the ACRIN 6666 investigators and effects of feedback on breast ultrasound interpretive performance and agreement in BI-RADS ultrasound feature analysis. AJR Am J Roentgenol 199:224–235CrossRefPubMedCentralGoogle Scholar
  13. 13.
    Monticciolo DL, Rebner M, Appleton CM et al (2013) The ACR/Society of Breast Imaging Resident and Fellowship Training Curriculum for Breast Imaging, updated. J Am Coll Radiol 10:207–210 e204CrossRefGoogle Scholar
  14. 14.
    Education and Practical Standards Committee, European Federation of Societies for Ultrasound in Medicine and Biology (2006) Minimum training recommendations for the practice of medical ultrasound. Ultraschall Med 27:79–105Google Scholar
  15. 15.
    Hooley RJ, Greenberg KL, Stackhouse RM, Geisel JL, Butler RS, Philpotts LE (2012) Screening US in patients with mammographically dense breasts: initial experience with Connecticut Public Act 09-41. Radiology 265:59–69CrossRefGoogle Scholar
  16. 16.
    Berg WA, Mendelson EB (2014) Technologistperformed handheld screening breast US imaging: how is it performed and what are the outcomes to date? Radiology 272(1):12–27Google Scholar
  17. 17.
    Berg WA, D’Orsi CJ, Jackson VP et al (2002) Does training in the Breast Imaging Reporting and Data System (BI-RADS) improve biopsy recommendations or feature analysis agreement with experienced breast imagers at mammography? Radiology 224:871–880CrossRefGoogle Scholar
  18. 18.
    Youk JH, Jung I, Yoon JH et al (2016) Comparison of inter-observer variability and diagnostic performance of the fifth edition of BI-RADS for breast ultrasound of static versus video images. Ultrasound Med Biol 42:2083–2088CrossRefGoogle Scholar
  19. 19.
    Cook DA, Beckman TJ (2010) Reflections on experimental research in medical education. Adv Health Sci Educ Theory Pract 15:455–464CrossRefGoogle Scholar

Copyright information

© European Society of Radiology 2019

Authors and Affiliations

  • Jung Hyun Yoon
    • 1
  • Hye Sun Lee
    • 2
  • You Me Kim
    • 3
  • Ji Hyun Youk
    • 4
  • Sung Hun Kim
    • 5
  • Sun Hye Jeong
    • 6
  • Ji Young Hwang
    • 7
  • Jin Hee Moon
    • 7
  • Young Mi Park
    • 8
  • Min Jung Kim
    • 1
    Email author
  1. 1.Department of Radiology, Research Institute of Radiological ScienceYonsei University, College of MedicineSeoulSouth Korea
  2. 2.Biostatistics Collaboration Unit, Medical Research CenterYonsei University, College of MedicineSeoulSouth Korea
  3. 3.Department of Radiology, Dankook University HospitalDankook University, College of MedicineCheonanSouth Korea
  4. 4.Department of Radiology, Gangnam Severance HospitalYonsei University, College of MedicineSeoulSouth Korea
  5. 5.Department of RadiologySeoul St. Mary’s HospitalSeoulSouth Korea
  6. 6.Department of RadiologySoonchunhyang University, Bucheon HospitalBucheonSouth Korea
  7. 7.Department of Radiology, Kangnam Sacred HospitalHallym University, College of MedicineSeoulSouth Korea
  8. 8.Department of Radiology, Inje University Busan Paik HospitalInje University College of MedicineGimhaeSouth Korea

Personalised recommendations