Journal of Digital Imaging

, Volume 32, Issue 3, pp 417–419 | Cite as

Evaluating Completeness of a Radiology Glossary Using Iterative Refinement

  • Peter Y. W. Chan
  • Charles E. KahnJrEmail author


A lay-language glossary of radiology, built to help patients better understand the content of their radiology reports, has been analyzed for its coverage and readability, but not for its completeness. We present an iterative method to sample radiology reports, identify “missing” terms, and measure the glossary’s completeness. We hypothesized that the refinement process would reduce the number of missing terms to fewer than 1 per report. A random sample of 1000 radiology reports from a large US academic health system was divided into 10 cohorts of 100 reports each. Each cohort was reviewed in sequence by two investigators to identify terms (single words and multi-word phrases) absent from the glossary. Terms marked as new were added to the glossary and hence was shown as matched in subsequent cohorts. This HIPAA-compliant study was IRB-approved; informed consent was waived. The refinement process added a mean of 288.0 new terms per 100 reports in the first 5 cohorts vs. a mean of 66.0 new terms per 100 reports in the last 5 cohorts; the difference was statistically significant (p < .01). After reviewing 500 reports, the review process found fewer than 1 new term per report in each of 500 subsequent reports. The findings suggest that 500 to 1000 reports is adequate to test the completeness of a glossary, and that the glossary after iterative refinement achieved a high level of completeness to cover the vocabulary of radiology reports.


Radiology reports Patient-centered care Vocabulary Patient portals Glossary 


  1. 1.
    Kemp JL, Mahoney MC, Mathews VP, Wintermark M, Yee J, Brown SD: Patient-centered radiology: Where are we, where do we want to be, and how do we get there? Radiology 285:601–608, 2017CrossRefPubMedGoogle Scholar
  2. 2.
    Itri JN: Patient-centered radiology. RadioGraphics 35:1835–1846, 2015CrossRefPubMedGoogle Scholar
  3. 3.
    Bruno MA, Petscavage-Thomas JM, Mohr MJ, Bell SK, Brown SD: The "open letter": radiologists' reports in the era of patient web portals. J Am Coll Radiol 11:863–867, 2014CrossRefPubMedGoogle Scholar
  4. 4.
    Gefen R, Bruno MA, Abujudeh HH: Online portals: Gateway to patient-centered radiology. AJR Am J Roentgenol:1–5, 2017Google Scholar
  5. 5.
    Henshaw D, Okawa G, Ching K, Garrido T, Qian H, Tsai J: Access to radiology reports via an online patient portal: Experiences of referring physicians and patients. J Am Coll Radiol 12 e581:582–586, 2015CrossRefGoogle Scholar
  6. 6.
    Miles RC, Hippe DS, Elmore JG, Wang CL, Payne TH, Lee CI: Patient access to online radiology reports. frequency and sociodemographic characteristics associated with use. Acad Radiol 23:1162–1169, 2016PubMedGoogle Scholar
  7. 7.
    How to write easy-to-read health materials. Available at Accessed 30 January 2017
  8. 8.
    Weiss BD: Health literacy and patient safety: Help patients understand: Manual for clinicians. Chicago, IL: American Medical Association Foundation. American Medical Association, 2007Google Scholar
  9. 9.
    Sierra AE, Bisesi MA, Rosenbaum TL, Potchen EJ: Readability of the radiologic report. Invest Radiol 27:236–239, 1992CrossRefPubMedGoogle Scholar
  10. 10.
    Trofimova A, Vey BL, Safdar NM, Duszak R Jr, Kadom N: Radiology report readability: An opportunity to improve patient communication. J Am Coll Radiol: Published online:, 2018
  11. 11.
    Oh SC, Cook TS, Kahn CE Jr: PORTER: A prototype system for patient-oriented radiology reporting. J Digit Imaging 29:450–454, 2016CrossRefPubMedPubMedCentralGoogle Scholar
  12. 12.
    Martin-Carreras T, Kahn CE Jr: Coverage and readability of information resources to help patients understand radiology reports. J Am Coll Radiol., 2017
  13. 13.
    Mervak BM, Davenport MS, Flynt KA, Kazerooni EA, Weadock WJ: What the patient wants: An analysis of radiology-related inquiries from a web-based patient portal. J Am Coll Radiol 13:1311–1318, 2016CrossRefPubMedGoogle Scholar
  14. 14.
    Okawa G, Ching K, Qian H, Feng Y: Automatic release of radiology reports via an online patient portal. J Am Coll Radiol 14:1219–1221, 2017CrossRefPubMedGoogle Scholar
  15. 15.
    Otte-Trojel T, de Bont A, Rundall TG, van de Klundert J: What do we know about developing patient portals? A systematic literature review. J Am Med Informatics Assoc 23:e162–e168, 2016CrossRefGoogle Scholar
  16. 16.
    Lee CI, Langlotz CP, Elmore JG: Implications of direct patient online access to radiology reports through patient web portals. J Am Coll Radiol 13:1608–1614, 2016CrossRefPubMedGoogle Scholar
  17. 17.
    Cook TS, Oh SC, Kahn CE Jr: Patients' use and evaluation of an online system to annotate radiology reports with lay-language definitions. Acad Radiol 24:1169–1174, 2017CrossRefPubMedGoogle Scholar
  18. 18.
    Chen W, Durkin C, Huang Y, Adler B, Rust S, Lin S: Simplified readability metric drives improvement of radiology reports: An experiment on ultrasound reports at a pediatric hospital. J Digit Imaging 30:710–717, 2017CrossRefPubMedPubMedCentralGoogle Scholar
  19. 19.
    Hansberry DR, Agarwal N, Baker SR: Health literacy and online educational resources: An opportunity to educate patients. AJR Am J Roentgenol 204:111–116, 2015CrossRefPubMedGoogle Scholar
  20. 20.
    Gunn AJ, Mangano MD, Choy G, Sahani DV: Rethinking the role of the radiologist: Enhancing visibility through both traditional and nontraditional reporting practices. RadioGraphics 35:416–423, 2015CrossRefPubMedGoogle Scholar

Copyright information

© Society for Imaging Informatics in Medicine 2018

Authors and Affiliations

  1. 1.Department of RadiologyUniversity of PennsylvaniaPhiladelphiaUSA

Personalised recommendations