Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Interreader variability in prostate MRI reporting using Prostate Imaging Reporting and Data System version 2.1

Abstract

Objectives

To evaluate the agreement among readers with different expertise in detecting suspicious lesions at prostate multiparametric MRI using Prostate Imaging Reporting and Data System (PI-RADS) version 2.1.

Methods

We evaluated 200 consecutive biopsy-naïve or previously negative biopsy men who underwent MRI for clinically suspected prostate cancer (PCa) between May and September 2017. Of them, 132 patients underwent prostate biopsy. Seven radiologists (four dedicated uro-radiologists and three non-dedicated abdominal radiologists) reviewed and scored all MRI examinations according to PI-RADS v2.1. Agreement on index lesion detection was evaluated with Conger’s k coefficient, agreement coefficient 1 (AC1), percentage of agreement (PA), and indexes of specific positive and negative agreement. Clinical and radiological features that may influence variability were evaluated.

Results

Agreement in index lesion detection among all readers was substantial (AC1 0.738; 95% CI 0.695–0.782); dedicated radiologists showed higher agreement compared with non-dedicated readers. Clinical and radiological parameters that positively influenced agreement were PSA density ≥ 0.15 ng/mL/cc, pre-MRI high risk for PCa, positivity threshold of PI-RADS score 4 + 5, PZ lesions, homogeneous signal intensity of the PZ, and subjectively easy interpretation of MRI. Positive specific agreement was significantly higher among dedicated readers, up to 93.4% (95% CI 90.7–95.4) in patients harboring csPCa. Agreement on absence of lesions was excellent for both dedicated and non-dedicated readers (respectively 85.1% [95% CI 78.4–92.3] and 82.0% [95% CI 77.2–90.1]).

Conclusions

Agreement on index lesion detection among radiologists of various experiences is substantial to excellent using PI-RADS v2.1. Concordance on absence of lesions is excellent across readers’ experience.

Key Points

• Agreement on index lesion detection among radiologists of various experiences is substantial to excellent using PI-RADS v2.1.

• Concordance between experienced readers is higher than between less-experienced readers.

• Concordance on absence of lesions is excellent across readers’ experience.

This is a preview of subscription content, log in to check access.

Fig. 1

Abbreviations

csPCa:

Clinically significant prostate cancer

mpMRI:

Multiparametric magnetic resonance imaging

PA:

Percentage of agreement

PCa:

Prostate cancer

P neg :

Proportion of negative agreement

P pos :

Proportion of positive agreement

PSAD:

PSA density

PZ:

Peripheral zone

SI:

Signal intensity

TRUS-Bx:

Transrectal ultrasound guided biopsy

TZ:

Transitional zone

References

  1. 1.

    Kasivisvanathan V, Rannikko AS, Borghi M et al (2018) MRI-targeted or standard biopsy for prostate-cancer diagnosis. N Engl J Med 378:1767–1777. https://doi.org/10.1056/NEJMoa1801993

  2. 2.

    Ahmed HU, El-Shater Bosaily A, Brown LC et al (2017) Diagnostic accuracy of multi-parametric MRI and TRUS biopsy in prostate cancer (PROMIS): a paired validating confirmatory study. Lancet. https://doi.org/10.1016/S0140-6736(16)32401-1

  3. 3.

    Mottet N, van den Bergh RCN, Briers E et al (2019) Guidelines on prostate cancer 2019. Eur Assoc Urol. Available via https://uroweb.org/guideline/prostate-cancer/

  4. 4.

    Drost FJH, Osses DF, Nieboer D et al (2019) Prostate MRI, with or without MRI-targeted biopsy, and systematic biopsy for detecting prostate cancer. Cochrane Database Syst Rev. https://doi.org/10.1002/14651858.CD012663.pub2

  5. 5.

    Barentsz JO, Richenberg J, Clements R et al (2012) ESUR prostate MR guidelines 2012. Eur Radiol. https://doi.org/10.1007/s00330-011-2377-y

  6. 6.

    Weinreb JC, Barentsz JO, Choyke PL et al (2016) PI-RADS Prostate Imaging - Reporting and Data System: 2015, version 2. Eur Urol. https://doi.org/10.1016/j.eururo.2015.08.052

  7. 7.

    Turkbey B, Rosenkrantz AB, Haider MA et al (2019) Prostate Imaging Reporting and Data System Version 2.1: 2019 Update of Prostate Imaging Reporting and Data System version 2. Eur Urol. https://doi.org/10.1016/j.eururo.2019.02.033

  8. 8.

    Rosenkrantz AB, Ginocchio LA, Cornfeld D et al (2016) Interobserver reproducibility of the PI-RADS version 2 lexicon: a multicenter study of six experienced prostate radiologists. Radiology. https://doi.org/10.1148/radiol.2016152542

  9. 9.

    Schimmöller L, Quentin M, Arsov C et al (2013) Inter-reader agreement of the ESUR score for prostate MRI using in-bore MRI-guided biopsies as the reference standard. Eur Radiol. https://doi.org/10.1007/s00330-013-2922-y

  10. 10.

    Muller BG, Shih JH, Sankineni S et al (2015) Prostate cancer: interobserver agreement and accuracy with the revised Prostate Imaging Reporting and Data System at multiparametric MR imaging. Radiology. https://doi.org/10.1148/radiol.2015142818

  11. 11.

    Sonn GA, Fan RE, Ghanouni P et al (2018) Prostate magnetic resonance imaging interpretation varies substantially across radiologists. Eur Urol Focus. https://doi.org/10.1016/j.euf.2017.11.010

  12. 12.

    Pickersgill NA, Vetter JM, Andriole GL et al (2018) Accuracy and variability of prostate multiparametric magnetic resonance imaging interpretation using the prostate imaging reporting and data system: a blinded comparison of radiologists. Eur Urol Focus. https://doi.org/10.1016/j.euf.2018.10.008

  13. 13.

    Smith CP, Harmon SA, Barrett T et al (2019) Intra- and interreader reproducibility of PI-RADSv2: a multireader study. J Magn Reson Imaging. https://doi.org/10.1002/jmri.26555

  14. 14.

    Girometti R, Giannarini G, Greco F et al (2019) Interreader agreement of PI-RADS v. 2 in assessing prostate cancer with multiparametric MRI: a study using whole-mount histology as the standard of reference. J Magn Reson Imaging 49:546–555. https://doi.org/10.1002/jmri.26220

  15. 15.

    Feinstein AR, Cicchetti DV (1990) High agreement but low kappa: I. the problems of two paradoxes. J Clin Epidemiol. https://doi.org/10.1016/0895-4356(90)90158-L

  16. 16.

    Cicchetti DV, Feinstein AR (1990) High agreement but low kappa: II. Resolving the paradoxes. J Clin Epidemiol. https://doi.org/10.1016/0895-4356(90)90159-M

  17. 17.

    Shih JH, Greer MD, Turkbey B (2018) The problems with the kappa statistic as a metric of interobserver agreement on lesion detection using a third-reader approach when locations are not prespecified. Acad Radiol. https://doi.org/10.1016/j.acra.2018.01.030

  18. 18.

    Hötker AM, Dappa E, Mazaheri Y et al (2019) The influence of background signal intensity changes on cancer detection in prostate MRI. Am J Roentgenol 212:823–829. https://doi.org/10.2214/AJR.18.20295

  19. 19.

    Stabile A, Dell’Oglio P, De Cobelli F et al (2018) Association between Prostate Imaging Reporting and Data System (PI-RADS) score for the index lesion and multifocal, clinically significant prostate cancer. Eur Urol Oncol 1:29–36. https://doi.org/10.1016/j.euo.2018.01.002

  20. 20.

    Dell’Oglio P, Stabile A, Soligo M et al (2019) There is no way to avoid systematic prostate biopsies in addition to multiparametric magnetic resonance imaging targeted biopsies. Eur Urol Oncol 3:112–118. https://doi.org/10.1016/j.euo.2019.03.002

  21. 21.

    Conger AJ (1980) Integration and generalization of kappas for multiple raters. Psychol Bull. https://doi.org/10.1037/0033-2909.88.2.322

  22. 22.

    Gwet KL (2008) Computing inter-rater reliability and its variance in the presence of high agreement. Br J Math Stat Psychol 61:29–48. https://doi.org/10.1348/000711006X126600

  23. 23.

    Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics. https://doi.org/10.2307/2529310

  24. 24.

    Rais-Bahrami S, Siddiqui MM, Vourganti S et al (2015) Diagnostic value of biparametric magnetic resonance imaging (MRI) as an adjunct to prostate-specific antigen (PSA)-based detection of prostate cancer in men without prior biopsies. BJU Int 115:381–388. https://doi.org/10.1111/bju.12639

  25. 25.

    Roobol MJ, Verbeek JFM, van der Kwast T, Kümmerlin IP, Kweldam CF, van Leenders GJLH (2017) Improving the Rotterdam European randomized study of screening for prostate cancer risk calculator for initial prostate biopsy by incorporating the 2014 International Society of Urological Pathology Gleason Grading and Cribriform growth. Eur Urol. https://doi.org/10.1016/j.eururo.2017.01.033

  26. 26.

    Greer MD, Brown AM, Shih JH et al (2017) Accuracy and agreement of PIRADSv2 for prostate cancer mpMRI: a multireader study. J Magn Reson Imaging 45:579–585. https://doi.org/10.1002/jmri.25372

  27. 27.

    Greer MD, Shih JH, Lay N et al (2019) Interreader variability of prostate imaging reporting and data system version 2 in detecting and assessing prostate cancer lesions at prostate MRI. AJR Am J Roentgenol 212:1197–1205. https://doi.org/10.2214/AJR.18.20536

  28. 28.

    Rouvière O, Puech P, Renard-Penna R et al (2019) Use of prostate systematic and targeted biopsy on the basis of multiparametric MRI in biopsy-naive patients (MRI-FIRST): a prospective, multicentre, paired diagnostic study. Lancet Oncol 20:100–109. https://doi.org/10.1016/S1470-2045(18)30569-2

  29. 29.

    Luzzago S, Petralia G, Musi G et al (2019) Multiparametric magnetic resonance imaging second opinion may reduce the number of unnecessary prostate biopsies: time to improve radiologists’ training program? Clin Genitourin Cancer 17:88–96. https://doi.org/10.1016/j.clgc.2018.10.006

Download references

Funding

This work has not received any funding.

Author information

Correspondence to Giorgio Brembilla.

Ethics declarations

Guarantor

The scientific guarantor of this publication is Francesco De Cobelli, M.D.

Conflict of interest

The authors of this manuscript declare no relationships with any companies whose products or services may be related to the subject matter of the article.

Statistics and biometry

Paola Maria Vittoria Rancoita kindly provided statistical advice for this manuscript, and is one of the authors.

Informed consent

Written informed consent was obtained from all subjects (patients) in this study.

Ethical approval

Institutional Review Board approval was obtained.

Study subjects or cohorts overlap

None of the study subjects or cohorts has been previously reported.

Methodology

• Retrospective

• Observational

• Performed at one institution

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

ESM 1

(DOCX 26 kb)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Brembilla, G., Dell’Oglio, P., Stabile, A. et al. Interreader variability in prostate MRI reporting using Prostate Imaging Reporting and Data System version 2.1. Eur Radiol (2020). https://doi.org/10.1007/s00330-019-06654-2

Download citation

Keywords

  • Magnetic resonance imaging
  • Prostate cancer
  • Inter-observer variability