To evaluate the agreement among readers with different expertise in detecting suspicious lesions at prostate multiparametric MRI using Prostate Imaging Reporting and Data System (PI-RADS) version 2.1.
We evaluated 200 consecutive biopsy-naïve or previously negative biopsy men who underwent MRI for clinically suspected prostate cancer (PCa) between May and September 2017. Of them, 132 patients underwent prostate biopsy. Seven radiologists (four dedicated uro-radiologists and three non-dedicated abdominal radiologists) reviewed and scored all MRI examinations according to PI-RADS v2.1. Agreement on index lesion detection was evaluated with Conger’s k coefficient, agreement coefficient 1 (AC1), percentage of agreement (PA), and indexes of specific positive and negative agreement. Clinical and radiological features that may influence variability were evaluated.
Agreement in index lesion detection among all readers was substantial (AC1 0.738; 95% CI 0.695–0.782); dedicated radiologists showed higher agreement compared with non-dedicated readers. Clinical and radiological parameters that positively influenced agreement were PSA density ≥ 0.15 ng/mL/cc, pre-MRI high risk for PCa, positivity threshold of PI-RADS score 4 + 5, PZ lesions, homogeneous signal intensity of the PZ, and subjectively easy interpretation of MRI. Positive specific agreement was significantly higher among dedicated readers, up to 93.4% (95% CI 90.7–95.4) in patients harboring csPCa. Agreement on absence of lesions was excellent for both dedicated and non-dedicated readers (respectively 85.1% [95% CI 78.4–92.3] and 82.0% [95% CI 77.2–90.1]).
Agreement on index lesion detection among radiologists of various experiences is substantial to excellent using PI-RADS v2.1. Concordance on absence of lesions is excellent across readers’ experience.
• Agreement on index lesion detection among radiologists of various experiences is substantial to excellent using PI-RADS v2.1.
• Concordance between experienced readers is higher than between less-experienced readers.
• Concordance on absence of lesions is excellent across readers’ experience.
This is a preview of subscription content, log in to check access.
Buy single article
Instant access to the full article PDF.
Price includes VAT for USA
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
This is the net price. Taxes to be calculated in checkout.
Clinically significant prostate cancer
Multiparametric magnetic resonance imaging
Percentage of agreement
- P neg :
Proportion of negative agreement
- P pos :
Proportion of positive agreement
Transrectal ultrasound guided biopsy
Kasivisvanathan V, Rannikko AS, Borghi M et al (2018) MRI-targeted or standard biopsy for prostate-cancer diagnosis. N Engl J Med 378:1767–1777. https://doi.org/10.1056/NEJMoa1801993
Ahmed HU, El-Shater Bosaily A, Brown LC et al (2017) Diagnostic accuracy of multi-parametric MRI and TRUS biopsy in prostate cancer (PROMIS): a paired validating confirmatory study. Lancet. https://doi.org/10.1016/S0140-6736(16)32401-1
Mottet N, van den Bergh RCN, Briers E et al (2019) Guidelines on prostate cancer 2019. Eur Assoc Urol. Available via https://uroweb.org/guideline/prostate-cancer/
Drost FJH, Osses DF, Nieboer D et al (2019) Prostate MRI, with or without MRI-targeted biopsy, and systematic biopsy for detecting prostate cancer. Cochrane Database Syst Rev. https://doi.org/10.1002/14651858.CD012663.pub2
Barentsz JO, Richenberg J, Clements R et al (2012) ESUR prostate MR guidelines 2012. Eur Radiol. https://doi.org/10.1007/s00330-011-2377-y
Weinreb JC, Barentsz JO, Choyke PL et al (2016) PI-RADS Prostate Imaging - Reporting and Data System: 2015, version 2. Eur Urol. https://doi.org/10.1016/j.eururo.2015.08.052
Turkbey B, Rosenkrantz AB, Haider MA et al (2019) Prostate Imaging Reporting and Data System Version 2.1: 2019 Update of Prostate Imaging Reporting and Data System version 2. Eur Urol. https://doi.org/10.1016/j.eururo.2019.02.033
Rosenkrantz AB, Ginocchio LA, Cornfeld D et al (2016) Interobserver reproducibility of the PI-RADS version 2 lexicon: a multicenter study of six experienced prostate radiologists. Radiology. https://doi.org/10.1148/radiol.2016152542
Schimmöller L, Quentin M, Arsov C et al (2013) Inter-reader agreement of the ESUR score for prostate MRI using in-bore MRI-guided biopsies as the reference standard. Eur Radiol. https://doi.org/10.1007/s00330-013-2922-y
Muller BG, Shih JH, Sankineni S et al (2015) Prostate cancer: interobserver agreement and accuracy with the revised Prostate Imaging Reporting and Data System at multiparametric MR imaging. Radiology. https://doi.org/10.1148/radiol.2015142818
Sonn GA, Fan RE, Ghanouni P et al (2018) Prostate magnetic resonance imaging interpretation varies substantially across radiologists. Eur Urol Focus. https://doi.org/10.1016/j.euf.2017.11.010
Pickersgill NA, Vetter JM, Andriole GL et al (2018) Accuracy and variability of prostate multiparametric magnetic resonance imaging interpretation using the prostate imaging reporting and data system: a blinded comparison of radiologists. Eur Urol Focus. https://doi.org/10.1016/j.euf.2018.10.008
Smith CP, Harmon SA, Barrett T et al (2019) Intra- and interreader reproducibility of PI-RADSv2: a multireader study. J Magn Reson Imaging. https://doi.org/10.1002/jmri.26555
Girometti R, Giannarini G, Greco F et al (2019) Interreader agreement of PI-RADS v. 2 in assessing prostate cancer with multiparametric MRI: a study using whole-mount histology as the standard of reference. J Magn Reson Imaging 49:546–555. https://doi.org/10.1002/jmri.26220
Feinstein AR, Cicchetti DV (1990) High agreement but low kappa: I. the problems of two paradoxes. J Clin Epidemiol. https://doi.org/10.1016/0895-4356(90)90158-L
Cicchetti DV, Feinstein AR (1990) High agreement but low kappa: II. Resolving the paradoxes. J Clin Epidemiol. https://doi.org/10.1016/0895-4356(90)90159-M
Shih JH, Greer MD, Turkbey B (2018) The problems with the kappa statistic as a metric of interobserver agreement on lesion detection using a third-reader approach when locations are not prespecified. Acad Radiol. https://doi.org/10.1016/j.acra.2018.01.030
Hötker AM, Dappa E, Mazaheri Y et al (2019) The influence of background signal intensity changes on cancer detection in prostate MRI. Am J Roentgenol 212:823–829. https://doi.org/10.2214/AJR.18.20295
Stabile A, Dell’Oglio P, De Cobelli F et al (2018) Association between Prostate Imaging Reporting and Data System (PI-RADS) score for the index lesion and multifocal, clinically significant prostate cancer. Eur Urol Oncol 1:29–36. https://doi.org/10.1016/j.euo.2018.01.002
Dell’Oglio P, Stabile A, Soligo M et al (2019) There is no way to avoid systematic prostate biopsies in addition to multiparametric magnetic resonance imaging targeted biopsies. Eur Urol Oncol 3:112–118. https://doi.org/10.1016/j.euo.2019.03.002
Conger AJ (1980) Integration and generalization of kappas for multiple raters. Psychol Bull. https://doi.org/10.1037/0033-2909.88.2.322
Gwet KL (2008) Computing inter-rater reliability and its variance in the presence of high agreement. Br J Math Stat Psychol 61:29–48. https://doi.org/10.1348/000711006X126600
Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics. https://doi.org/10.2307/2529310
Rais-Bahrami S, Siddiqui MM, Vourganti S et al (2015) Diagnostic value of biparametric magnetic resonance imaging (MRI) as an adjunct to prostate-specific antigen (PSA)-based detection of prostate cancer in men without prior biopsies. BJU Int 115:381–388. https://doi.org/10.1111/bju.12639
Roobol MJ, Verbeek JFM, van der Kwast T, Kümmerlin IP, Kweldam CF, van Leenders GJLH (2017) Improving the Rotterdam European randomized study of screening for prostate cancer risk calculator for initial prostate biopsy by incorporating the 2014 International Society of Urological Pathology Gleason Grading and Cribriform growth. Eur Urol. https://doi.org/10.1016/j.eururo.2017.01.033
Greer MD, Brown AM, Shih JH et al (2017) Accuracy and agreement of PIRADSv2 for prostate cancer mpMRI: a multireader study. J Magn Reson Imaging 45:579–585. https://doi.org/10.1002/jmri.25372
Greer MD, Shih JH, Lay N et al (2019) Interreader variability of prostate imaging reporting and data system version 2 in detecting and assessing prostate cancer lesions at prostate MRI. AJR Am J Roentgenol 212:1197–1205. https://doi.org/10.2214/AJR.18.20536
Rouvière O, Puech P, Renard-Penna R et al (2019) Use of prostate systematic and targeted biopsy on the basis of multiparametric MRI in biopsy-naive patients (MRI-FIRST): a prospective, multicentre, paired diagnostic study. Lancet Oncol 20:100–109. https://doi.org/10.1016/S1470-2045(18)30569-2
Luzzago S, Petralia G, Musi G et al (2019) Multiparametric magnetic resonance imaging second opinion may reduce the number of unnecessary prostate biopsies: time to improve radiologists’ training program? Clin Genitourin Cancer 17:88–96. https://doi.org/10.1016/j.clgc.2018.10.006
This work has not received any funding.
The scientific guarantor of this publication is Francesco De Cobelli, M.D.
Conflict of interest
The authors of this manuscript declare no relationships with any companies whose products or services may be related to the subject matter of the article.
Statistics and biometry
Paola Maria Vittoria Rancoita kindly provided statistical advice for this manuscript, and is one of the authors.
Written informed consent was obtained from all subjects (patients) in this study.
Institutional Review Board approval was obtained.
Study subjects or cohorts overlap
None of the study subjects or cohorts has been previously reported.
• Performed at one institution
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
About this article
Cite this article
Brembilla, G., Dell’Oglio, P., Stabile, A. et al. Interreader variability in prostate MRI reporting using Prostate Imaging Reporting and Data System version 2.1. Eur Radiol (2020). https://doi.org/10.1007/s00330-019-06654-2
- Magnetic resonance imaging
- Prostate cancer
- Inter-observer variability