Journal of Digital Imaging

, Volume 32, Issue 6, pp 1089–1096 | Cite as

Variabilities in Reference Standard by Radiologists and Performance Assessment in Detection of Pulmonary Embolism in CT Pulmonary Angiography

  • Chuan ZhouEmail author
  • Heang-Ping Chan
  • Aamer Chughtai
  • Smita Patel
  • Jean Kuriakose
  • Lubomir M. Hadjiiski
  • Jun Wei
  • Ella A. Kazerooni


Annotating lesion locations by radiologists’ manual marking is a key step to provide reference standard for the training and testing of a computer-aided detection system by supervised machine learning. Inter-reader variability is not uncommon in readings even by expert radiologists. This study evaluated the variability of the radiologist-identified pulmonary emboli (PEs) to demonstrate the importance of improving the reliability of the reference standard by a multi-step process for performance evaluation. In an initial reading of 40 CTPA PE cases, two experienced thoracic radiologists independently marked the PE locations. For markings from the two radiologists that did not agree, each radiologist re-read the cases independently to assess the discordant markings. Finally, for markings that still disagreed after the second reading, the two radiologists read together to reach a consensus. The variability of radiologists was evaluated by analyzing the agreement between two radiologists. For the 40 cases, 475 and 514 PEs were identified by radiologists R1 and R2 in the initial independent readings, respectively. For a total of 545 marks by the two radiologists, 81.5% (444/545) of the marks agreed but 101 marks in 36 cases differed. After consensus, 65 (64.4%) and 36 (35.6%) of the 101 marks were determined to be true PEs and false positives (FPs), respectively. Of these, 48 and 17 were false negatives (FNs) and 14 and 22 were FPs by R1 and R2, respectively. Our study demonstrated that there is substantial variability in reference standards provided by radiologists, which impacts the performance assessment of a lesion detection system. Combination of multiple radiologists’ readings and consensus is needed to improve the reliability of a reference standard.


Computer-aided detection Pulmonary embolism Computed tomographic pulmonary angiography (CTPA) Reader variability Reference standard 



This work is supported by NIH grant R01-HL092044


  1. 1.
    Zhou C et al.: Robustness evaluation of a computer-aided detection system for pulmonary embolism (PE) in CTPA using independent test set from multiple institutions. SPIE Medical Imaging 9414:9414081–9414087, 2015Google Scholar
  2. 2.
    Zhou C et al.: Automated detection of pulmonary embolism (PE) in computed tomographic pulmonary angiographic (CTPA) images: Multiscale hierachical expectation-maximization segmentation of vessels and PEs. In: Proc. SPIE, Vol. 6514, 2007, pp. 2F1–2F8Google Scholar
  3. 3.
    Rathbun S, Raskob G, Whitsett T: Sensitivity and specificity of helical computed tomography in the diagnosis of pulmonary embolism: A systematic review. Annals of Internal Medicine 132:227–232, 2000PubMedGoogle Scholar
  4. 4.
    Cross J, Kemp P, Walsh C, Flower C, Dixon A: A randomized trial of spiral CT and ventilation perfusion scintigraphy for the diagnosis of pulmonary embolism. Clinical Radiology 53:177–182, 1998PubMedGoogle Scholar
  5. 5.
    Garg K, Welsh C, Feyerabend A: Pulmonary embolism: Diagnosis with spiral CT and ventilation-perfusion scanning-correlation with pulmonary angiographic results or clinical outcome. Radiology 208:201–208, 1998PubMedGoogle Scholar
  6. 6.
    Rossum A v, Pattynama P, Mallens W, Herman J: Can helical CT replace scintigraphy in the diagnostic process in suspected pulmonary embolism? A retrolective-prolective cohort study focusing on total diagnostic yield. Eur. Radiology 8:90–96, 1998Google Scholar
  7. 7.
    Drucker E, Rivitz S, Shepard J: Acute pulmonary embolism: Assessment of helical CT for diagnosis. Radiology 209:235–241, 1998PubMedGoogle Scholar
  8. 8.
    Ferretti G, Bosson J, Buffaz P: Acute pulmonary embolism: Role of helical CT in 164 patients with intermediate probability at ventilation-perfusion scintigraphy and normal results at duplex US of the legs. Radiology 205:453–458, 1997PubMedGoogle Scholar
  9. 9.
    Mayo J, Remy-Jardin M, Muller N: Pulmonary embolism: Prospective comparison of spiral CT with ventilation perfusion scintigraphy. Radiology 205:447–452, 1997PubMedGoogle Scholar
  10. 10.
    Russi T, Libby D, Henschke C: Clinical utility of computed tomography in the diagnosis of pulmonary embolism. Clinical Imaging 21:175–182, 1997PubMedGoogle Scholar
  11. 11.
    Rossum A v, Pattynama P, Ton E: Pulmonary embolism: Validation of spiral CT angiography in 149 patients. Radiology 201(2):467–470, 1996PubMedGoogle Scholar
  12. 12.
    Remy-Jardin M, Remy J, O C, Petyt L, Wannebroucq J, Beregi J: Diagnosis of central pulmonary embolism with helical CT: Role of two-dimensional multiplanar reformations. AJR Am J Roentgenol. 165:1131–1138, 1995PubMedGoogle Scholar
  13. 13.
    Blachere H, Latrabe V, Montaudon M: Pulmonary embolism revealed on helical CT angiography: Comparison with ventilation-perfusion radionuclide lung scanning. Am J Roentgenology 174:1041–1047, 2000Google Scholar
  14. 14.
    Remy-Jardin M, Remy J: Spiral CT angiography of the pulmonary circulation. Radiology 212:615–636, 1999PubMedGoogle Scholar
  15. 15.
    Goodman L, Curtin J, Mewissen M: Detection of pulmonary embolism in patients with unsolved clinical and scintigraphic diagnosis: Helical CT versus angiography. Am J Roentgenology 164(6):1369–1374, 1995Google Scholar
  16. 16.
    Goodman L: CT diagnosis of pulmonary embolism and deep venous thrombosis. Radiographics 20:1201–1205, 2000PubMedGoogle Scholar
  17. 17.
    Qanadli SD, Hajjam ME, Mesurolle B, Barré O, Bruckert F, Joseph T, Mignon F, Vieillard-Baron A, Dubourg O, Lacombe P: Pulmonary embolism detection: Prospective evaluation of dual-section helical CT versus selective pulmonary arteriography in 157 patients. Radiology 217:447–455, 2000PubMedGoogle Scholar
  18. 18.
    Stein PD, Fowler SE, Goodman LR, Gottschalk A, Hales CA, Hull RD, Leeper KV Jr, Popovich J Jr, Quinn DA, Sos TA, Sostman HD, Tapson VF, Wakefield TW, Weg JG, Woodard PK, PIOPED II Investigators: Multidetector computed tomography for acute pulmonary embolism. New England Journal of Medicine 354(22):2317–2327, 2006PubMedGoogle Scholar
  19. 19.
    Goodman LR: Small pulmonary emboli: What do we know? Radiology 234:654–658, 2005PubMedGoogle Scholar
  20. 20.
    Remy-Jardin M, Remy J, Wattinne L, Giraud F: Central pulmonary thromboembolism: Diagnosis with spiral volumetric CT with the single-breath-hold technique-comparison with pulmonary angiography. Radiology 185:381–387, 1992PubMedGoogle Scholar
  21. 21.
    Patel S, Kazerooni E, Cascade P: Pulmonary embolism: Optimization of small pulmonary artery visualization at multi-detector row CT. Radiology 227(2):455–460, 2003PubMedGoogle Scholar
  22. 22.
    Schoepf UJ, Holzknecht N, Helmberger TK, Crispin A, Hong C, Becker CR, Reiser MF: Subsegmental pulmonary emboli: Improved detection with thin-collimation multi-detector row spiral CT. Radiology 222:483–490, 2002PubMedGoogle Scholar
  23. 23.
    Costantino G, Norsa AH, Amadori R, Ippolito S, Resta F, Bianco R, Casazza G, Biagiotti S, Rusconi AM, Montano N: "Interobserver agreement in the interpretation of computed tomography in acute pulmonary embolism," (in eng). Am J Emerg Med 27(9):1109–1111, Nov 2009PubMedGoogle Scholar
  24. 24.
    Courtney DM, Miller C, Smithline H, Klekowski N, Hogg M, Kline JA: "Prospective multicenter assessment of interobserver agreement for radiologist interpretation of multidetector computerized tomographic angiography for pulmonary embolism," (in eng). J Thromb Haemost 8(3):533–539, Mar 2010PubMedGoogle Scholar
  25. 25.
    Diffin D, Leyendecker J, Johnson S, Zucker R, Grebe P: Effect of anatomic distribution of pulmonary emboli on interobserver agreement in the interpretation of pulmonary angiography. Am J Roentgenology 171:1085–1089, 1998Google Scholar
  26. 26.
    Le Gal G, Righini R, Parent F, Strijen MV, Couturaud F: Diagnosis and management of subsegmental pulmonary embolism. Journal of Thrombosis and Haemostasis 4:724–731, 2005Google Scholar
  27. 27.
    Wu AS, Pezzullo JA, Cronan HJ, Hou DD, Mayo-Smith WW: CT pulmonary angiography: Quantification of pulmonary embolus as a predictor of patient outcome - initial experience. Radiology 230(3):831–835, 2004PubMedGoogle Scholar
  28. 28.
    Bankier A, Janata K, Fleischmann D et al.: severity assessment of acute pulmonary embolism with spiral CT: Evaluation of two modified angiographic scores and comparison with clinical data. J Thorac Imaging 12:150–158, 1997PubMedGoogle Scholar
  29. 29.
    Qanadli SD, el Hajjam M, Vieillard-Baron A, Joseph T, Mesurolle B, Oliva VL, Barré O, Bruckert F, Dubourg O, Lacombe P: New CT index to quantify arterial obstruction in pulmonary embolism: Comparison with angiographic index and echocardiography. AJR Am J Roentgenol. 176:1415–1420, 2001PubMedGoogle Scholar
  30. 30.
    Mastora I, Remy-Jardin M, P M et al.: severity of acute pulmonary embolism: Evaluation of a new spiral CT angiographic score in correlation with echocardiographic data. Eur Radiology 13:29–35, 2003Google Scholar
  31. 31.
    Wood K, Visani L, De Rosa M: Major pulmonary embolism: Review of a pathophysiologic approach to the golden hour of hemodynamically signification pulmonary embolism. Chest 121:877–905, 2002PubMedGoogle Scholar
  32. 32.
    Araoz PA, Gotway MB, Trowbridge RL, Bailey RA, Auerbach AD et al.: helical CT pulmonary angiography predictors of in-hospital morbidity and mortality in patients with acute pulmonary embolism. J Thorac Imaging 18(4):207–216, 2003PubMedGoogle Scholar
  33. 33.
    Zhou C, Chan H-P: Detection of pulmonary embolism. In: Li Q, Nishikawa R Eds. Computer-aided detection and diagnosis in medical imaging. Boca Raton: CRC Press, 2015, pp. 261–278Google Scholar
  34. 34.
    Ko JP, Naidich DP: Computer-aided diagnosis and the evaluation of lung disease. J Thorac Imaging 19(3):136–155, 2004PubMedGoogle Scholar
  35. 35.
    Schoepf UJ, Costello P: CT angiography for diagnosis of pulmonary embolism: State of the art. Radiology 230:329–337, 2004PubMedGoogle Scholar
  36. 36.
    Chan HP, Hadjiiski LM, Zhou C, Sahiner B: Computer-aided diagnosis of lung Cancer and pulmonary embolism in computed tomography—A review. Academic Radiology 15(5):535–555, 2008PubMedPubMedCentralGoogle Scholar
  37. 37.
    Schoepf UJ, Schneider AC, Das M, Wood SA, Cheema JI, Costello P: Pulmonary embolism: Computer-aided detection at multidetector row spiral computed tomography. Journal of Thoracic Imaging 22(4):319–323, 2007PubMedGoogle Scholar
  38. 38.
    Maizlin ZV, Vos PM, Godoy MB, Cooperberg PL: Computer-aided detection of pulmonary embolism on CT angiography: Initial experience. Journal of Thoracic Imaging 22(4):324–329, 2007PubMedGoogle Scholar
  39. 39.
    Buhmann S, Herzog P, Liang J, Wolf M, Salganicoff M, Kirchhoff C, Reiser M, Becker CH: Clinical evaluation of a computer-aided diagnosis (CAD) prototype for the detection of pulmonary embolism. Academic Radiology 14(6):651–658, 2007PubMedGoogle Scholar
  40. 40.
    Engelke C, Schmidt S, Bakai A, Auer F, Marten K: Computer-assisted detection of pulmonary embolism: Performance evaluation in consensus with experienced and inexperienced chest radiologists. European Radiology 18(2):298–307, 2008PubMedGoogle Scholar
  41. 41.
    Zhou C, Chan HP, Sahiner B, Hadjiiski LM, Chughtai A, Patel S, Wei J, Cascade PN, Kazerooni EA: Computer-aided detection of pulmonary embolism in computed tomographic pulmonary angiography (CTPA): Performance evaluation with independent data sets. Medical Physics 36(8):3385–3396, 2009PubMedPubMedCentralGoogle Scholar
  42. 42.
    Adams R, Bischof L: Seeded region growing. IEEE Trans. on PAMI 16(6):641–647, 1994Google Scholar
  43. 43.
    Chang Y-L, Li X: Adaptive image region-growing. IEEE Transactions on Image Processing 3(6):868–872, 1994PubMedGoogle Scholar
  44. 44.
    Mehnert A: An improved seeded region growing algorithm. Pattern Recognition Letters 18:1065–1071, 1997Google Scholar
  45. 45.
    Zhou C, Chan HP, Sahiner B, Hadjiiski LM, Chughtai A, Patel S, Wei J, Ge J, Cascade PN, Kazerooni EA: Automatic multiscale enhancement and hierarchical segmentation of pulmonary vessels in CT pulmonary angiography (CTPA) images for CAD applications. Medical Physics 34(12):4567–4577, 2007PubMedPubMedCentralGoogle Scholar
  46. 46.
    Chartrand-Lefebvre C, Howarth N, Lucidarme O, Beigelman C, Cluzel P, Mourey-Gérosa I, Cadi M, Grenier P: Contrast-enhanced helical CT for pulmonary embolism detection: Inter- and intraobserver agreement among radiologists with variable experience. American Journal of Roentgenology 172:107–112, 1999PubMedGoogle Scholar
  47. 47.
    Domingo ML, Martí-Bonmatí L, Dosdá R, Pallardó Y: Interobserver agreement in the diagnosis of pulmonary embolism with helical CT. European Journal of Radiology 34(2):136–140, 2000PubMedGoogle Scholar
  48. 48.
    Dorfman DD, Berbaum KS, Metz CE: ROC rating analysis: Generalization to the population of readers and cases with the jackknife method. Investigative Radiology 27:723–731, 1992PubMedGoogle Scholar
  49. 49.
    Petrick N, Sahiner B, Armato, III SG, Bert A, Correale L, Delsanto S, Freedman MT, Fryd D, Gur D, Hadjiiski L, Huo Z, Jiang Y, Morra L, Paquerault S, Raykar V, Samuelson F, Summers RM, Tourassi G, Yoshida H, Zheng B, Zhou C, Chan HP: "Evaluation of computer-aided detection and diagnosis systems," (in eng). Med Phys 40(8):087001, Aug 2013PubMedPubMedCentralGoogle Scholar

Copyright information

© Society for Imaging Informatics in Medicine 2019

Authors and Affiliations

  1. 1.Department of RadiologyUniversity of MichiganAnn ArborUSA

Personalised recommendations