European Radiology

, Volume 28, Issue 10, pp 4254–4264 | Cite as

Interobserver and intermodality agreement of standardized algorithms for non-invasive diagnosis of hepatocellular carcinoma in high-risk patients: CEUS-LI-RADS versus MRI-LI-RADS

  • Barbara Schellhaas
  • Matthias Hammon
  • Deike Strobel
  • Lukas Pfeifer
  • Christian Kielisch
  • Ruediger S. Goertz
  • Alexander Cavallaro
  • Rolf Janka
  • Markus F. Neurath
  • Michael Uder
  • Hannes SeussEmail author



We compared the interobserver agreement for the recently introduced contrast-enhanced ultrasound (CEUS)-based algorithm CEUS-LI-RADS (Liver Imaging Reporting and Data System) versus the well-established magnetic resonance imaging (MRI)-LI-RADS for non-invasive diagnosis of hepatocellular carcinoma (HCC) in high-risk patients.


Focal liver lesions in 50 high-risk patients (mean age 66.2 ± 11.8 years; 39 male) were assessed retrospectively with CEUS and MRI. Two independent observers reviewed CEUS and MRI examinations, separately, classifying observations according to CEUS-LI-RADSv.2016 and MRI-LI-RADSv.2014. Interobserver agreement was assessed with Cohen’s kappa.


Forty-three lesions were HCCs; two were intrahepatic cholangiocarcinomas; five were benign lesions. Arterial phase hyperenhancement was perceived less frequently with CEUS than with MRI (37/50 / 38/50 lesions = 74%/78% [CEUS; observer 1/observer 2] versus 46/50 / 44/50 lesions = 92%/88% [MRI; observer 1/observer 2]). Washout appearance was observed in 34/50 / 20/50 lesions = 68%/40% with CEUS and 31/50 / 31/50 lesions = 62%/62%) with MRI. Interobserver agreement was moderate for arterial hyperenhancement (ĸ = 0.511/0.565 [CEUS/MRI]) and “washout” (ĸ = 0.490/0.582 [CEUS/MRI]), fair for CEUS-LI-RADS category (ĸ = 0.309) and substantial for MRI-LI-RADS category (ĸ = 0.609). Intermodality agreement was fair for arterial hyperenhancement (ĸ = 0.329), slight to fair for “washout” (ĸ = 0.202) and LI-RADS category (ĸ = 0.218)


Interobserver agreement is substantial for MRI-LI-RADS and only fair for CEUS-LI-RADS. This is mostly because interobserver agreement in the perception of washout appearance is better in MRI than in CEUS. Further refinement of the LI-RADS algorithms and increasing education and practice may be necessary to improve the concordance between CEUS and MRI for the final LI-RADS categorization.

Key Points

• CEUS-LI-RADS and MRI-LIRADS enable standardized non-invasive diagnosis of HCC in high-risk patients.

• With CEUS, interobserver agreement is better for arterial hyperenhancement than for “washout”.

• Interobserver agreement for major features is moderate for both CEUS and MRI.

• Interobserver agreement for LI-RADS category is substantial for MRI, and fair for CEUS.

• Interobserver-agreement for CEUS-LI-RADS will presumably improve with ongoing use of the algorithm.


Carcinoma, hepatocellular Magnetic resonance imaging Ultrasonography Diagnostic techniques and procedures Liver cirrhosis 



American College of Radiology




Body mass index


Contrast-enhanced computed tomography


Contrast-enhanced magnetic resonance imaging


Contrast-enhanced Ultrasound




Computed tomography


Diffusion-weighted image


Fat saturation


Gradient echo


Half-Fourier acquisition single-shot turbo spin echo


Hepatocellular carcinoma


Intrahepatic cholangiocellular carcinoma


Liver Imaging Reporting and Data System


LI-RADS category


Magnetic resonance imaging








Echo time


Repetition time




Turbo spin echo


Volumetric-interpolated breath-hold examination



We thank the editors of European Radiology and those who reviewed this article.


The authors state that this work has not received any funding.

Compliance with ethical standards


The scientific guarantor of this publication is Dr. Hannes Seuss.

Conflict of interest

The authors of this manuscript declare no relationships with any companies whose products or services may be related to the subject matter of the article.

Statistics and biometry

One of the authors has significant statistical expertise.

Informed consent

Written informed consent was waived by the institutional review board.

Ethical approval

Institutional review board approval was obtained.


• retrospective

• diagnostic study

• performed at one institution

Supplementary material

330_2018_5379_MOESM1_ESM.docx (164 kb)
ESM 1 (DOCX 164 kb)


  1. 1.
    Westwood M, Joore M, Grutters J et al (2013) Contrast-enhanced ultrasound using SonoVue(R) (sulphur hexafluoride microbubbles) compared with contrast-enhanced computed tomography and contrast-enhanced magnetic resonance imaging for the characterisation of focal liver lesions and detection of liver metastases: a systematic review and cost-effectiveness analysis. Health Technol Assess 17:1–243CrossRefGoogle Scholar
  2. 2.
    Claudon M, Dietrich CF, Choi BI et al (2013) Guidelines and good clinical practice recommendations for contrast enhanced ultrasound (CEUS) in the liver - update 2012: a WFUMB-EFSUMB initiative in cooperation with representatives of AFSUMB, AIUM, ASUM, FLAUS and ICUS. Ultrasound Med Biol 39:187–210CrossRefGoogle Scholar
  3. 3.
    Bota S, Piscaglia F, Marinelli S, Pecorelli A, Terzi E, Bolondi L (2012) Comparison of international guidelines for noninvasive diagnosis of hepatocellular carcinoma. Liver Cancer 1:190–200CrossRefGoogle Scholar
  4. 4.
    Bruix J, Sherman M (2011) Management of hepatocellular carcinoma: an update. Hepatology 53:1020–1022CrossRefGoogle Scholar
  5. 5.
    Kokudo N, Hasegawa K, Akahane M et al (2015) Evidence-based clinical practice guidelines for hepatocellular carcinoma: The Japan Society of Hepatology 2013 update (3rd JSH-HCC guidelines). Hepatol Res 45. CrossRefGoogle Scholar
  6. 6.
    Friedrich-Rust M, Klopffleisch T, Nierhoff J et al (2013) Contrast-enhanced ultrasound for the differentiation of benign and malignant focal liver lesions: a meta-analysis. Liver Int 33:739–755CrossRefGoogle Scholar
  7. 7.
    Shah S, Shukla A, Paunipagar B (2014) Radiological features of hepatocellular carcinoma. J Clin Exp Hepatol 4:S63–S66CrossRefGoogle Scholar
  8. 8.
    Leoni S, Piscaglia F, Granito A et al (2013) Characterization of primary and recurrent nodules in liver cirrhosis using contrast-enhanced ultrasound: which vascular criteria should be adopted? Ultraschall Med 34:280–287CrossRefGoogle Scholar
  9. 9.
    Mitchell DG, Bruix J, Sherman M, Sirlin CB (2015) LI-RADS (Liver Imaging Reporting and Data System): summary, discussion, and consensus of the LI-RADS Management Working Group and future directions. Hepatology 61:1056–1065CrossRefGoogle Scholar
  10. 10.
    Elsayes KM, Kielar AZ, Agrons MM et al (2017) Liver Imaging Reporting and Data System: an expert consensus statement. J Hepatocell Carcinoma 4:29–39CrossRefGoogle Scholar
  11. 11.
    AWMF-Leitlinie S3 - Leitlinie Deutschland: “Hepatozelluläres Karzinom”, Registernummer 032/053OL Stand: 01.05.2013 , gültig bis 30.04.2018; accessed: May 19 2017. Available via Accessed May 19 2017
  12. 12.
    Claudon M, Dietrich CF, Choi BI et al (2013) Guidelines and good clinical practice recommendations for contrast enhanced ultrasound (CEUS) in the liver–update 2012: a WFUMB-EFSUMB initiative in cooperation with representatives of AFSUMB, AIUM, ASUM, FLAUS and ICUS. Ultraschall Med 34:11–29CrossRefGoogle Scholar
  13. 13.
    Strobel D, Bernatik T, Blank W et al (2011) Diagnostic accuracy of CEUS in the differential diagnosis of small (</ = 20 mm) and subcentimetric (</ = 10 mm) focal liver lesions in comparison with histology. Results of the DEGUM multicenter trial. Ultraschall Med 32:593–597CrossRefGoogle Scholar
  14. 14.
    Strobel D, Seitz K, Blank W et al (2009) Tumor-specific vascularization pattern of liver metastasis, hepatocellular carcinoma, hemangioma and focal nodular hyperplasia in the differential diagnosis of 1,349 liver lesions in contrast-enhanced ultrasound (CEUS). Ultraschall Med 30:376–382CrossRefGoogle Scholar
  15. 15.
    Seitz K, Greis C, Schuler A et al (2011) Frequency of tumor entities among liver tumors of unclear etiology initially detected by sonography in the noncirrhotic or cirrhotic livers of 1349 patients. Results of the DEGUM multicenter study. Ultraschall Med 32:598–603CrossRefGoogle Scholar
  16. 16.
    Darnell A, Forner A, Rimola J et al (2015) Liver Imaging Reporting and Data System with MR imaging: evaluation in nodules 20 mm or smaller detected in cirrhosis at screening US. Radiology 275:698–707CrossRefGoogle Scholar
  17. 17.
    Seitz K, Piscaglia F (2013) Ultrasound: the only "one stop shop" for modern management of liver disease. Ultraschall Med 34:500–503CrossRefGoogle Scholar
  18. 18.
    Wildner D, Bernatik T, Greis C, Seitz K, Neurath MF, Strobel D (2015) CEUS in hepatocellular carcinoma and intrahepatic cholangiocellular carcinoma in 320 patients - early or late washout matters: a subanalysis of the DEGUM multicenter trial. Ultraschall Med 36:132–139CrossRefGoogle Scholar
  19. 19.
    Wildner D, Pfeifer L, Goertz RS et al (2014) Dynamic contrast-enhanced ultrasound (DCE-US) for the characterization of hepatocellular carcinoma and cholangiocellular carcinoma. Ultraschall Med 35:522–527CrossRefGoogle Scholar
  20. 20.
    McEvoy SH, McCarthy CJ, Lavelle LP et al (2013) Hepatocellular carcinoma: illustrated guide to systematic radiologic diagnosis and staging according to guidelines of the American Association for the Study of Liver Diseases. Radiographics 33:1653–1668CrossRefGoogle Scholar
  21. 21.
    Seitz K, Bernatik T, Strobel D et al (2010) Contrast-enhanced ultrasound (CEUS) for the characterization of focal liver lesions in clinical practice (DEGUM Multicenter Trial): CEUS vs. MRI–a prospective comparison in 269 patients. Ultraschall Med 31:492–499CrossRefGoogle Scholar
  22. 22.
    Schellhaas B, Wildner D, Pfeifer L et al (2016) LI-RADS-CEUS - proposal for a contrast-enhanced ultrasound algorithm for the diagnosis of hepatocellular carcinoma in high-risk populations. Ultraschall Med 37:627–634CrossRefGoogle Scholar
  23. 23.
    Karlas T (2017) LI-RADS-CEUS for the classification of HCC risk in liver lesions. Z Gastroenterol 55:507–508CrossRefGoogle Scholar
  24. 24.
    Piscaglia F, Wilson SR, Lyshchik A et al (2017) American College of Radiology Contrast Enhanced Ultrasound Liver Imaging Reporting and Data System (CEUS LI-RADS) for the diagnosis of hepatocellular carcinoma: a pictorial essay. Ultraschall Med. CrossRefGoogle Scholar
  25. 25.
    ACR American College of Radiology: Liver Imaging Reporting and Data System Accessed 12 Sep 2017
  26. 26.
    Zhang YD, Zhu FP, Xu X et al (2016) Liver Imaging Reporting and Data System: substantial discordance between CT and MR for imaging classification of hepatic nodules. Acad Radiol 23:344–352CrossRefGoogle Scholar
  27. 27.
    Corwin MT, Fananapazir G, Jin M, Lamba R, Bashir MR (2016) Differences in liver imaging and reporting data system categorization between MRI and CT. AJR Am J Roentgenol 206:307–312CrossRefGoogle Scholar
  28. 28.
    Chernyak V, Flusberg M, Law A, Kobi M, Paroder V, Rozenblit AM (2017) Liver Imaging Reporting and Data System: discordance between computed tomography and gadoxetate-enhanced magnetic resonance imaging for detection of hepatocellular carcinoma major features. J Comput Assist Tomogr.
  29. 29.
    Hope TA, Aslam R, Weinstein S et al (2017) Change in Liver Imaging Reporting and Data System characterization of focal liver lesions using gadoxetate disodium magnetic resonance imaging compared with contrast-enhanced computed tomography. J Comput Assist Tomogr 41:376–381CrossRefGoogle Scholar
  30. 30.
    Barth BK, Donati OF, Fischer MA et al (2016) Reliability, validity, and reader acceptance of LI-RADS-An in-depth analysis. Acad Radiol 23:1145–1153CrossRefGoogle Scholar
  31. 31.
    Bashir MR, Huang R, Mayes N et al (2015) Concordance of hypervascular liver nodule characterization between the organ procurement and transplant network and liver imaging reporting and data system classifications. J Magn Reson Imaging 42:305–314CrossRefGoogle Scholar
  32. 32.
    Davenport MS, Khalatbari S, Liu PS et al (2014) Repeatability of diagnostic features and scoring systems for hepatocellular carcinoma by using MR imaging. Radiology 272:132–142CrossRefGoogle Scholar
  33. 33.
    Giorgio A, Montesarchio L, Gatti P et al (2016) Contrast-enhanced ultrasound: a simple and effective tool in defining a rapid diagnostic work-up for small nodules detected in cirrhotic patients during surveillance. J Gastrointestin Liver Dis 25:205–211PubMedGoogle Scholar
  34. 34.
    Quaia E, Alaimo V, Baratella E et al (2010) Effect of observer experience in the differentiation between benign and malignant liver tumors after ultrasound contrast agent injection. J Ultrasound Med 29:25–36CrossRefGoogle Scholar
  35. 35.
    Berg WA, D'Orsi CJ, Jackson VP et al (2002) Does training in the Breast Imaging Reporting and Data System (BI-RADS) improve biopsy recommendations or feature analysis agreement with experienced breast imagers at mammography? Radiology 224:871–880CrossRefGoogle Scholar
  36. 36.
    Zhang YD, Zhu FP, Xu X et al (2016) Classifying CT/MR findings in patients with suspicion of hepatocellular carcinoma: comparison of liver imaging reporting and data system and criteria-free Likert scale reporting models. J Magn Reson Imaging 43:373–383CrossRefGoogle Scholar
  37. 37.
    Ehman EC, Behr SC, Umetsu SE et al (2016) Rate of observation and inter-observer agreement for LI-RADS major features at CT and MRI in 184 pathology proven hepatocellular carcinomas. Abdom Radiol (NY) 41:963–969CrossRefGoogle Scholar
  38. 38.
    McHugh ML (2012) Interrater reliability: the kappa statistic. Biochem Med (Zagreb) 22:276–282CrossRefGoogle Scholar

Copyright information

© European Society of Radiology 2018

Authors and Affiliations

  • Barbara Schellhaas
    • 1
  • Matthias Hammon
    • 2
  • Deike Strobel
    • 1
  • Lukas Pfeifer
    • 1
  • Christian Kielisch
    • 1
  • Ruediger S. Goertz
    • 1
  • Alexander Cavallaro
    • 2
  • Rolf Janka
    • 2
  • Markus F. Neurath
    • 1
  • Michael Uder
    • 2
  • Hannes Seuss
    • 2
    Email author
  1. 1.Department of Internal Medicine 1Friedrich-Alexander-Universität Erlangen-Nürnberg, University Hospital ErlangenErlangenGermany
  2. 2.Department of RadiologyFriedrich-Alexander-Universität Erlangen-Nürnberg, University Hospital ErlangenErlangenGermany

Personalised recommendations