Impact of acquisition and interpretation on total inter-observer variability in echocardiography: results from the quality assurance program of the STAAB cohort study
Variability related to image acquisition and interpretation is an important issue of echocardiography in clinical trials. Nevertheless, there is no broadly accepted standard method for quality assessment of echocardiography in clinical research reports. We present analyses based on the echocardiography quality-assurance program of the ongoing STAAB cohort study (characteristics and course of heart failure stages A–B and determinants of progression). In 43 healthy individuals (mean age 50 ± 14 years; 18 females), duplicate echocardiography scans were acquired and mutually interpreted by one of three trained sonographers and an EACVI certified physician, respectively. Acquisition (AcV), interpretation (InV), and inter-observer variability (IOV; i.e., variability between the acquisition-interpretation sequences of two different observers), were determined for selected M-mode, B-mode, and Doppler parameters. We calculated Bland–Altman upper 95% limits of absolute differences, implying that 95% of measurement differences were smaller/equal to the given value: e.g. LV end-diastolic volume (mL): 25.0, 25.0, 27.9; septal e′ velocity (cm/s): 3.03, 1.25, 3.58. Further, 90, 85, and 80% upper limits of absolute differences were determined for the respective parameters. Both, acquisition and interpretation, independently and sizably contributed to IOV. As such, separate assessment of AcV and InV is likely to aid in echocardiography training and quality-assurance. Our results further suggest to routinely determine IOV in clinical trials as a comprehensive measure of imaging quality. The derived 95, 90, 85, and 80% upper limits of absolute differences are suggested as reproducibility targets of future studies, thus contributing to the international efforts of standardization in quality-assurance.
KeywordsEchocardiography Quality Reproducibility Variability Observer Acquisition
Peak late- diastolic lengthening velocity
Peak early diastolic lengthening velocity
European Association of Cardiovascular Imaging
Intraclass correlation coefficient
Characteristics and course of heart failure stages A–B and determinants of progression cohort study
Limits of agreement
Left ventricular end-diastolic diameter
Left ventricular end-diastolic volume
Left ventricular end-systolic volume
Mitral annular plane systolic excursion
Peak systolic shortening velocity
Tricuspid annular plane systolic excursion
Tissue Doppler imaging
We thank Martina Bauer, lead sonographer of the Comprehensive Heart Failure Center (CHFC), for excellent echocardiography teaching, as well as Jasmin Simon, Martin Cramer and Frances Knobeloch, CHFC sonographers, for their valuable contribution to this study.
Compliance with ethical standards
Conflict of interest
CM reports a co-investigatorship in a trial sponsored by NOVARTIS, a travel grant from THERMO FISHER and a speaker honorarium from AMGEN. GG reports grants from University Göttingen within the FIND-AF-randomized trial (FIND-AF-randomized is supported by an unrestricted grant to the University Göttingen from Boehringer-Ingelheim), grants from University Göttingen within DZHK analysis projects (the DZHK is funded by a BMBF grant), personal fees from Charité - Universitätsmedizin Berlin within the TIM-HF-II trial for data safety and monitoring board membership (TIM-HF-II is supported by a BMBF grant to the Charité), grants from University Hospital Würzburg within the MOOD-HF trial for biometry and steering committee membership (MOOD-HF is supported by a BMBF grant and an unrestricted grant to the University Hospital Würzburg from Lundbeck), outside the submitted work. PUH reports research grants from the German Ministry of Research and Education, European Union, Charité, Berlin Chamber of Physicians, German Parkinson Society, University Hospital Würzburg, Robert-Koch-Institute, German Heart Foundation, Charité–Universitätsmedizin Berlin (within MonDAFIS; MonDAFIS is supported by an unrestricted research grant to the Charité from Bayer), University Göttingen (within FIND-AF-randomized; FIND-AF randomized is supported by an unrestricted research grant to the University Göttingen from Boehringer-Ingelheim), and University Hospital Heidelberg (within RASUNOA-prime; RASUNOA-prime is supported by an unrestricted research grant to the University Hospital Heidelberg from Bayer, BMS, Boehringer-Ingelheim, Daiichi Sankyo), outside submitted work. SS reports research grants from the German Ministry of Education and Research, European Union, University Hospital Würzburg; participation in Data Safety Monitoring and Event Adjudication Boards in trials sponsored by ROCHE and MEDTRONIC; principal investigator in trials (co-)sponsored by BOEHRINGER, NOVARTIS, BAYER, LUNDBECK; speaker honaria by BOEHRINGER, SERVIER, NOVARTIS, ASTRA-ZENECA, PFIZER, BAYER. MW declares that he has no conflict of interest. TT declares that she has no conflict of interest. MB declares that she has no conflict of interest.
The STAAB cohort study protocol and procedures received positive votes from the Ethics Committee of the Medical Faculty (vote 98/13) as well as from the data protection officer of the University of Würzburg (J-117.605-09/13). All procedures performed in this study were in accordance with the ethical standards of the institutional and national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
All participants provided written informed consent prior to any study examination.
- 1.Crowley AL, Yow E, Barnhart HX, Daubert MA, Bigelow R, Sullivan DC, Pencina M, Douglas PS (2016) Critical review of current approaches for echocardiographic reproducibility and reliability assessment in clinical research. J Am Soc Echocardiogr 29(12):1144–1154.e7. https://doi.org/10.1016/j.echo.2016.08.006 CrossRefPubMedGoogle Scholar
- 2.Pellikka PA, Douglas PS, Miller JG, Abraham TP, Baumann R, Buxton DB, Byrd BF 3rd, Chen P, Cook NL, Gardin JM, Hansen G, Houle HC, Husson S, Kaul S, Klein AL, Lang RM, Leong-Poi H, Lopez H, Mahmoud TM, Maslak S, McCulloch ML, Metz S, Nagueh SF, Pearlman AS, Pibarot P, Picard MH, Porter TR, Prater D, Rodriguez R, Sarano ME, Scherrer-Crosbie M, Shirali GS, Sinusas A, Slosky JJ, Sugeng L, Tatpati A, Villanueva FS, von Ramm OT, Weissman NJ, Zamani S (2013) American society of echocardiography cardiovascular technology and research summit: a roadmap for 2020. J Am Soc Echocardiogr 26(4):325–338. https://doi.org/10.1016/j.echo.2013.02.003 CrossRefPubMedGoogle Scholar
- 3.Kardys I, Deckers JW, Stricker BH, Vletter WB, Hofman A, Witteman J (2010) Distribution of echocardiographic parameters and their associations with cardiovascular risk factors in the Rotterdam Study. Eur J Epidemiol 25(7):481–490. https://doi.org/10.1007/s10654-010-9453-5 CrossRefPubMedPubMedCentralGoogle Scholar
- 4.Kou S, Caballero L, Dulgheru R, Voilliot D, De Sousa C, Kacharava G, Athanassopoulos GD, Barone D, Baroni M, Cardim N, Gomez De Diego JJ, Hagendorff A, Henri C, Hristova K, Lopez T, Magne J, De La Morena G, Popescu BA, Penicka M, Ozyigit T, Rodrigo Carbonero JD, Salustri A, Van De Veire N, Von Bardeleben RS, Vinereanu D, Voigt JU, Zamorano JL, Donal E, Lang RM, Badano LP, Lancellotti P (2014) Echocardiographic reference ranges for normal cardiac chamber size: results from the NORRE study. Eur Heart J Cardiovasc Imaging 15(6):680–690. https://doi.org/10.1093/ehjci/jet284 CrossRefPubMedPubMedCentralGoogle Scholar
- 5.Jenkins C, Bricknell K, Hanekom L, Marwick TH (2004) Reproducibility and accuracy of echocardiographic measurements of left ventricular parameters using real-time three-dimensional echocardiography. J Am Coll Cardiol 44(4):878–886. https://doi.org/10.1016/j.jacc.2004.05.050 CrossRefPubMedGoogle Scholar
- 6.Wagner M, Tiffe T, Morbach C, Gelbrich G, Stork S, Heuschmann PU, Consortium S (2016) Characteristics and course of heart failure stages A-B and determinants of progression—design and rationale of the STAAB cohort study. Eur J Prev Cardiol 24(5):468–479. https://doi.org/10.1177/2047487316680693 CrossRefPubMedGoogle Scholar
- 7.Lang RM, Badano LP, Mor-Avi V, Afilalo J, Armstrong A, Ernande L, Flachskampf FA, Foster E, Goldstein SA, Kuznetsova T, Lancellotti P, Muraru D, Picard MH, Rietzschel ER, Rudski L, Spencer KT, Tsang W, Voigt JU (2015) Recommendations for cardiac chamber quantification by echocardiography in adults: an update from the American Society of Echocardiography and the European Association of Cardiovascular Imaging. Eur Heart J Cardiovasc Imaging 16(3):233–270. https://doi.org/10.1093/ehjci/jev014 CrossRefPubMedGoogle Scholar
- 10.Stritzke J, Linsel-Nitschke P, Markus MR, Mayer B, Lieb W, Luchner A, Doring A, Koenig W, Keil U, Hense HW, Schunkert H, Investigators MK (2009) Association between degenerative aortic valve disease and long-term exposure to cardiovascular risk factors: results of the longitudinal population-based KORA/MONICA survey. Eur Heart J 30(16):2044–2053. https://doi.org/10.1093/eurheartj/ehp287 CrossRefPubMedGoogle Scholar
- 11.Colan SD, Shirali G, Margossian R, Gallagher D, Altmann K, Canter C, Chen S, Golding F, Radojewski E, Camitta M, Carboni M, Rychik J, Stylianou M, Tani LY, Selamet Tierney ES, Wang Y, Sleeper LA, Pediatric Heart Network I (2012) The ventricular volume variability study of the Pediatric Heart Network: study design and impact of beat averaging and variable type on the reproducibility of echocardiographic measurements in children with chronic dilated cardiomyopathy. J Am Soc Echocardiogr 25(8):842–854.e6. https://doi.org/10.1016/j.echo.2012.05.004 CrossRefPubMedPubMedCentralGoogle Scholar
- 13.Barnhart HX, Yow E, Crowley AL, Daubert MA, Rabineau D, Bigelow R, Pencina M, Douglas PS (2014) Choice of agreement indices for assessing and improving measurement reproducibility in a core laboratory setting. Stat Methods Med Res 25(6):2939–2958. https://doi.org/10.1177/0962280214534651 CrossRefPubMedGoogle Scholar