Advertisement

The Accuracy of Acquiring Heart Rate Variability from Portable Devices: A Systematic Review and Meta-Analysis

  • Ward C. DobbsEmail author
  • Michael V. Fedewa
  • Hayley V. MacDonald
  • Clifton J. Holmes
  • Zackary S. Cicone
  • Daniel J. Plews
  • Michael R. Esco
Systematic Review

Abstract

Background

Advancements in wearable technology have provided practitioners and researchers with the ability to conveniently measure various health and/or fitness indices. Specifically, portable devices have been devised for convenient recordings of heart rate variability (HRV). Yet, their accuracies remain questionable.

Objective

The aim was to quantify the accuracy of portable devices compared to electrocardiography (ECG) for measuring a multitude of HRV metrics and to identify potential moderators of this effect.

Methods

This meta-analysis was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement. Articles published before July 29, 2017 were located via four electronic databases using a combination of the terms related to HRV and validity. Separate effect sizes (ESs), defined as the absolute standardized difference between the HRV value recorded using the portable device compared to ECG, were generated for each HRV metric (ten metrics analyzed in total). A multivariate, multi-level model, incorporating random-effects assumptions, was utilized to quantify the mean ES and 95% confidence interval (CI) and explore potential moderators.

Results

Twenty-three studies yielded 301 effects and revealed that HRV measurements acquired from portable devices differed from those obtained from ECG (ES = 0.23, 95% CI 0.05–0.42), although this effect was small and highly heterogeneous (I2 = 78.6%, 95% CI 76.2–80.7). Moderator analysis revealed that HRV metric (p <0.001), position (p = 0.033), and biological sex (β = 0.45, 95% CI 0.30–0.61; p <0.001), but not portable device, modulated the degree of absolute error. Within metric, absolute error was significantly higher when expressed as standard deviation of all normal–normal (R–R) intervals (SDNN) (ES = 0.44) compared to any other metric, but was no longer significantly different after a sensitivity analysis removed outliers. Likewise, the error associated with the tilt/recovery position was significantly higher than any other position and remained significantly different without outliers in the model.

Conclusions

Our results suggest that HRV measurements acquired using portable devices demonstrate a small amount of absolute error when compared to ECG. However, this small error is acceptable when considering the improved practicality and compliance of HRV measurements acquired through portable devices in the field setting. Practitioners and researchers should consider the cost–benefit along with the simplicity of the measurement when attempting to increase compliance in acquiring HRV measurements.

Notes

Author Contributions

Ward Dobbs designed the study, coded and analyzed effects, carried out the initial analysis, drafted the initial manuscript, and approved the final manuscript as submitted. Michael Fedewa conceptualized and designed the study, coded and analyzed effects, carried out the initial analysis, drafted the initial manuscript, and approved the final manuscript as submitted. Hayley MacDonald designed the study, coded and analyzed effects, reviewed and revised the initial manuscript, and approved the final manuscript as submitted. Clifton Holmes coded and analyzed effects, reviewed and revised the initial manuscript, and approved the final manuscript as submitted. Zackary Cicone coded and analyzed effects, reviewed and revised the initial manuscript, and approved the final manuscript as submitted. Daniel Plews reviewed and revised the initial manuscript, and approved the final manuscript as submitted. Michael Esco conceptualized the study, coded and analyzed effects, drafted the initial manuscript, and approved the final manuscript as submitted.

Compliance with Ethical Standards

Funding

No sources of funding were used to assist in the preparation of this article.

Conflict of interest

Ward Dobbs, Michael Fedewa, Hayley MacDonald, Clifton Holmes, Zackary Cicone, Daniel Plews and Michael Esco declare that they have no conflicts of interest relevant to the content of this review.

Supplementary material

40279_2019_1061_MOESM1_ESM.docx (17 kb)
Electronic Supplementary Material Table S1a. Standard for Reporting Diagnostic Accuracy Studies Guidelines for Heart Rate Variability Research (STARDHRV) Methodology Study Quality Assessment Tool for Primary-Level Evidence. BMI, body mass index; ES, effect size; ICC, intra-class correlation; LOA, limits of agreement; SD, standard deviation
40279_2019_1061_MOESM2_ESM.docx (18 kb)
Electronic Supplementary Material Table S1b. Item Description of the Standard for Reporting Diagnostic Accuracy Studies Guidelines for Heart Rate Variability Research (STARDHRV) Methodology Study Quality Assessment Tool for Primary-Level Evidence. The source of the STARDHRV quality item is provided, and if applicable, describes whether the item was used in its original form, modified, or newly added by authors. Abbreviations: BMI, body mass index; ECG, electrocardiogram; GRAPH, Guidelines for Reporting Articles on Psychiatry and Heart rate variability; ICC, intra-class correlation; LOA, limits of agreement; STARD, Standard for Reporting Diagnostic Accuracy Studies Guidelines
40279_2019_1061_MOESM3_ESM.docx (29 kb)
Electronic Supplementary Material Table S2. Description of selected studies (k = 23) examining the validity of heart rate variability measurements obtained from portable heart rate monitors compared to a criterion electrocardiogram.  %, percentage; a, calculated from data provided; ApEn, approximate entropy; AR, autoregressive; b/w, between; BMI, body mass index; kg/m, kilograms/meters; CWT, continuous wavelet transform; ECG, electrocardiogram; FFT, fast Fourier transform; HF, high frequency; HR, heart rate; HRV, heart rate variability; LF, low frequency; lnRMSSD, log transformed RMSSD; LoA, limits of agreement; N, study population size; NR, not reported; PA, physical activity; pNN50, percentage of consecutive N-N intervals that deviate from one another by more than 50 ms; RR, RR interval; RMSSD, square root of the mean squared differences between normal adjacent R-R intervals; SaEn, sample entropy; SD1, dispersion of points perpendicular to the line of identity; SD2, dispersion of points along the line of identity; SDNN, standard deviation of all normal–normal (R-R) intervals; TP, total power; VLF, very low frequency; WP, Welch’s paradigm; yrs, year
40279_2019_1061_MOESM4_ESM.docx (20 kb)
Electronic Supplementary Material Table S3. Item-by-item summary of methodology study quality for the included studies (k = 23) using a version of the Standard for Reporting Diagnostic Accuracy Studies modified for the use of heart rate variability (STARDHRV). n/a, not applicable due to the study design
40279_2019_1061_MOESM5_ESM.docx (21 kb)
Electronic Supplementary Material Table S4. Individual moderator analyses for categorical variables of interest after removal of outliers (n = 275 total effects).  %, proportion of effects accounted for; a, significant omnibus test; b, significantly different from all other measurements within the variable; ES, estimated absolute standardized mean difference effect size; HF, high frequency; HR, heart rate; HRV, heart rate variability; LF, low frequency; LF:HF, LF to HF ratio; n, number of effects; pNN50, percentage of consecutive N-N intervals that deviate from one another by more than 50 ms; PPG, photoplethysmogrophy; RMSSD, square root of the mean squared differences between normal adjacent R-R intervals; SD1, dispersion of points perpendicular to the line of identity; SD2, dispersion of points along the line of identity; SDNN, standard deviation of all normal–normal (R-R) intervals; SE, standard error; TP, total power; VLF, very low frequency
40279_2019_1061_MOESM6_ESM.docx (13 kb)
Electronic Supplementary Material Table S5: Moderating relationships greater than zero within the multiple moderator model after removal of outliers (n = 275 total effects). CI, confidence interval; ES, estimated absolute standardized mean difference effect size; HF, high frequency power; LF, low frequency power; SDNN, standard deviation of all normal–normal (R-R) intervals; SE, standard error; RMSSD, square root of the mean squared differences between normal adjacent R-R intervals

References

  1. 1.
    Thompson WR. Worldwide survey of fitness trends for 2017. ACSMs Health Fit J. 2016;20(6):8–17.Google Scholar
  2. 2.
    Thompson WR. Worldwide survey of fitness trends for 2018: the CREP edition. ACSMs Health Fit J. 2017;21(6):10–9.CrossRefGoogle Scholar
  3. 3.
    Plews DJ, Laursen PB, Stanley J, Kilding AE, Buchheit M. Training adaptation and heart rate variability in elite endurance athletes: opening the door to effective monitoring. Sports Med. 2013;43(9):773–81.CrossRefGoogle Scholar
  4. 4.
    Silva VP, Oliveira NA, Silveira H, Mello RGT, Deslandes AC. Heart rate variability indexes as a marker of chronic adaptation in athletes: a systematic review. Ann Noninvasive Electrocardiol. 2015;20(2):108–18.CrossRefGoogle Scholar
  5. 5.
    Malik M, Camm AJ, Bigger JT Jr, Breithardt G, Cerutti S, Cohen RJ, et al. Heart rate variability. Standards of measurement, physiological interpretation, and clinical use. Eur Heart J. 1996;17(3):354–81.CrossRefGoogle Scholar
  6. 6.
    Radespiel-Tröger M, Rauh R, Mahlke C, Gottschalk T, Muck-Weymann M. Agreement of two different methods for measurement of heart rate variability. Clin Auton Res. 2003;13(2):99–102.CrossRefGoogle Scholar
  7. 7.
    Flatt AA, Esco MR, Nakamura FY, Plews DJ. Interpreting daily heart rate variability changes in collegiate female soccer players. J Sports Med Phys Fitness. 2017;57(6):907–15.Google Scholar
  8. 8.
    Flatt AA, Esco MR. Evaluating individual training adaptation with smartphone-derived heart rate variability in a collegiate female soccer team. J Strength Cond Res. 2016;30(2):378–85.CrossRefGoogle Scholar
  9. 9.
    Esco MR, Flatt AA, Nakamura FY. Initial weekly HRV response is related to the prospective change in VO2max in female soccer players. Int J Sports Med. 2016;37(6):436–41.CrossRefGoogle Scholar
  10. 10.
    Nakamura FY, Pereira LA, Esco MR, Flatt AA, Moraes JE, Abad CCC, et al. Intraday and interday reliability of ultra-short-term heart rate variability in rugby union players. J Strength Cond Res. 2017;31(2):548–51.Google Scholar
  11. 11.
    Flatt AA, Esco MR, Allen JR, Robinson JB, Earley RL, Fedewa MV, et al. Heart rate variability and training load among National Collegiate Athletic Association Division 1 college football players throughout spring camp. J Strength Cond Res. 2017.  https://doi.org/10.1519/jsc.0000000000002241 (Epub ahead of print).Google Scholar
  12. 12.
    Flatt AA, Esco MR. Heart rate variability stabilization in athletes: towards more convenient data acquisition. Clin Physiol Funct Imaging. 2016;36(5):331–6.CrossRefGoogle Scholar
  13. 13.
    Berntson GG, Bigger JT Jr, Eckberg DL, Grossman P, Kaufmann PG, Malik M, et al. Heart rate variability: origins, methods, and interpretive caveats. Psychophysiology. 1997;34(6):623–48.CrossRefGoogle Scholar
  14. 14.
    Board L, Ispoglou T, Ingle L. Validity of telemetric-derived measures of heart rate variability: a systematic review. J Exerc Physiol Online. 2016;19(6):64–84.Google Scholar
  15. 15.
    Barbosa MP, da Silva NT, de Azevedo FM, Pastre CM, Vanderlei LC. Comparison of Polar(R) RS800G3 heart rate monitor with Polar(R) S810i and electrocardiogram to obtain the series of RR intervals and analysis of heart rate variability at rest. Clin Physiol Funct Imaging. 2016;36(2):112–7.CrossRefGoogle Scholar
  16. 16.
    Bolkhovsky JB, Scully CG, Chon KH, editors. Statistical analysis of heart rate and heart rate variability monitoring through the use of smart phone cameras. In: Engineering in Medicine and Biology Society (EMBC), 2012 annual international conference of the IEEE; 2012: IEEE.Google Scholar
  17. 17.
    Wallen MB, Hasson D, Theorell T, Canlon B, Osika W. Possibilities and limitations of the Polar RS800 in measuring heart rate variability at rest. Eur J Appl Physiol. 2012;112(3):1153–65.CrossRefGoogle Scholar
  18. 18.
    Hernando D, Garatachea N, Almeida R, Casajús JA, Bailón R. Validation of heart rate monitor Polar RS800 for heart rate variability analysis during exercise. J Strength Cond Res. 2018;32(3):716–25.Google Scholar
  19. 19.
    Kingsley M, Lewis MJ, Marson RE. Comparison of Polar 810 s and an ambulatory ECG System for RR interval measures during progressive exercise. Int J Sports Med. 2005;26(1):39–44.CrossRefGoogle Scholar
  20. 20.
    Romagnoli M, Alis R, Guillen J, Basterra J, Villacastin J, Guillen S. A novel device based on smart textile to control heart’s activity during exercise. Australas Phys Eng Sci Med. 2014;37(2):377–84.CrossRefGoogle Scholar
  21. 21.
    Weippert M, Kumar M, Kreuzfeld S, Arndt D, Rieger A, Stoll R. Comparison of three mobile devices for measuring R-R intervals and heart rate variability: Polar S810i, Suunto t6 and an ambulatory ECG system. Eur J Appl Physiol. 2010;109(4):779–86.CrossRefGoogle Scholar
  22. 22.
    Chellakumar PJ, Brumfield A, Kunderu K, Schopper AW. Heart rate variability: comparison among devices with different temporal resolutions. Physiol Meas. 2005;26(6):979–86.CrossRefGoogle Scholar
  23. 23.
    Russoniello CV, Zhirnov YN, Pougatchev VI, Gribkov EN. Heart rate variability and biological age: implications for health and gaming. Cyberpsychol Behav Soc Netw. 2013;16(4):302–8.CrossRefGoogle Scholar
  24. 24.
    Esco MR, Flatt AA, Nakamura FY. Agreement between a smartphone pulse sensor application and electrocardiography for determining lnRMSSD. J Strength Cond Res. 2017;31(2):380–5.Google Scholar
  25. 25.
    Flatt AA, Esco MR. Validity of the ithlete™ smart phone application for determining ultra-short-term heart rate variability. J Hum Kinet. 2013;39:85–92.CrossRefGoogle Scholar
  26. 26.
    Vanderlei LCM, Silva RA, Pastre CM, Azevedo FMD, Godoy MF. Comparison of the Polar S810i monitor and the ECG for the analysis of heart rate variability in the time and frequency domains. Braz J Med Biol Res. 2008;41(10):854–9.CrossRefGoogle Scholar
  27. 27.
    Gamelin FX, Berthoin S, Bosquet L. Validity of the Polar S810 heart rate monitor to measure R-R intervals at rest. Med Sci Sports Exerc. 2006;38(5):887–93.CrossRefGoogle Scholar
  28. 28.
    Giles D, Draper N, Neil W. Validity of the Polar V800 heart rate monitor to measure RR intervals at rest. Eur J Appl Physiol. 2016;116(3):563–71.CrossRefGoogle Scholar
  29. 29.
    Montaño A, Brown F, Credeur DP, Williams MA, Stoner L. Telemetry-derived heart rate variability responses to a physical stressor. Clin Physiol Funct Imaging. 2017;37(4):421–7.CrossRefGoogle Scholar
  30. 30.
    Vasconcellos FV, Seabra A, Cunha FA, Montenegro RA, Bouskela E, Farinatti P. Heart rate variability assessment with fingertip photoplethysmography and Polar RS800cx as compared with electrocardiography in obese adolescents. Blood Press Monit. 2015;20(6):351–60.CrossRefGoogle Scholar
  31. 31.
    Nunan D, Jakovljevic DG, Donovan G, Hodges LD, Sandercock GR, Brodie DA. Levels of agreement for RR intervals and short-term heart rate variability obtained from the Polar S810 and an alternative system. Eur J Appl Physiol. 2008;103(5):529–37.CrossRefGoogle Scholar
  32. 32.
    Moher D, Liberati A, Tetzlaff J, Altman D. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:332–6.CrossRefGoogle Scholar
  33. 33.
    Cohen JF, Korevaar DA, Altman DG, Bruns DE, Gatsonis CA, Hooft L, et al. STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration. BMJ Open. 2016;6(11):e012799.CrossRefGoogle Scholar
  34. 34.
    Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, et al. Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. Standards for reporting of diagnostic accuracy. Clin Chem. 2003;49(1):1–6.CrossRefGoogle Scholar
  35. 35.
    Quintana DS, Alvares GA, Heathers JA. Guidelines for reporting articles on psychiatry and heart rate variability (GRAPH): recommendations to advance research communication. Transl Psychiatry. 2016;6:e803.CrossRefGoogle Scholar
  36. 36.
    Laborde S, Mosley E, Thayer JF. Heart rate variability and cardiac vagal tone in psychophysiological research—recommendations for experiment planning, data analysis, and data reporting. Front Psychol. 2017;8:213.CrossRefGoogle Scholar
  37. 37.
    Cohen J. A power primer. Psychol Bull. 1992;112(1):155–9.CrossRefGoogle Scholar
  38. 38.
    Hedges L, Olkin I. Statistical methods for meta-analysis. 6th ed. San Diego: Academic Press; 1985. p. 79–201.Google Scholar
  39. 39.
    Crossingham IR, Nethercott DR, Columb MO. Comparing cardiac output monitors and defining agreement: a systematic review and meta-analysis. J Intensive Care Soc. 2016;17(4):302–13.CrossRefGoogle Scholar
  40. 40.
    Zaki R, Bulgiba A, Ismail R, Ismail NA. Statistical methods used to test for agreement of medical instruments measuring continuous variables in method comparison studies: a systematic review. PLoS One. 2012;7(5):e37908.CrossRefGoogle Scholar
  41. 41.
    Plews DJ, Scott B, Altini M, Wood M, Kilding AE, Laursen PB. Comparison of heart rate variability recording with smart phone photoplethysmographic, Polar H7 chest strap and electrocardiogram methods. Int J Sports Physiol Perform. 2017;12(10):1324–8.CrossRefGoogle Scholar
  42. 42.
    Boos CJ, Bakker-Dyos J, Watchorn J, Woods DR, O’Hara JP, Macconnachie L, et al. A comparison of two methods of heart rate variability assessment at high altitude. Clin Physiol Funct Imaging. 2017;37(6):582–7.CrossRefGoogle Scholar
  43. 43.
    Hox J. Multilevel analysis: techniques and applications. 2nd edn. New York, NY: Taylor & Francis; 2010. p. 205–32.CrossRefGoogle Scholar
  44. 44.
    Singer JD, Using SAS. PROC MIXED to fit multilevel models, hierarchical models, and individual growth models. J Educ Behav Stat. 1998;23(4):323–55.CrossRefGoogle Scholar
  45. 45.
    Higgins J, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med. 2002;21(11):1539–58.CrossRefGoogle Scholar
  46. 46.
    Higgins JP, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003;327(7414):557–60.CrossRefGoogle Scholar
  47. 47.
    Schäfer A, Vagedes J. How accurate is pulse rate variability as an estimate of heart rate variability? A review on studies comparing photoplethysmographic technology with an electrocardiogram. Int J Cardiol. 2013;166(1):15–29.CrossRefGoogle Scholar
  48. 48.
    Egger M, Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997;315(7109):629–34.CrossRefGoogle Scholar
  49. 49.
    Rosenberg M. The file-drawer problem revisited: a general weighted method for calculating fail-safe numbers in meta-analysis. Evolution. 2005;59(2):464–8.CrossRefGoogle Scholar
  50. 50.
    Viechtbauer W. Conducting meta-analyses in R with the metafor package. J Stat Softw. 2010;36(3):1–48.CrossRefGoogle Scholar
  51. 51.
    R Development Core Team. R: a language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. 2017.Google Scholar
  52. 52.
    Gamelin FX, Baquet G, Berthoin S, Bosquet L. Validity of the Polar S810 to measure R-R intervals in children. Int J Sports Med. 2008;29(2):134–8.CrossRefGoogle Scholar
  53. 53.
    Heathers JAJ. Smartphone-enabled pulse rate variability: an alternative methodology for the collection of heart rate variability in psychophysiological research. Int J Psychophysiol. 2013;89:297–304.CrossRefGoogle Scholar
  54. 54.
    Nunan D, Donovan G, Jakovljevic DG, Hodges LD, Sandercock GR, Brodie DA. Validity and reliability of short-term heart-rate variability from the Polar S810. Med Sci Sports Exerc. 2009;41(1):243–50.CrossRefGoogle Scholar
  55. 55.
    Porto LGG, Junqueira J. Comparison of time-domain short-term heart interval variability analysis using a wrist-worn heart rate monitor and the conventional electrocardiogram. Pacing Clin Electrophysiol. 2009;32(1):43–51.CrossRefGoogle Scholar
  56. 56.
    Guzik P, Piekos C, Pierog O, Fenech N, Krauze T, Piskorski J, et al. Classic electrocardiogram-based and mobile technology derived approaches to heart rate variability are not equivalent. Int J Cardiol. 2018;258:154–6.CrossRefGoogle Scholar
  57. 57.
    Sterne JAC, Sutton AJ, Ioannidis JPA, Terrin N, Jones DR, Lau J, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ. 2011;343:d4002.CrossRefGoogle Scholar
  58. 58.
    Begg CB, Berlin JA. Publication bias: a problem in interpreting medical data. J R Stat Soc Ser A. 1988;151(3):419–63.CrossRefGoogle Scholar
  59. 59.
    Sterne JAC, Egger M. Funnel plots for detecting bias in meta-analysis: guidelines on choice of axis. J Clin Epidemiol. 2001;54(10):1046–55.CrossRefGoogle Scholar
  60. 60.
    Rosenthal R. Meta-analytic procedures for social research. 2nd edn. Newbury Park, CA: SAGE Publications; 1991.CrossRefGoogle Scholar
  61. 61.
    Sammito S, Böckelmann I. Options and limitations of heart rate measurement and analysis of heart rate variability by mobile devices: a systematic review. Herzschrittmacherther Elektrophysiol. 2016;27(1):38–45.CrossRefGoogle Scholar
  62. 62.
    Buchheit M, Mendez-Villanueva A. Improbable effect of carbohydrate diet on cardiac autonomic modulation during exercise. Eur J Appl Physiol. 2010;109(3):571–4.CrossRefGoogle Scholar
  63. 63.
    Schmitt L, Regnard J, Millet GP. Monitoring fatigue status with HRV measures in elite athletes: an avenue beyond RMSSD? Front Physiol. 2015;6:343.CrossRefGoogle Scholar
  64. 64.
    Penttila J, Helminen A, Jartti T, Kuusela T, Huikuri HV, Tulppo MP, et al. Time domain, geometrical and frequency domain analysis of cardiac vagal outflow: effects of various respiratory patterns. Clin Physiol. 2001;21(3):365–76.CrossRefGoogle Scholar
  65. 65.
    Buchheit M. Monitoring training status with HR measures: do all roads lead to Rome? Front Physiol. 2014;5:73.CrossRefGoogle Scholar
  66. 66.
    Weir JP. Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. J Strength Cond Res. 2005;19(1):231–40.Google Scholar
  67. 67.
    Esco MR, Flatt AA. Ultra-short-term heart rate variability indexes at rest and post-exercise in athletes: evaluating the agreement with accepted recommendations. J Sports Sci Med. 2014;13(3):535–41.Google Scholar
  68. 68.
    Altini M, Amft O. HRV4Training: large-scale longitudinal training load analysis in unconstrained free-living settings using a smartphone application. Conf Proc IEEE Eng Med Biol Soc. 2016;2016:2610–3.Google Scholar
  69. 69.
    Plews DJ, Laursen PB, Buchheit M. Day-to-day heart-rate variability recordings in world-champion rowers: appreciating unique athlete characteristics. Int J Sports Physiol Perform. 2017;12(5):697–703.CrossRefGoogle Scholar
  70. 70.
    Nuuttila OP, Nikander A, Polomoshnov D, Laukkanen JA, Hakkinen K. Effects of HRV-guided vs. predetermined block training on performance, HRV and serum hormones. Int J Sports Med. 2017;38(12):909–20.CrossRefGoogle Scholar
  71. 71.
    Plews DJ, Laursen PB, Kilding AE, Buchheit M. Evaluating training adaptation with heart-rate measures: a methodological comparison. Int J Sports Physiol Perform. 2013;8(6):688–91.CrossRefGoogle Scholar
  72. 72.
    Lemeshow AR, Blum RE, Berlin JA, Stoto MA, Colditz GA. Searching one or two databases was insufficient for meta-analysis of observational studies. J Clin Epidemiol. 2005;58(9):867–73.CrossRefGoogle Scholar
  73. 73.
    Whiting P, Westwood M, Burke M, Sterne J, Glanville J. Systematic reviews of test accuracy should search a range of databases to identify primary studies. J Clin Epidemiol. 2008;61(4):357–64.CrossRefGoogle Scholar
  74. 74.
    Papaioannou D, Sutton A, Carroll C, Booth A, Wong R. Literature searching for social science systematic reviews: consideration of a range of search techniques. Health Inf Libr J. 2010;27(2):114–22.CrossRefGoogle Scholar
  75. 75.
    Linder SK, Kamath GR, Pratt GF, Saraykar SS, Volk RJ. Citation searches are more sensitive than keyword searches to identify studies using specific measurement instruments. J Clin Epidemiol. 2015;68(4):412–7.CrossRefGoogle Scholar
  76. 76.
    Allen IE, Olkin I. Estimating time to conduct a meta-analysis from number of citations retrieved. JAMA. 1999;282(7):634–5.CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Department of Exercise and Sport ScienceUniversity of Wisconsin - La CrosseLa CrosseUSA
  2. 2.Department of KinesiologyThe University of AlabamaTuscaloosaUSA
  3. 3.Sports Performance Research Institute New Zealand (SPRINZ)Auckland University of TechnologyAucklandNew Zealand

Personalised recommendations