Skip to main content

Advertisement

SpringerLink
Log in

A survey of surveys: an evaluation of the quality of published surveys in neurosurgery

  • Original Article
  • Published:
Acta Neurochirurgica Aims and scope Submit manuscript

Abstract

Purpose

Surveys generate valuable data in epidemiologic and qualitative clinical research. The quality of a survey depends on its design, the number of responses it receives, and the reporting of the results. In this study, we aimed to assess the quality of surveys in neurosurgery.

Methods

Neurosurgical surveys published between 2000 and 2020 (inclusive) were identified from PubMed. Various datapoints regarding the surveys were collated. The number of citations received by the papers was determined from Google Scholar. A 6-dimensional quality assessment tool was applied to the surveys. Parameters from this tool were combined with the number of responses received to create the survey quality score (SQS).

Results

A total of 618 surveys were included for analysis. The target sample size correlated with the number of responses received. The response rate correlated positively with the target sample size and the number of reminders sent and negatively with the number of questions in the survey. The median number of authors on neurosurgery survey papers was 6. The number of authors correlated with the SQS and the number of citations received by published survey papers. The median normalized SQS for neurosurgical surveys was 65%. The nSQS independently predicted the citations received per year by surveys.

Conclusions

The modifiable factors that correlated with improvements in survey design were optimizing the number of questions, maximizing the target sample size, and incorporating reminders in the survey design. Increasing the number of contributing authors led to improvements in survey quality. The SQS was validated and correlated well with the citations received by surveys.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Abbreviations

COVID:

Coronavirus disease

CPY:

Citations per year

IF:

Impact Factor ®

IQR:

Interquartile range

nSQS:

Normalized Survey Quality Score

QET:

Quality evaluation tool

Rn :

Number of responses received

RR:

Response rate

RScore:

Response score

SD:

Standard deviation

SQS:

Survey Quality Score

References

  1. Aerny-Perreten N, Dominguez-Berjon MF, Esteban-Vasallo MD, Garcia-Riolobos C (2015) Participation and factors associated with late or non-response to an online survey in primary care. J Eval Clin Pract 21:688–693. https://doi.org/10.1111/jep.12367

    Article  PubMed  Google Scholar 

  2. Ahmed A, Adam M, Ghafar NA, Muhammad M, Ebrahim NA (2016) Impact of article page count and number of authors on citations in disability related fields: a systematic review article. Iran J Public Health 45:1118–1125

    PubMed  PubMed Central  Google Scholar 

  3. Archer T (2017) Characteristics associated with increasing the response rates of web-based surveys. Practical Assessment, Research, and Evaluation 12. https://doi.org/10.7275/ec5p-zg50

  4. Artino AR Jr, Phillips AW, Utrankar A, Ta AQ, Durning SJ (2018) “The questions shape the answers”: assessing the quality of published survey instruments in health professions education research. Acad Med 93:456–463. https://doi.org/10.1097/ACM.0000000000002002

    Article  PubMed  Google Scholar 

  5. Attenello FJ, Buchanan IA, Wen T, Donoho DA, McCartney S, Cen SY, Khalessi AA, Cohen-Gadol AA, Cheng JS, Mack WJ, Schirmer CM, Swartz KR, Prall JA, Stroink AR, Giannotta SL, Klimo P (2018) Factors associated with burnout among US neurosurgery residents: a nationwide survey. J Neurosurg 129:1349–1363. https://doi.org/10.3171/2017.9.JNS17996

    Article  PubMed  Google Scholar 

  6. Badger F, Werrett J (2005) Room for improvement? Reporting response rates and recruitment in nursing research in the past decade. J Adv Nurs 51:502–510

    Article  PubMed  Google Scholar 

  7. Bennett C, Khangura S, Brehaut JC, Graham ID, Moher D, Potter BK, Grimshaw JM (2010) Reporting guidelines for survey research: an analysis of published guidance and reporting practices. PLoS Med 8:e1001069. https://doi.org/10.1371/journal.pmed.1001069

    Article  PubMed  Google Scholar 

  8. Bjertnaes OA, Garratt A, Botten G (2008) Nonresponse bias and cost-effectiveness in a Norwegian survey of family physicians. Eval Health Prof 31:65–80. https://doi.org/10.1177/0163278707311874

    Article  PubMed  Google Scholar 

  9. Bramstedt KA (2020) The carnage of substandard research during the COVID-19 pandemic: a call for quality. J Med Ethics 46:803–807. https://doi.org/10.1136/medethics-2020-106494

    Article  PubMed  Google Scholar 

  10. Burns KEA, Kho ME (2015) How to assess a survey report: a guide for readers and peer reviewers. CMAJ 187:E198–E205. https://doi.org/10.1503/cmaj.140545

    Article  PubMed  PubMed Central  Google Scholar 

  11. Cheung KL, Ten Klooster PM, Smit C, de Vries H, Pieterse ME (2017) The impact of non-response bias due to sampling in public health studies: a comparison of voluntary versus mandatory recruitment in a Dutch national survey on adolescent health. BMC Public Health 17:276. https://doi.org/10.1186/s12889-017-4189-8

    Article  PubMed  PubMed Central  Google Scholar 

  12. Chirico F, Teixeira da Silva JA, Magnavita N (2020) “Questionable” peer review in the publishing pandemic during the time of COVID-19: implications for policy makers and stakeholders. Croat Med J 61:300–301. https://doi.org/10.3325/cmj.2020.61.300

    Article  PubMed  PubMed Central  Google Scholar 

  13. Cohen-Gadol AA, Piepgras DG, Krishnamurthy S, Fessler RD (2005) Resident duty hours reform: results of a national survey of the program directors and residents in neurosurgery training programs. Neurosurgery 56:398–403; discussion 398–403. https://doi.org/10.1227/01.neu.0000147999.64356.57

  14. Compton J, Glass N, Fowler T (2019) Evidence of selection bias and non-response bias in patient satisfaction surveys. Iowa Orthop J 39:195–201

    PubMed  PubMed Central  Google Scholar 

  15. Cook JV, Dickinson HO, Eccles MP (2009) Response rates in postal surveys of healthcare professionals between 1996 and 2005: an observational study. BMC Health Serv Res 9:160. https://doi.org/10.1186/1472-6963-9-160

    Article  PubMed  PubMed Central  Google Scholar 

  16. Cuschieri S (2019) The STROBE guidelines. Saudi J Anaesth 13:S31–S34. https://doi.org/10.4103/sja.SJA_543_18

    Article  PubMed  PubMed Central  Google Scholar 

  17. de Koning R, Egiz A, Kotecha J, Ciuculete AC, Ooi SZY, Bankole NDA, Erhabor J, Higginbotham G, Khan M, Dalle DU, Sichimba D, Bandyopadhyay S, Kanmounye US (2021) Survey fatigue during the COVID-19 pandemic: an analysis of neurosurgery survey response rates. Front Surg 8:690680. https://doi.org/10.3389/fsurg.2021.690680

  18. Duffett M, Burns KE, Adhikari NK, Arnold DM, Lauzier F, Kho ME, Meade MO, Hayani O, Koo K, Choong K (2012) Quality of reporting of surveys in critical care journals: a methodologic review. Crit Care Med 40:441–449

    Article  PubMed  Google Scholar 

  19. Eysenbach G (2004) Improving the quality of Web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES). vol 6. Gunther Eysenbach Centre for Global eHealth Innovation, Toronto, Canada,

  20. Fincham JE (2008) Response rates and responsiveness for surveys, standards, and the Journal. American journal of pharmaceutical education 72

  21. Geyer ED, Miller R, Kim SS, Tobias JD, Nafiu OO, Tumin D (2020) Quality and impact of survey research among anesthesiologists: a systematic review. Adv Med Educ Pract 11:587–599. https://doi.org/10.2147/AMEP.S259908

    Article  PubMed  PubMed Central  Google Scholar 

  22. Greenberg JK, Jeffe DB, Carpenter CR, Yan Y, Pineda JA, Lumba-Brown A, Keller MS, Berger D, Bollo RJ, Ravindra VM, Naftel RP, Dewan MC, Shah MN, Burns EC, O’Neill BR, Hankinson TC, Whitehead WE, Adelson PD, Tamber MS, McDonald PJ, Ahn ES, Titsworth W, West AN, Brownson RC, Limbrick DD (2018) North American survey on the post-neuroimaging management of children with mild head injuries. J Neurosurg Pediatr 23:227–235. https://doi.org/10.3171/2018.7.PEDS18263

    Article  PubMed  PubMed Central  Google Scholar 

  23. Grimshaw J (2014) Surge (the survey reporting guideline). Guidelines for reporting health research: a user's manual 206–213

  24. Hlatshwako TG, Shah SJ, Kosana P, Adebayo E, Hendriks J, Larsson EC, Hensel DJ, Erausquin JT, Marks M, Michielsen K, Saltis H, Francis JM, Wouters E, Tucker JD (2021) Online health survey research during COVID-19. Lancet Digit Health 3:e76–e77. https://doi.org/10.1016/S2589-7500(21)00002-9

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  25. Jaykaran, (2011) How to increase response rate to a questionnaire study? Indian J Pharmacol 43:93–94. https://doi.org/10.4103/0253-7613.75687

    Article  CAS  PubMed  Google Scholar 

  26. Jones D, Story D, Clavisi O, Jones R, Peyton P (2006) An introductory guide to survey research in anaesthesia. Anaesth Intensive Care 34:245–253. https://doi.org/10.1177/0310057X0603400219

    Article  CAS  PubMed  Google Scholar 

  27. Kelley K, Clark B, Brown V, Sitzia J (2003) Good practice in the conduct and reporting of survey research. Int J Qual Health Care 15:261–266. https://doi.org/10.1093/intqhc/mzg031

    Article  PubMed  Google Scholar 

  28. Khan NR, Derstine PL, Gienapp AJ, Klimo P, Barbaro NM (2020) A survey of neurological surgery residency program mentorship practices compared to accreditation council for graduate medical education resident outcome data. Neurosurgery 87:E566–E572. https://doi.org/10.1093/neuros/nyz479

    Article  PubMed  Google Scholar 

  29. Langbecker D, Caffery LJ, Gillespie N, Smith AC (2017) Using survey methods in telehealth research: a practical guide. J Telemed Telecare 23:770–779. https://doi.org/10.1177/1357633X17721814

    Article  PubMed  Google Scholar 

  30. Leidman E, Mwirigi LM, Maina-Gathigi L, Wamae A, Imbwaga AA, Bilukha OO (2018) Assessment of anthropometric data following investments to ensure quality: Kenya demographic health surveys case study, 2008 to 2009 and 2014. Food Nutr Bull 39:406–419. https://doi.org/10.1177/0379572118783181

    Article  PubMed  PubMed Central  Google Scholar 

  31. Leung L (2015) Validity, reliability, and generalizability in qualitative research. Journal of family medicine and primary care 4:324

    Article  PubMed  PubMed Central  Google Scholar 

  32. Li AH-T, Thomas SM, Farag A, Duffett M, Garg AX, Naylor KL (2014) Quality of survey reporting in nephrology journals: a methodologic review. Clin J Am Soc Nephrol 9:2089

    Article  PubMed  PubMed Central  Google Scholar 

  33. Madhugiri VS, Moiyadi A, Nagella AB, Singh V, Shetty P (2021) A questionnaire-based survey of clinical neuro-oncological practice in India. Neurol India 69:659–664. https://doi.org/10.4103/0028-3886.319199

    Article  PubMed  Google Scholar 

  34. McFarlane E, Olmsted MG, Murphy J, Hill CA (2007) Nonresponse bias in a mail survey of physicians. Eval Health Prof 30:170–185. https://doi.org/10.1177/0163278707300632

    Article  PubMed  Google Scholar 

  35. McGhan WF, Al M, Doshi JA, Kamae I, Marx SE, Rindress D (2009) The ISPOR good practices for quality improvement of cost-effectiveness research task force report. Value Health 12:1086–1099. https://doi.org/10.1111/j.1524-4733.2009.00605.x

    Article  PubMed  Google Scholar 

  36. McGovern ME, Canning D, Barnighausen T (2018) Accounting for non-response bias using participation incentives and survey design: an application using gift vouchers. Econ Lett 171:239–244. https://doi.org/10.1016/j.econlet.2018.07.040

    Article  PubMed  PubMed Central  Google Scholar 

  37. Meng-Jia Wu KZ, Francisca Fils-Aime (2022) Response rates of online surveys in published research: a meta-analysis. Computers in Human Behavior Reports 7

  38. Miller C, Lundy P, Woodrow S (2020) International electives in neurological surgery training: a survey of program directors from Accreditation Council for Graduate Medical Education-approved neurological surgery programs. J Neurosurg 134:1967–1973. https://doi.org/10.3171/2020.4.JNS20618

    Article  PubMed  Google Scholar 

  39. Mukuria AG (2003) Using national health and nutrition surveys for policy and programs: experiences from the demographic and health surveys. Forum Nutr 56:207–208

    PubMed  Google Scholar 

  40. Nagella AB, Ravishankar M, Hemanth Kumar VR (2016) Anaesthesia practice and reproductive outcomes: facts unveiled. Indian J Anaesth 60:225. https://doi.org/10.4103/0019-5049.177883

    Article  PubMed  PubMed Central  Google Scholar 

  41. Pagano MB, Dunbar NM, Tinmouth A, Apelseth TO, Lozano M, Cohn CS, Stanworth SJ, Collaborative BEfST, (2018) A methodological review of the quality of reporting of surveys in transfusion medicine. Transfusion 58:2720–2727

    Article  PubMed  Google Scholar 

  42. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE (2021) The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Int J Surg 88:105906

    Article  PubMed  Google Scholar 

  43. Patel SS, Webster RK, Greenberg N, Weston D, Brooks SK (2020) Research fatigue in COVID-19 pandemic and post-disaster research: causes, consequences and recommendations. Disaster Prev Manag 29(4):445–455. https://doi.org/10.1108/DPM-05-2020-0164

    Article  PubMed  PubMed Central  Google Scholar 

  44. Pit SW, Vo T, Pyakurel S (2014) The effectiveness of recruitment strategies on general practitioner’s survey response rates - a systematic review. BMC Med Res Methodol 14:76. https://doi.org/10.1186/1471-2288-14-76

    Article  PubMed  PubMed Central  Google Scholar 

  45. Rathi K, Kamboj P, Bansal PG, Toteja GS (2018) A review of selected nutrition & health surveys in India. Indian J Med Res 148:596–611. https://doi.org/10.4103/ijmr.IJMR_1808_18

    Article  PubMed  PubMed Central  Google Scholar 

  46. Rosen T, Olsen J (2006) Invited commentary: the art of making questionnaires better. Am J Epidemiol 164:1145–1149

    Article  PubMed  Google Scholar 

  47. Rybakov KN, Beckett R, Dilley I, Sheehan AH (2020) Reporting quality of survey research articles published in the pharmacy literature. Res Social Adm Pharm 16:1354–1358. https://doi.org/10.1016/j.sapharm.2020.01.005

    Article  PubMed  Google Scholar 

  48. Sammut R, Griscti O, Norman IJ (2021) Strategies to improve response rates to web surveys: a literature review. Int J Nurs Stud 123:104058. https://doi.org/10.1016/j.ijnurstu.2021.104058

    Article  PubMed  Google Scholar 

  49. Schilling LM, Kozak K, Lundahl K, Dellavalle RP (2006) Inaccessible novel questionnaires in published medical research: hidden methods, hidden costs. Am J Epidemiol 164:1141–1144

    Article  PubMed  Google Scholar 

  50. Schulz KF, Altman DG, Moher D (2010) CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. J Pharmacol Pharmacother 1:100–107

    Article  PubMed  PubMed Central  Google Scholar 

  51. Sharma A, Minh Duc NT, Luu Lam Thang T, Nam NH, Ng SJ, Abbas KS, Huy NT, Marušić A, Paul CL, Kwok J (2021) A consensus-based checklist for reporting of survey studies (CROSS). J Gen Intern Med 36:3179–3187

    Article  PubMed  PubMed Central  Google Scholar 

  52. Singh S, Sagar R (2021) A critical look at online survey or questionnaire-based research studies during COVID-19. Asian J Psychiatr 65:102850. https://doi.org/10.1016/j.ajp.2021.102850

    Article  PubMed  PubMed Central  Google Scholar 

  53. So R, Shinohara K, Aoki T, Tsujimoto Y, Suganuma AM, Furukawa TA (2018) Effect of recruitment methods on response rate in a web-based study for primary care physicians: factorial randomized controlled trial. J Med Internet Res 20:e28. https://doi.org/10.2196/jmir.8561

    Article  PubMed  PubMed Central  Google Scholar 

  54. Story DA, Gin V, na Ranong V, Poustie S, Jones D; ANZCA Trials Group (2011) Inconsistent survey reporting in anesthesia journals. Anesth Analg 113(3):591–595. https://doi.org/10.1213/ANE.0b013e3182264aaf

    Article  Google Scholar 

  55. Koch GG, Gillings DB, Stokes ME (1980) Biostatistical implications of design, sampling, and measurement to health science data analysis. Annu Rev Public Health 1:163–225. https://doi.org/10.1146/annurev.pu.01.050180.001115

    Article  CAS  PubMed  Google Scholar 

  56. Teixeira da Silva JA, Bornemann-Cimenti H, Tsigaris P (2021) Optimizing peer review to minimize the risk of retracting COVID-19-related literature. Med Health Care Philos 24:21–26. https://doi.org/10.1007/s11019-020-09990-z

    Article  PubMed  Google Scholar 

  57. Tran EM, Tran MM, Clark MA, Scott IU, Margo CE, Cosenza C, Johnson TP, Greenberg PB (2020) Assessing the quality of published surveys in ophthalmology. Ophthalmic Epidemiol 27:339–343. https://doi.org/10.1080/09286586.2020.1746359

    Article  PubMed  Google Scholar 

  58. Turgeon AF, Lauzier F, Burns KE, Meade MO, Scales DC, Zarychanski R, Moore L, Zygun DA, McIntyre LA, Kanji S, Hebert PC, Murat V, Pagliarello G, Fergusson DA, Canadian Critical Care Trials G (2013) Determination of neurologic prognosis and clinical decision making in adult patients with severe traumatic brain injury: a survey of Canadian intensivists, neurosurgeons, and neurologists. Crit Care Med 41:1086–1093. https://doi.org/10.1097/CCM.0b013e318275d046

  59. Tyrer S, Heyman B (2016) Sampling in epidemiological research: issues, hazards and pitfalls. BJPsych Bull 40:57–60. https://doi.org/10.1192/pb.bp.114.050203

    Article  PubMed  PubMed Central  Google Scholar 

  60. Van Otterloo J, Richards JL, Seib K, Weiss P, Omer SB (2011) Gift card incentives and non-response bias in a survey of vaccine providers: the role of geographic and demographic factors. PLoS ONE 6:e28108. https://doi.org/10.1371/journal.pone.0028108

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  61. Vardhana Rao MV, Sahu D, Nair S, Sharma RK, Gulati BK, Acharya R, Mahapatra B, Ramesh S, Khan N, Chaudhuri T, Sandal K, Deepani V, Dey S, Saggurti N (2022) National guidelines for data quality in surveys: an overview. Indian J Med Res 156:715–720. https://doi.org/10.4103/ijmr.ijmr_1261_22

    Article  PubMed  Google Scholar 

  62. Zeleke AA, Worku AG, Demissie A, Otto-Sobotka F, Wilken M, Lipprandt M, Tilahun B, Rohrig R (2019) Evaluation of electronic and paper-pen data capturing tools for data quality in a public health survey in a health and demographic surveillance site, Ethiopia: randomized controlled crossover health care information technology evaluation. JMIR Mhealth Uhealth 7:e10995. https://doi.org/10.2196/10995

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Epilepsy Division, Department of Neurology, University of Florida, Gainesville, FL, USA

    Subeikshanan Venkatesan

  2. Division of Thoracic Surgery, Roswell Park Comprehensive Cancer Center, Buffalo, NY, USA

    Sukumar Kalvapudi

  3. Department of Neurosurgery, Jawaharlal Institute of Postgraduate Medical Education and Research, Pondicherry, India

    Varun Muppidi & Karthik Ajith

  4. Department of General Surgery, All India Institute of Medical Sciences, Jodhpur, Rajasthan, India

    Akshat Dutt

  5. Gamma Knife Center, Department of Radiation Medicine, Roswell Park Comprehensive Cancer Center, Buffalo, NY, USA

    Venkatesh Shankar Madhugiri

Authors
  1. Subeikshanan Venkatesan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sukumar Kalvapudi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Varun Muppidi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Karthik Ajith
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Akshat Dutt
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Venkatesh Shankar Madhugiri
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Venkatesh Shankar Madhugiri.

Ethics declarations

Ethics approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Venkatesan, S., Kalvapudi, S., Muppidi, V. et al. A survey of surveys: an evaluation of the quality of published surveys in neurosurgery. Acta Neurochir 166, 150 (2024). https://doi.org/10.1007/s00701-024-06042-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00701-024-06042-w

Keywords

Search

Navigation