Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Comparing web-based versus face-to-face and paper-and-pencil questionnaire data collected through two Belgian health surveys



Using the European Health Interview Survey (EHIS) questionnaire, a web-based survey was organized alongside a face-to-face (F2F) survey including a paper-and-pencil (P&P) questionnaire for sensitive topics. Associated with these different modes, other design features varied too (e.g., recruitment, incentives, sampling). We assessed whether these whole data collection systems developed around the modes produced equivalent health estimates.


Data were obtained from two population-based surveys: the EHISWEB (web-administered, n = 1010) and the Belgian Health Interview Survey 2018 (BHIS2018) (interviewer-administered, n = 2748). Logistic regression analyses were used to assess mode system differences while adjusting for socio-demographic differences in the net samples.


For the P&P mode of the BHIS, significant mode system differences were detected for 2 of the 9 health indicators. Among the indicators collected via the F2F mode, 9 of the 18 indicators showed significant differences.


Indicators collected via the web-based and P&P self-administered modes were generally more comparable than indicators collected via the web-based and F2F mode. Furthermore, fewer differences were detected for indicators based on simple and factual questions compared to indicators based on subjective or complex questions.

This is a preview of subscription content, log in to check access.


  1. Adams SA, Matthews CE, Ebbeling CB et al (2005) The effect of social desirability and social approval on self-reports of physical activity. Am J Epidemiol 161:389–398.

  2. Biemer PP, Lyberg LE (2003) Introduction to survey quality. Wiley, Hoboken

  3. Bowling A (2005) Mode of questionnaire administration can have serious effects on data quality. J Public Health 27:281–291.

  4. Braekman E, Berete F, Charafeddine R et al (2018) Measurement agreement of the self-administered questionnaire of the Belgian Health Interview Survey: paper-and-pencil versus web-based mode. PLoS ONE 13:e0197434.

  5. Braekman E, Charafeddine R, Demarest S et al (2019) Is the European Health Interview Survey online yet? Response and net sample composition of a web-based data collection. Eur J Public Health.

  6. Burkill S, Copas A, Couper MP et al (2016) Using the web to collect data on sensitive behaviours: a study looking at mode effects on the British National Survey of Sexual Attitudes and Lifestyles. PLoS ONE 11:e0147983.

  7. Christensen AI, Ekholm O, Glümer C et al (2014) Effect of survey mode on response patterns: comparison of face-to-face and self-administered modes in health surveys. Eur J Public Health 24:327–332.

  8. Cox B, Van Oyen H, Cambois E et al (2009) The reliability of the minimum European health module. Int J Public Health 54:55–60.

  9. De Leeuw ED (2008) Choosing the method of data collection. In: De Leeuw ED, Hox JJ, Dillman DA (eds) International handbook of survey methodology. Lawrence Erlbaum Associates, New York, pp 113–135

  10. De Leeuw ED, Hox JJ, Dillman DA (2008) Mixed-mode surveys: when and why. In: De Leeuw ED, Hox JJ, Dillman DA (eds) International handbook of survey methodology. Lawrence Erlbaum Associates, New York, pp 299–316

  11. Demarest S, Berete F, Charafeddine R, Van der Heyden J. Gezondheidsenquête 2018: Methodologie. Report number: D/2019/14.440/34. Sciensano, Brussels. Accessed 6 November 2019.

  12. Demarest S, Van der Heyden J, Charafeddine R et al (2013) Methodological basics and evolution of the Belgian Health Interview Survey 1997–2008. Arch Public Health 71:24.

  13. European Union (2018) Commission Regulation (EU) 2018/255: Implementing Regulation (EC) No 1338/2008 of the European Parliament and of the Council as regards statistics based on the European Health Interview Survey (EHIS). Off J Eur Union 48/12:dd21-2-2018.

  14. Eurostat (2018) European Health Interview Survey (EHIS wave 3) methodological manual. Publication Office of the European Union. Accessed 30 May 2018

  15. Feveile H, Olsen O, Hogh A (2007) A randomized trial of mailed questionnaires versus telephone interviews: response patterns in a survey. BMC Med Res Methodol 7:27.

  16. Finger JD, Tafforeau J, Gisle L et al (2015) Development of the European Health Interview Survey-Physical Activity Questionnaire (EHIS-PAQ) to monitor physical activity in the European Union. Arch Public Health 73:59.

  17. Gnambs T, Kaspar K (2015) Disclosure of sensitive behaviors across self-administered survey modes: a meta-analysis. Behav Res Methods 47:1237–1259.

  18. Hoebel J, von der Lippe E, Lange C et al (2014) Mode differences in a mixed-mode health interview survey among adults. Arch Public Health 72:46.

  19. Holbrook A, Cho YI, Johnson T (2006) The impact of question and respondent characteristics on comprehension and mapping difficulties. Public Opin Q 70:565–595.

  20. Keeter S, McGeeney K, Igielnik R et al (2015) From telephone to the web: the challenge of mode of interview effects in public opinion polls. Pew Research Center. Accessed 30 May 2019

  21. Kralj M, Zaletel M, Lavtar D et al (2015) Do respondents answer differently in web survey than in face-to-face interview: field work experiment from the European Health Interview Survey (EHIS). Accessed 30 May 2019

  22. Kroenke K, Spitzer RL, Williams JB (2001) The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med 16:606–613.

  23. Krosnick JA (1991) Response strategies for coping with the cognitive demands of attitude measures in surveys. Appl Cogn Psychol 5:213–236.

  24. Meltzer H (2003) Development of a common instrument for mental health. In: Nosikov A, Gudex C (eds) EUROHIS: developing common instruments for health surveys. IOS Press, Amsterdam, pp 35–60

  25. Oehlert GW (1992) A note on the Delta method. Am Stat 46:27–29.

  26. Santourian A, Kitromilidou S (2018) Quality report of the second wave of the European Health Interview Survey. Publication Office of the European Union. Accessed 30 May 2019

  27. Schnell R, Noack M, Torregroza S (2017) Differences in general health of internet users and non-users and implications for the use of web surveys. Surv Res Methods 11:105–123.

  28. Schwarz N, Hippler H-J (1991) Response alternatives: the impact of their choice and presentation order. In: Biemer PP, Groves RM, Lyberg LE, Mathiowetz NA, Sudman S (eds) Measurement errors in surveys. Wiley, New York, pp 41–56

  29. Struminskaya B, de Leeuw ED, Kaczmirek L (2015) Mode system effects in an online panel study: comparing a probability-based online panel with two face-to-face reference surveys. Methods Data Anal 9:3–56.

  30. Tagseth M, Sund ER, Hallman GT et al (2019) May telephone surveys provide reliable public health surveillance data for municipalities? Mode effects differ between categories of questions. The HUNT Study. Norway. Norsk Epidemiologi 28:105–116.

  31. Tipping S, Hope S, Pickering K et al (2010) The effect of mode and context on survey results: analysis of data from the Health Survey for England 2006 and the Boost Survey for London. BMC Med Res Methodol 10:84.

  32. Tourangeau R, Conrad FG, Couper MP (2013) The science of web surveys. Oxford University Press, New York

  33. Tourangeau R, Rips LJ, Rasinski K (2000) The psychology of survey response. Cambridge University Press, Cambridge

  34. Vergnaud AC, Touvier M, Méjean C et al (2011) Agreement between web-based and paper versions of a socio-demographic questionnaire in the NutriNet-Santé study. Int J Public Health 56:407–417.

  35. Verschuuren M, Gissler M, Kilpeläinen K et al (2013) Public health indicators for the EU: the joint action for ECHIM (European Community Health Indicators & Monitoring). Arch Public Health 71:12.

  36. Ware JE, Sherbourne CD (1992) The MOS 36-item short-form health survey (SF-36): I. Conceptual framework and item selection. Med care 30:473–483

  37. Washington group (2012) Washington group on disability statistics (WG), Budapest Initiative (BI) & United Nations Economic & Social Commission for Asia & the Pacific (UNESCAP). Development of disability measures for surveys: the extended set on functioning. Accessed 30 May 2019

  38. Weisband S, Kiesler S (1996) Self disclosure on computer forms: meta-analysis and implications. In: Bilger R, Guest S, Tauber MJ (eds) CHI‘96 Electronic Proceedings. Association for Computing Machinery Inc, New York, pp 1–14

Download references


We wish to thank the EHISWEB and BHIS2018 study participants.


This research project was funded by the Belgian Science Policy (Belspo).

Author information

Correspondence to Elise Braekman.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

All procedures performed in these studies were in accordance with the ethical standards of the Ethics Committee of the University Hospital of Ghent.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Braekman, E., Charafeddine, R., Demarest, S. et al. Comparing web-based versus face-to-face and paper-and-pencil questionnaire data collected through two Belgian health surveys. Int J Public Health 65, 5–16 (2020).

Download citation


  • Health surveys
  • Data collection
  • Web-based surveys
  • Face-to-face surveys
  • Mode systems
  • Data comparability