Journal of Well-Being Assessment

, Volume 2, Issue 1, pp 41–55 | Cite as

Incentivised Online Panel Recruitment and Subjective Wellbeing: Caveat Emptor

  • Melissa K. WeinbergEmail author
  • Robert A. Cummins
  • David A. WebbEmail author
  • Wencke Gwozdz
Original Research


It is generally assumed that if a sample represents its broader population on key demographic variables, the data it yields will also be representative. Here we present evidence to suggest that this is not necessarily so when subjective wellbeing is measured from participants recruited through online panels. Using data from six countries: Australia, Germany, Sweden, the Netherlands, UK and USA, we reveal significant differences in subjective wellbeing between online panel data and nationally representative data, even though both are demographically comparable. These findings indicate that the online panels comprised an abnormally high proportion of people with low subjective wellbeing, thus rendering their data non-representative. Given the widespread use of online panels to collect data in the modern era, we issue a caveat emptor.


Online panels Data collection Subjective wellbeing Sampling 


Compliance with Ethical Standards

Conflict of Interest Statement

On behalf of all authors, the corresponding author states that there is no conflict of interest.


  1. Andrews, F. M., & Withey, S. B. (1976). Social indicators of well-being: American's perceptions of life quality. New York: Plenum Press.CrossRefGoogle Scholar
  2. Australian Bureau of Statistics (2016). Household use of information technology, Australia, 2014–2015, Category no. 8146.0. Canberra, Government of Australia.Google Scholar
  3. Baker, R., Blumberg, S. J., Brick, J. M., Couper, M. P., Courtright, M., Dennis, J. M., Dillman, D., Frankel, M. R., Garland, P., Groves, R. M., Kennedy, C., Krosnick, J., Lavrakas, P. J., Lee, S., Link, M., Piekarski, L., Rao, K., Thomas, R. K., & Zahs, D. (2010). Research synthesis: AAPOR report on online panels. Public Opinion Quarterly, 74(4), 711–781.CrossRefGoogle Scholar
  4. Barber, T., Chilvers, D., & Kaul, S. (2013). Moving an established survey online - or not? International Journal of Market Research, 55(2), 2–11.CrossRefGoogle Scholar
  5. Bethlehem, J., & Stoop, I. (2007). Online panels - a paradigm theft. In The challenges of a changing world. Proceedings of the fifth international conference of the Association for Survey Computing, September, 113–137.Google Scholar
  6. Blanchflower, D. G., & Oswald, A. J. (2004). Well-being over time in Britain and the USA. Journal of Public Economics, 88, 1359–1386.
  7. Capic, T., Li, N., & Cummins, R. A. (2017). Set-points for subjective wellbeing: a replication and extension. Social Indicators Research, (in press).Google Scholar
  8. Couper, M. P., Kapteyn, A., Schonlau, M., & Winter, J. (2007). Noncoverage and nonresponse in an Internet survey. Social Science Research, 36, 131–148.CrossRefGoogle Scholar
  9. Cummins, R. A. (1995). On the trail of the gold standard for life satisfaction. Social Indicators Research, 35(2), 179–200. Scholar
  10. Cummins, R. A. (1998). The second approximation to an international standard of life satisfaction. Social Indicators Research, 43(3), 307–334. Scholar
  11. Cummins, R. A. (2010). Subjective wellbeing, homeostatically protected mood and depression: a synthesis. Journal of Happiness Studies, 11, 1–17. Scholar
  12. Cummins, R. A. (2016). The theory of subjective wellbeing homeostasis: A contribution to understanding life quality. In F. Maggino (Ed.), A life devoted to quality of life – Festschrift in honor of Alex C. Michalos (Vol. 60, pp. 61–79). Dordrecht: Springer.CrossRefGoogle Scholar
  13. Cummins, R. A., & Weinberg, M. K. (2015). Multi-item measurement of subjective wellbeing: Subjective approaches. In W. Glatzer, L. Camfield, V. Møller, & M. Rojas (Eds.), Global handbook of quality of life: exploration of well-being of nations and continents (pp. 239–268). Dordrecht: Springer.CrossRefGoogle Scholar
  14. Cummins, R. A., Woerner, J., Weinberg, M., Collard, J., Hartley-Clark, L., & Horfiniak, K. (2013). Australian Unity Wellbeing Index: Report 30.0 - The wellbeing of Australians: Social media, personal achievement, and work. Melbourne: Australian Centre on Quality of Life, School of Psychology, Deakin
  15. De Leeuw, E. D. (2005). To mix or not to mix data collection modes in surveys. Journal of Official Statistics, 21, 233–255.Google Scholar
  16. De Leeuw, E. D., Mellenbergh, G. J., & Hox, J. J. (1996). The influence of data collection method on structural models: a comparison of a mail, a telephone, and a face-to-face survey. Sociology Methods Research, 24, 442–472.CrossRefGoogle Scholar
  17. DeSimone, J. A., Harms, P. D., & DeSimone, A. J. (2015). Best practice recommendations for data screening. Journal of Organizational Behavior, 36, 171–181.CrossRefGoogle Scholar
  18. Douglas, S. P., & Craig, C. S. (1983). International Marketing Research. Englewood Cliffs: Prentice-Hall.Google Scholar
  19. European Social Survey. (2012). ESS round 6 (2012) project instructions (PAPI). London: Centre for Comparative Social Surveys, City University London Last accessed 13 April 2015.Google Scholar
  20. Gallup. (2009). Gallup World Poll. Washington, DC: Gallup.Google Scholar
  21. Helliwell, J., Layard, R., & Sachs, J. (2017). World Happiness Report 2017. New York: Sustainable Development Solutions Network.Google Scholar
  22. International Wellbeing Group. (2013). Personal Wellbeing Index Manual. Melbourne: Deakin University Scholar
  23. Maggino, F. (2016) Challenges, needs and risks in defining wellbeing indicators. In F. Maggino (Ed.), A life devoted to quality of life. Social indicators research series, vol 60. Cham: Springer.
  24. Meade, A.W., & Craig, S.B. (2011). Identifying careless responses in survey data. Paper presented at the 26th Annual Meeting of Society for Industrial and Organizational Psychology, Chicago, IL.Google Scholar
  25. OECD. (2013). OECD guidelines on measuring subjective well-being. Paris: OECD Publishing. Scholar
  26. OECD. (2017). How’s life? 2017: measuring well-being. Paris: OECD Publishing. Scholar
  27. Prince, K. R., Litovsky, A. R., & Friedman-Wheeler, D. G. (2012). Internet-mediated research: beware of bots. The Behavior Therapist, 35, 85–88.Google Scholar
  28. Pruchno, R. A., & Hayden, J. M. (2000). Interview modality: effects on costs and data quality in a sample of older women. Journal of Aging & Health, 12, 3–24.CrossRefGoogle Scholar
  29. Shawver, Z., Griffith, J. D., Adams, L. T., Evans, J. V., Benchoff, B., & Sargent, R. (2016). An examination of the WHOQOL-BREF using four popular data collection methods. Computers in Human Behavior, 55, 446–454.CrossRefGoogle Scholar
  30. Suh, E., Diener, E., & Fujita, F. (1996). Events and subjective well-being: only recent events matter. Journal of Personality and Social Psychology, 70, 1091–1102.CrossRefGoogle Scholar
  31. The World Bank (2016). Accessed online January 25 2017 via
  32. Tomyn, A. J., Weinberg, M. K., & Cummins, R. A. (2015). Intervention efficacy among 'at-risk' adolescents: a test of subjective wellbeing homeostasis theory. Social Indicators Research, 120, 883–895.
  33. Tourangeau, R., & Yan, T. (2007). Sensitive questions in surveys. Psychological Bulletin, 133, 859–883.CrossRefGoogle Scholar
  34. WHO Regional Office for Europe. (2012). Measurement of and target-setting for well-being Second meeting of the expert group, Paris, 25–26 June 2012. Copenhagen: WHO Regional Office for Europe.Google Scholar
  35. WHOQOL Group. (1998). Development of the World Health Organization WHOQOL-BREF Quality of Life Assessment. Psychological Medicine, 28, 551–558.CrossRefGoogle Scholar
  36. Williams, J. (2012). Survey methods in an age of austerity: driving value in survey design. International Journal of Market Research, 54(1), 35–48.CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.School of Psychology, Faculty of HealthDeakin UniversityMelbourneAustralia
  2. 2.University of Western AustraliaCrawleyAustralia
  3. 3.Copenhagen Business SchoolFrederiksbergDenmark
  4. 4.Justus-Liebig-Universität GießenGießenGermany

Personalised recommendations