Measuring Participation or Participating in Measurement? The Cautionary Tale of an Accidental Experiment in Survey Accuracy
While survey results are always subject to measurement error, it is generally assumed that surveys of cultural participation are no less accurate than surveys in other areas of social science. The present chapter casts doubt on this assumption via a cautionary tale of events that befell an established national survey in Ireland. An organisational change led the survey to be conducted via the same method but using a different set of interviewers. The result was a surprising and dramatic increase in the headline figures, which related to participation in sport, both active and social. Subsequent data pointed to a systematic relationship between the decision to participate in the different surveys and in the activity being measured. The implication is that surveys of cultural participation may be subject to a specific form of selection bias. Furthermore, the effect size reported here suggests that this bias may be discomfortingly large.
KeywordsMeasuring participation Survey design Sampling problems
For assistance related to the material in this chapter, I thank the Irish Sports Council, Dorothy Watson, Peter Smyth and John O’Hagan. I also thank the patient staff of Companies A and B, who did their best to get to the bottom of the difficult issues raised.
- Carter, P. (2005). Review of national sport effort and resources. Report for HM Treasury and The Department of Culture, Media and Sport, London, UK.Google Scholar
- Central Statistics Office. (2007). Quarterly National Household Survey: Sport and physical exercise. Cork: Central Statistics Office.Google Scholar
- Deroin, V. (2011, December). European statistical works on culture (Culture Études, 2011–08).Google Scholar
- ESSnet Culture. (2012). European statistical network on culture: Final report. Luxembourg: Eurostat.Google Scholar
- European Commission. (2010). Sport and physical activity. Special Eurobarometer 334. Brussels: European Commission.Google Scholar
- Eurostat. (2011). Cultural statistics. Luxembourg: Publications Office of the European Union.Google Scholar
- Fahey, T., Layte, R., & Gannon, B. (2004). Sports participation and health among adults in Ireland. Dublin: Economic and Social Research Institute.Google Scholar
- Lunn, P., & Layte, R. (2009). The Irish Sports Monitor: Second annual report, 2008. Dublin: Irish Sports Council/Economic and Social Research Institute.Google Scholar
- Lunn, P., Layte, R., & Watson, D. (2009). The Irish Sports Monitor: First annual report, 2007. Dublin: Irish Sports Council/Economic and Social Research Institute.Google Scholar
- Morrone, A. (2006). Guidelines for measuring cultural participation. Montreal: UNESCO Institute for Statistics.Google Scholar
- UNESCO. (2012). Measuring cultural participation. 2009 Framework for Cultural Statistics Handbook No. 2. Montreal: UNESCO Institute for Statistics.Google Scholar