Non-Response and Measurement Error

Abstract

This chapter deals with some aspects of two sources of systematic error in surveys: non-response and measurement error. The quality of the obtained response is discussed first with a focus on non-response bias estimation and non-response bias reduction. Measurement error is studied by evaluating the quality of registered responses through question wording, order, and response scale effects. The different approaches to measurement error are discussed. Practical examples of dealing with bias and measurement error are offered. These include the evaluation of non-(response) rates and response enhancement strategies, comparing cooperative and reluctant respondents on non-response issues based on the analysis of European Social Survey. Concerning measurement error, the split ballot approach and multitrait multimethod are extensively discussed including theoretical concepts and methods/models, and an example of acquiescence when a balanced set of items is available. The chapter also presents some debates concerning theoretical construct and model developments in the respective fields, and emphasizes that both errors are strongly related. This means response distributions, correlations, and regression parameters can be seriously affected.

Appendix

The MTMM true-score model for two correlated variables measured with two different methods (response scales), and with unique components and random error

• y_i1 and y_i2: observed indicator of i measured with methods 1 and 2;

• F₁ and F₂: state of latent variables 1 and 2 which are correlated;

• M₁ and M₂: latent method factors 1 and 2 that are assumed un-correlated;

• T_i1 and T_i2: true score of variable Fi repeatedly measured with methods 1 and 2;

• e_i1 and e_i2: random measurement error for item i measured with methods 1 and 2;

• r_i1 and r_i2: the reliability coefficients for item i measured with methods 1 and 2; the square of these are estimates of the test–retest reliability in case of repeated identical methods)

• v_i1 and v_i2: the true score validity coefficients for item i measured with methods 1 and 2; v_i1² (or v_i2²) is the variance in T_i1 (or the T_i2) explained by the variable F _i that one intends to measure;

• m_i1 and m_i2: the method effect on the true score of methods 1 and 2; m_i1² (or m_i21²) is together with u_i1 (or u_i2) the part in T_i1 (or T_i2) not explained by the intended variable, and thus invalidity;

• u_i1 and u_i2: the residual variance of the true score of item i measured with methods 1 and 2;

• ρ(F₁F₂): the correlation between the two latent (intended) variables

$$ y_{ij} = \, r_{ij} T_{ij} + \, e_{ij} $$

(10.1)

$$ T_{ij} = \, v_{ij} F_{i} + \, m_{ij} M_{j} + \, u_{ij} $$

(10.2)

This model is a simplified version of the true score MTMM model used. It is in reality not possible to estimate a ‘2(traits) × 2(methods)’ model. One needs at least three traits and three methods. But even then is it not possible to estimate the parameters, unless certain assumptions are made. Most simple (and acceptable) is to assume that the unique residual variances of the true scores (u _ij ) are zero (Saris 1990b, p. 119; Scherpenzeel and Saris 1997, pp. 344–347).

In line with Eqs (10.1) and (10.2), and assuming zero unique variance u _ij, the true score model is written as:

$$ y_{ij} = \, r_{ij}\times v_{ij} F_{i} + \, r_{ij}\times m_{ij} M_{j} + \, e_{i} $$

(10.3)

Path analysis suggests that the observed correlation between the measures of two traits, both measured with method M ₁, is (Scherpenzeel and Saris 1997, pp. 372–373):

$$ {\text{Corr}}\left( {y_{11} ,y_{2,1} } \right) \, = v_{11}\times r_{11}\times \rho \left( {F_{1} ,F_{2} } \right)\times v_{2}\times r_{21} + r_{11}\times m_{1 \, 1}\times m_{21}\times r_{21} $$

(10.4)

From which one can derive:

$$ {\text{r}}\left( {F_{1} F_{2} } \right) = \, [{\text{cor}}\left( {y_{11} y_{21} } \right) \, - \, (r_{11}\times m_{1 \, 1 }\times m_{21 }\times r_{21 } )]/(v_{11 }\times r_{11 }\times v_{2 }\times r_{21} ) $$

(10.5)

This expression is useful for adjusting observed correlations between variables when one can rely on validity, reliability, and invalidity estimates (e.g. based on meta-analysis of MTMM data).