# The Methodological Framework of Evaluation

Chapter

## Keywords

Propensity Score Methodological Framework Common Support Unemployment Spell Matching Estimator
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

## Preview

Unable to display preview. Download preview PDF.

## References

- 2.See Heckman, LaLonde, and Smith (1999) for an overview on these approaches.Google Scholar
- 3.The model has been extended for the case of multiple treatments by Lechner (2001) and Imbens (2000).Google Scholar
- 4.Alternatively, in the case of
*J*mutually exclusive treatments (e.g., for the case of evaluating different ALMP programmes),*D*could be an indicator for the*J*+ 1 possible states the individual faces.*D*could also be ℝ_{+}:= [0,∞), representing a continuum of doses of some medication, for example (see Abbring, 2003).Google Scholar - 5.It should be noted that since job creation schemes have been used to a large extent especially in East Germany, assuming no spill-over effects on non-participants may be questionable. Thus, microeconometric evaluation can only analyse partial equilibrium effects of the programmes. Further macroeconometric analyses of programme effects are necessary for a full evaluation, see, e.g., Hujer and Zeiss (2005).Google Scholar
- 6.Heckman, LaLonde, and Smith (1999) discuss further parameters that may be of interest: for example, the average effect of treatment (ATE) defined as: \( \Delta ^{ATE} = E\left( \Delta \right) = E\left( {Y^1 - Y^0 } \right) = E\left( {Y^1 } \right) - E\left( {Y^0 } \right). \) The ATE computes the difference of the expected outcomes after participation and nonparticipation. It answers the question what the impact of treatment would be if individuals are randomly assigned to treatment. However, for policy implications it is only of minor relevance as persons are included for whom the programme was never intended (Heckman, 1997). Further parameters of interest may be the proportion of people taking the programme who benefit from it, or the increase in the proportion of outcomes above a certain threshold outcome value due to a policy. As the empirical analyses are based on the ATT, I will concentrate the discussion on this parameter.Google Scholar
- 9.See Heckman, Ichimura, and Todd (1997), for a description of how randomisation solves the evaluation problem.Google Scholar
- 10.The interested reader is referred to the paper of LaLonde (1986) and the responses and extensions by Dehejia and Wahba (1999; 2002) and Smith and Todd (2005a).Google Scholar
- 11.This is the true econometric selection bias resulting from’ selection on unobservables’ (Blundell and Costa Dias, 2000).Google Scholar
- 13.See, e.g., for job creation schemes in Germany Bergemann and Schultz (2000) and Bergemann et al. (2000) who analyse the occurrence of Ashenfelter’s Dip.Google Scholar
- 14.See Blundell and Costa Dias (2002) for a further discussion.Google Scholar
- 15.General surveys are provided by Angrist and Krueger (1999), Heckman, LaLonde, and Smith (1999) and Blundell and Costa Dias (2002). For instrumental variable methods see also Imbens and Angrist (1994) and Angrist, Imbens, and Rubin (1996). Blundell and Costa Dias (2000) provide further information on the Heckman selection estimator in the evaluation context. A good example for an application of the regression discontinuity estimator is given by Angrist and Lavy (1999).Google Scholar
- 16.See, e.g., Rubin (1974; 1977; 1979; 1991), Rosenbaum and Rubin (1983b; 1985), and the overview by Rosenbaum (2002). However, the idea of matching is not new. Heckman, Ichimura, Smith, and Todd (1998) note that the method of matching was first used by Fechner (1860).Google Scholar
- 17.One example is the so-called conditional DiD suggested by Heckman, Ichimura, and Todd (1997) that combines matching and the DiD estimator.Google Scholar
- 20.The idea of conditioning on
*X*to eliminate selection bias may also justify linear regression. However, two drawbacks of this method relative to matching have to be noted. First, matching is a non-parametric method and therefore does not require any parametric assumption, like the linearity implicit in linear regression. Second, matching emphasises the common support problem, whereas in analyses that estimate impacts simply by running regressions on*X*, the issue is rarely even investigated (Smith, 2000a).Google Scholar - 22.Rosenbaum (1986) uses a linear probability model (LPM) but states that logistic regression models are preferable for the well-known shortcomings of the LPM, especially the unlikeliness of the functional form when the response variable is highly skewed as well as predictions that are outside the [0, 1] bound of probabilities.Google Scholar
- 24.Stratification matching is also termed interval matching, blocking or subclassification (Rosenbaum and Rubin, 1983b).Google Scholar
- 25.With an sufficiently large
*M*, stratification matching is close to the weighting estimator. See Imbens (2004) for further details.Google Scholar - 27.There is a growing literature applying this method to evaluate the effects of ALMP programmes. For example, Hujer, Thomsen, and Zeiss (2006b) apply a MMPH model to estimate the effects of vocational training programmes in Eastern Germany. Hujer, Thomsen, and Zeiss (2006a) analyse the effects of short-term training measures on the duration of unemployment in West Germany. Similar approaches have been applied in a number of studies for other countries, like Lalive, van Ours, and Zweimüller (2002) for Switzerland, Richardson and van den Berg (2001) for Sweden, Bonnal, Fougere, and Serandon (1997) for France, and van Ours (2001) for Slovakia. The interested reader is also referred to the comprehensive survey on the methodology by van den Berg (2001).Google Scholar
- 28.Matching or sampling from a risk set for the time up to an outcome event has been discussed already by Prentice and Breslow (1978), Oakes (1981), and Prentice (1986), but not for the time up to treatment (see Li et al., 2001).Google Scholar
- 29.See also the discussion in Fredriksson and Johansson (2004).Google Scholar
- 30.See, for example, the discussion in Smith (1997).Google Scholar
- 31.For the multiple treatment case, the choice of the model becomes more important. For example, whereas the multinomial logit model requires strong assumptions (independence of irrelevant alternatives), the more flexible multinomial probit is computationally burdensome. See the discussion in Lechner (2001).Google Scholar
- 32.Some variables have reached a notable importance in the applied literature. From the accumulated evidence of the evaluation of labour market policy programmes, in particular the labour market history of the individual (see, e.g., Heckman, Ichimura, and Todd, 1997) and the regional labour market environment (see, e.g., Heckman, Ichimura, Smith, and Todd, 1998) are especially important. Furthermore, economic theory suggests to control for qualification level and work tenure as proxies for the reservation wage.Google Scholar
- 33.See, e.g., the study of Lechner (2002) who analyses the effects of ALMP programmes in Switzerland and complements the propensity score by sex, duration of unemployment and native language in the matching procedure.Google Scholar
- 34.This approach has been used, e.g., by Heckman, Ichimura, and Todd (1997) and Heckman, Ichimura, Smith, and Todd (1998).Google Scholar
- 35.Alternatively, Heckman, Ichimura, and Todd (1998) and Smith and Todd (2005a) suggest to use a ‘trimming’ procedure.Google Scholar
- 37.The data used in the analysis in chapter 6 is the same as in Caliendo et al. (2006a).Google Scholar
- 38.The version of Rosenbaum and Rubin (1985) is a bit simpler: \( st\_dif = 100 \cdot \left( {\bar X_1 - \bar X_{0M} } \right)/\left[ {\left( {Var_1 + Var_0 } \right)/2} \right]^{1/2} , \) where \( \bar X_1 \) and \( \bar X_{0M} \) are the sample means in the treated and matched comparison group, and Var
_{1}and Var_{0}are the sample variances in the treated group and the comparison reservoir.Google Scholar - 39.An approach that considers the possible influence of preceding treatments on the selection into further treatments and the resulting impacts based on the matching estimator is suggested by Lechner and Miquel (2002). Lechner (2004) proposes an estimation procedure and provides an application to Swiss data.Google Scholar

## Copyright information

© Physica-Verlag Heidelberg 2007