The Methodological Framework of Evaluation

Part of the ZEW Economic Studies book series (ZEW, volume 36)


Propensity Score Methodological Framework Common Support Unemployment Spell Matching Estimator 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 2.
    See Heckman, LaLonde, and Smith (1999) for an overview on these approaches.Google Scholar
  2. 3.
    The model has been extended for the case of multiple treatments by Lechner (2001) and Imbens (2000).Google Scholar
  3. 4.
    Alternatively, in the case of J mutually exclusive treatments (e.g., for the case of evaluating different ALMP programmes), D could be an indicator for the J + 1 possible states the individual faces. D could also be ℝ+:= [0,∞), representing a continuum of doses of some medication, for example (see Abbring, 2003).Google Scholar
  4. 5.
    It should be noted that since job creation schemes have been used to a large extent especially in East Germany, assuming no spill-over effects on non-participants may be questionable. Thus, microeconometric evaluation can only analyse partial equilibrium effects of the programmes. Further macroeconometric analyses of programme effects are necessary for a full evaluation, see, e.g., Hujer and Zeiss (2005).Google Scholar
  5. 6.
    Heckman, LaLonde, and Smith (1999) discuss further parameters that may be of interest: for example, the average effect of treatment (ATE) defined as: \( \Delta ^{ATE} = E\left( \Delta \right) = E\left( {Y^1 - Y^0 } \right) = E\left( {Y^1 } \right) - E\left( {Y^0 } \right). \) The ATE computes the difference of the expected outcomes after participation and nonparticipation. It answers the question what the impact of treatment would be if individuals are randomly assigned to treatment. However, for policy implications it is only of minor relevance as persons are included for whom the programme was never intended (Heckman, 1997). Further parameters of interest may be the proportion of people taking the programme who benefit from it, or the increase in the proportion of outcomes above a certain threshold outcome value due to a policy. As the empirical analyses are based on the ATT, I will concentrate the discussion on this parameter.Google Scholar
  6. 9.
    See Heckman, Ichimura, and Todd (1997), for a description of how randomisation solves the evaluation problem.Google Scholar
  7. 10.
    The interested reader is referred to the paper of LaLonde (1986) and the responses and extensions by Dehejia and Wahba (1999; 2002) and Smith and Todd (2005a).Google Scholar
  8. 11.
    This is the true econometric selection bias resulting from’ selection on unobservables’ (Blundell and Costa Dias, 2000).Google Scholar
  9. 13.
    See, e.g., for job creation schemes in Germany Bergemann and Schultz (2000) and Bergemann et al. (2000) who analyse the occurrence of Ashenfelter’s Dip.Google Scholar
  10. 14.
    See Blundell and Costa Dias (2002) for a further discussion.Google Scholar
  11. 15.
    General surveys are provided by Angrist and Krueger (1999), Heckman, LaLonde, and Smith (1999) and Blundell and Costa Dias (2002). For instrumental variable methods see also Imbens and Angrist (1994) and Angrist, Imbens, and Rubin (1996). Blundell and Costa Dias (2000) provide further information on the Heckman selection estimator in the evaluation context. A good example for an application of the regression discontinuity estimator is given by Angrist and Lavy (1999).Google Scholar
  12. 16.
    See, e.g., Rubin (1974; 1977; 1979; 1991), Rosenbaum and Rubin (1983b; 1985), and the overview by Rosenbaum (2002). However, the idea of matching is not new. Heckman, Ichimura, Smith, and Todd (1998) note that the method of matching was first used by Fechner (1860).Google Scholar
  13. 17.
    One example is the so-called conditional DiD suggested by Heckman, Ichimura, and Todd (1997) that combines matching and the DiD estimator.Google Scholar
  14. 20.
    The idea of conditioning onX to eliminate selection bias may also justify linear regression. However, two drawbacks of this method relative to matching have to be noted. First, matching is a non-parametric method and therefore does not require any parametric assumption, like the linearity implicit in linear regression. Second, matching emphasises the common support problem, whereas in analyses that estimate impacts simply by running regressions on X, the issue is rarely even investigated (Smith, 2000a).Google Scholar
  15. 22.
    Rosenbaum (1986) uses a linear probability model (LPM) but states that logistic regression models are preferable for the well-known shortcomings of the LPM, especially the unlikeliness of the functional form when the response variable is highly skewed as well as predictions that are outside the [0, 1] bound of probabilities.Google Scholar
  16. 24.
    Stratification matching is also termed interval matching, blocking or subclassification (Rosenbaum and Rubin, 1983b).Google Scholar
  17. 25.
    With an sufficiently large M, stratification matching is close to the weighting estimator. See Imbens (2004) for further details.Google Scholar
  18. 27.
    There is a growing literature applying this method to evaluate the effects of ALMP programmes. For example, Hujer, Thomsen, and Zeiss (2006b) apply a MMPH model to estimate the effects of vocational training programmes in Eastern Germany. Hujer, Thomsen, and Zeiss (2006a) analyse the effects of short-term training measures on the duration of unemployment in West Germany. Similar approaches have been applied in a number of studies for other countries, like Lalive, van Ours, and Zweimüller (2002) for Switzerland, Richardson and van den Berg (2001) for Sweden, Bonnal, Fougere, and Serandon (1997) for France, and van Ours (2001) for Slovakia. The interested reader is also referred to the comprehensive survey on the methodology by van den Berg (2001).Google Scholar
  19. 28.
    Matching or sampling from a risk set for the time up to an outcome event has been discussed already by Prentice and Breslow (1978), Oakes (1981), and Prentice (1986), but not for the time up to treatment (see Li et al., 2001).Google Scholar
  20. 29.
    See also the discussion in Fredriksson and Johansson (2004).Google Scholar
  21. 30.
    See, for example, the discussion in Smith (1997).Google Scholar
  22. 31.
    For the multiple treatment case, the choice of the model becomes more important. For example, whereas the multinomial logit model requires strong assumptions (independence of irrelevant alternatives), the more flexible multinomial probit is computationally burdensome. See the discussion in Lechner (2001).Google Scholar
  23. 32.
    Some variables have reached a notable importance in the applied literature. From the accumulated evidence of the evaluation of labour market policy programmes, in particular the labour market history of the individual (see, e.g., Heckman, Ichimura, and Todd, 1997) and the regional labour market environment (see, e.g., Heckman, Ichimura, Smith, and Todd, 1998) are especially important. Furthermore, economic theory suggests to control for qualification level and work tenure as proxies for the reservation wage.Google Scholar
  24. 33.
    See, e.g., the study of Lechner (2002) who analyses the effects of ALMP programmes in Switzerland and complements the propensity score by sex, duration of unemployment and native language in the matching procedure.Google Scholar
  25. 34.
    This approach has been used, e.g., by Heckman, Ichimura, and Todd (1997) and Heckman, Ichimura, Smith, and Todd (1998).Google Scholar
  26. 35.
    Alternatively, Heckman, Ichimura, and Todd (1998) and Smith and Todd (2005a) suggest to use a ‘trimming’ procedure.Google Scholar
  27. 37.
    The data used in the analysis in chapter 6 is the same as in Caliendo et al. (2006a).Google Scholar
  28. 38.
    The version of Rosenbaum and Rubin (1985) is a bit simpler: \( st\_dif = 100 \cdot \left( {\bar X_1 - \bar X_{0M} } \right)/\left[ {\left( {Var_1 + Var_0 } \right)/2} \right]^{1/2} , \) where \( \bar X_1 \) and \( \bar X_{0M} \) are the sample means in the treated and matched comparison group, and Var1 and Var0 are the sample variances in the treated group and the comparison reservoir.Google Scholar
  29. 39.
    An approach that considers the possible influence of preceding treatments on the selection into further treatments and the resulting impacts based on the matching estimator is suggested by Lechner and Miquel (2002). Lechner (2004) proposes an estimation procedure and provides an application to Swiss data.Google Scholar

Copyright information

© Physica-Verlag Heidelberg 2007

Personalised recommendations