Linking PISA Competencies over Three Cycles – Results from Germany

Abstract

Since the publication of the PISA 2006 study results the question of reporting trends over the PISA cycles has received a lot of interest. This chapter discusses the possibilities and limitations of trend analyses based on data from this international comparative study and using complex test designs. The chapter succeeds trend analyses which were carried out with the German data from the first three PISA studies in 2000, 2003 and 2006 (Carstensen CH, Prenzel M, Baumert J, Trendanalysen in PISA: Wie haben sich die Kompetenzen in Deutschland zwischen PISA 2000 und PISA 2006 entwickelt? [Trend analyses in PISA: how did competencies in Germany develop between PISA 2000 and PISA 2006?] Zeitschrift für Erziehungswissenschaften, Sonderheft 10:11–34, 2008; Prenzel M, Artelt C, Baumert J, Blum W, Hammann M, Klieme E et al (eds), PISA 2006. Die Ergebnisse der dritten internationalen Vergleichsstudie [PISA 2006. Results of the third international comparison]. Waxmann, Münster, 2007).

The choice of a scaling and trend analysis model depends on the focus of the analysis and on the assessment design. With respect to international comparisons, very strict assumptions on the uni-dimensionality of the test instruments used have to be made to allow for trend analyses. What if these conditions are not met across all participating countries for all assessment cycles? This paper presents an alternative model for trend analyses, assuming uni-dimensionality only within a particular country but not across all participating countries. Trend results with this model can only be interpreted within the particular country and are not intended for use in international comparisons.

To establish the validity of the presented trend model, an empirical analysis of the different tests and subscales used in different assessment cycles was performed. As far as different versions of the instruments were administered within cycles, the correlations of these test forms give an empirical indication of the uni-dimensionality of the underlying constructs. Monte Carlo simulations were performed to analyse whether the correlations of these test forms indicate a uni-dimensional construct being measured over time. Having analyzed the correlations of the tests, a fit analysis at the item level followed. Further assumptions refer to the stability of item difficulties over time. This is addressed by estimating item by time interaction parameters, allowing for a descriptive analysis of items changing their difficulty over time and a model fit comparison to check whether item drift has an impact on their difficulties.

Results show that trends might be reported for the German data, using the short test for reading and using all pair wise link items in Mathematics and Science. In the conclusion, the results and some implications for the design of future PISA assessments will be discussed.