Advertisement

Multilevel Modeling with HLM: Taking a Second Look at PISA

  • John O. AndersonEmail author
  • Todd MilfordEmail author
  • Shelley P. RossEmail author
Chapter

The purpose of this book is to provide a synthesis of thought and practice in research in literacy and science education intended to lead to evidence-based results and generalizations that will serve as a foundation for public policy and informed curriculum, teaching, and assessment practices in education. The Gold Standard of educational research funding in the United States can be viewed as a response to the general dissatisfaction with the utility of educational research; this federal mandate fosters a shift of educational research toward positivist empirical research approaches, such as random controlled trials (RCTs). There is an expectation of greater generalization and policy relevance as Gold Standard research is conducted and reported. It should be noted this dissatisfaction is not confined to the United States. An international response to this dissatisfaction with educational research—systematic reviews of educational research such as the Campbell Collaboration (Campbell Collaboration, n.d.), the What Works Clearinghouse of the US Office of Education (US Institute of Education Sciences, n.d.), and the Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre) in the United Kingdom (EPPI-Centre, n.d.)— has also identified the general dearth of rigorous empirical research that can support meaningful generalization of research findings.

The deliberations about qualitative—quantitative approaches to educational research over the last 25 years have established parallels for quantitative data considerations (reliability, validity, significance, objectivity) that consider dependability, credibility, believability, and confirmability of information (Howe & Eisenhart, 1990; Husen, 1988; Lather, 1992; Phillips, 2005; Pring, 2000). The enactment of these considerations has produced a diverse collection of qualitative, quantitative, and mixed-methods research in literacy and science education. Inspection of these research studies during the 2nd Island Conference revealed concerns about the clarity of constructs involved in the research, measurements of these constructs, interpretative frameworks and scoring rubrics, data scales (nominal, ordinal, internal, ratio), and statistical analyses of these data. Procedural rigor, clarity, mining and secondary analysis of existing databases and information resources, generalization of research results, research ethics, and mixed-methods and other innovative approaches were identified as issues to enhance research quality (see Yore & Boscolo, Chap. 2).

Keywords

Intrinsic Motivation School Climate Mathematics Achievement Teacher Support Mathematics Score 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Anderson, J. O., Lin, H.-L., Treagust, D. F., Ross, S. P., & Yore, L. D. (2007). Using large-scale assessment datasets for research in science and mathematics education: Programme for International Student Assessment (PISA). International Journal of Science & Mathematics Education, 5(4), 591–614.CrossRefGoogle Scholar
  2. Anderson, J. O., Monseur, C., & Cartwright, F. (2006, April). Procedures and issues associated with the data and analysis of PISA 2003 thematic research. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.Google Scholar
  3. Beaton, A. E., Postlewaite, T. N., Ross, K. N., Spearritt, D., & Worl, R. M. (1999). The benefits and limitations of international educational achievements studies. Retrieved April 22, 2008, from http://unesdoc.unesco.org/images/0011/001176/117629e.pdf
  4. Brooks-Gunn, J., Phelps, E., & Elder, G. H. (1991). Studying lives through time: Secondary data analyses in developmental psychology. Developmental Psychology, 27(6), 899–910.CrossRefGoogle Scholar
  5. Byrne, B. M. (2001). Structural equation modeling with AMOS: Basic concepts, applications and programming. Mahwah, NJ: Lawrence Erlbaum.Google Scholar
  6. Campbell Collaboration (n.d.). Homepage. Retrieved October 3, 2007, from http://www.camp-bellcollaboration.org
  7. Dohn, N. B. (2007). Knowledge and skills for PISA — Assessing the assessment. Journal of Philosophy of Education, 41(1), 1–16.CrossRefGoogle Scholar
  8. Evidence for Policy and Practice Information and Co-ordinating Centre. (n.d.). Homepage. Retrieved June 20, 2008, from http://eppi.ioe.ac.uk/cms/
  9. George, R., & Kaplan, D. (1998). A structural model of parent and teacher influences on science attitudes of eighth graders: Evidence from NELS: 88. Science Education, 82(1), 93–109.CrossRefGoogle Scholar
  10. Goh, M. (2006). A multilevel analysis of mathematics literacy: The effects of intrinsic motivation, teacher support, and student-teacher relations. Unpublished master's thesis, University of Victoria, Victoria, British Columbia, Canada.Google Scholar
  11. Goldstein, H. (2004). International comparisons of student attainment: Some issues arising from the PISA study. Assessment in Education: Principles, Policy & Practice, 11(3), 319–330.CrossRefGoogle Scholar
  12. Gu, Z. (2006). Students' beliefs about themselves, learning environment at school, and achievement. Unpublished master's thesis, University of Victoria, Victoria, British Columbia, Canada.Google Scholar
  13. Hakim, C. (1982). Secondary analysis in social research. Boston: Allen & Unwin.Google Scholar
  14. Hofferth, S. L. (2005). Secondary data analysis in family research. Journal of Marriage & Family, 67(4), 891–907.CrossRefGoogle Scholar
  15. Howe, K. R., & Eisenhart, M. (1990). Standards for qualitative (and quantitative) research: A prolegomenon. Educational Researcher, 19(4), 2–9.Google Scholar
  16. Hsu, J. C. (2007). Comparing the relationships between mathematics achievement and student characteristics in Canada and Hong Kong through HLM. Unpublished master's thesis, University of Victoria, Victoria, British Columbia, Canada.Google Scholar
  17. Husen, T. (1988). Research paradigms in education. In J. P. Keeves (Ed.), Educational research, methodology and measurement: An international handbook (pp. 17–20). Toronto, Ontario, Canada: Pergamon Press.Google Scholar
  18. Hyman, H. (1972). Secondary analysis of sample surveys: Principles, procedures, and potentialities. Toronto, Ontario, Canada: Wiley & Sons.Google Scholar
  19. Kennedy, M. M. (1999). Infusing educational decision making with research. In G. J. Cizek (Ed.), Handbook of educational policy (pp. 54–80). San Diego, CA: Academic Press.Google Scholar
  20. Kiecolt, K. J., & Nathan, L. E. (1985). Secondary analysis of survey data. Newbury Park, CA: Sage.Google Scholar
  21. Kline, R. B. (1998). Principles and practice of structural equation modeling. New York: Guilford.Google Scholar
  22. Lather, P. (1992). Critical frames in educational research: Feminist and poststructural perspectives. Theory into Practice, 31(2), 87–99.CrossRefGoogle Scholar
  23. Lindblom, C. E. (1968). The policy-making process. Englewood Cliffs, NJ: Prentice Hall.Google Scholar
  24. Lindblom, C. E. (1992). Inquiry and change: The troubled attempt to understand and shape society. New Haven, CT & New York: Yale University Press & Russell Sage Foundation.Google Scholar
  25. Luke, D. A. (2004). Multilevel modeling. Thousand Oaks, CA: Sage.Google Scholar
  26. Marsh, H. W., & Yeung, A. S. (1998). Longitudinal structural equation models of academic self-concept and achievement: Gender differences in the development of math and English constructs. American Educational Research Journal, 35(4), 705–738.Google Scholar
  27. McQueen, J., & Mendelovits, J. (2003). PISA reading: Cultural equivalence in a cross-cultural study. Language Testing, 20(2), 208–224.CrossRefGoogle Scholar
  28. Norris, S. P., & Phillips, L. M. (2003). How literacy in its fundamental sense is central to scientific literacy. Science Education, 87(2), 224–240.CrossRefGoogle Scholar
  29. Organisation for Economic Co-operation and Development. (2001). Knowledge and skills for life: First report from the OECD programme for international student assessment. Paris: Author.Google Scholar
  30. Organisation for Economic Co-operation and Development. (2002). PISA 2000 technical report. Paris: Author.Google Scholar
  31. Organisation for Economic Co-operation and Development. (2003a). The PISA 2003 assessment framework — mathematics, reading, science and problem solving: Knowledge and skills. Paris: Author.Google Scholar
  32. Organisation for Economic Co-operation and Development. (2003b). PISA 2003 technical report. Paris: Author.Google Scholar
  33. Organisation for Economic Co-operation and Development. (2006). Assessing scientific, reading and mathematical literacy: A framework for PISA 2006. Paris: Author.Google Scholar
  34. Organisation for Economic Co-operation and Development. (2007). What PISA assesses. Paris: Author.Google Scholar
  35. Organisation for Economic Co-operation and Development. (n.d.). PISA Homepage. Retrieved June 30, 2008, from http://www.pisa.oecd.org/pages/0,2987,en_32252351_32235731_1_1_1_ 1_1,00.html
  36. Phillips, D. C. (2005). The contested nature of empirical educational research (and why philosophy of education offers little help). Journal of Philosophy of Education, 39(4), 577–597.CrossRefGoogle Scholar
  37. Prais, S. J. (2003). Cautions on OECD's recent educational survey (PISA). Oxford Review of Education, 29(2), 139–163.CrossRefGoogle Scholar
  38. Pring, R. (2000). The ‘false dualism’ of educational research. Journal of Philosophy of Education, 34(2), 247–260.CrossRefGoogle Scholar
  39. Ram, A. (2007). A multilevel analysis of mathematics literacy in Canada and Japan: The effects of sex differences, teacher support, and the school learning environment. Unpublished master's thesis, University of Victoria, Victoria, British Columbia, Canada.Google Scholar
  40. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis (2nd edn.). Thousand Oaks, CA: Sage.Google Scholar
  41. Reynolds, A. J., & Walberg, H. J. (1991). A structural model of science achievement. Journal of Educational Psychology, 83(1), 97–107.CrossRefGoogle Scholar
  42. Rogers, W. T., Anderson, J. O., Klinger, D. A., & Dawber, T. (2006). Pitfalls and potential pitfalls of secondary data analysis of the Council of Ministers of Education, Canada, national assessment. Canadian Journal of Education, 29(3), 757–770.Google Scholar
  43. Ross, S. P. (2008). Motivation correlates of academic achievement: Exploring how motivation influences academic achievement in the PISA 2003 dataset. Unpublished doctoral dissertation, University of Victoria, Victoria, British Columbia, Canada.Google Scholar
  44. Statistics Canada. (n.d.). National longitudinal survey of children and youth for ages 16–17. Retrieved April 22, 2008, from http://www.statcan.ca/english/kits/microdata/microdata.htm
  45. TIMSS & PIRLS International Study Center. (n.d.). Homepage. Retrieved July 11, 2008, from http://timss.bc.edu/
  46. Turner, R., & Adams, R. J. (2007). The Programme for International Student Assessment: An overview. Journal of Applied Measurement, 8(3), 237–248.Google Scholar
  47. United States Institute of Education Sciences. (n.d.). What Works Clearinghouse: Homepage. Retrieved June 20, 2008, from http://ies.ed.gov/ncee/wwc/
  48. Yore, L. D., Craig, M. T., & Maguire, T. O. (1998). Index of science reading awareness: An interactive-constructive model, test verification, and grades 4–8 results. Journal of Research in Science Teaching, 35(1), 27–51.CrossRefGoogle Scholar
  49. Yore, L. D., Pimm, D., & Tuan, H.-L. (2007). The literacy component of mathematical and scientific literacy. International Journal of Science & Mathematics Education, 5(4), 559–589.CrossRefGoogle Scholar

Copyright information

© Springer Science + Business Media B.V 2009

Authors and Affiliations

  1. 1.Department of Educational PsychologyUniversity of VictoriaBritish ColumbiaCanada
  2. 2.Department of Family MedicineUniversity of AlbertaEdmontonCanada

Personalised recommendations