Learning about students’ knowledge and thinking in science through large-scale quantitative studies

  • Rolf V. Olsen
  • Are Turmo
  • Svein Lie


The main issue addressed in this article is that there is much to learn about students’ knowledge and thinking in science from largescale international quantitative studies beyond overall score measures. Response patterns on individual or groups of items can give valuable diagnostic insight into students’ conceptual understanding, but there is also a danger of drawing conclusions that may be too simple and nonvalid. We discuss how responses to multiple-choice items could be interpreted, and we also show how responses on constructed-response items can be systematised and analysed. Finally, we study, empirically, interactions between item characteristics and student responses. It is demonstrated that even small changes in the item wording and/or the item format may have a substantial influence on the response pattern. Therefore, we argue that interpretations of results from these kinds of studies should be based on a thorough analysis of the actual items used. We further argue that diagnostic information should be an integrated part of the international research aims of such large-scale studies. Examples of items and student responses presented are taken from The Third International Mathematics and Science Study (TIMSS).

Key words

Assessment Item format Quantitative methodology Science education TIMSS 


L’article met à jour le fait que les études quantitatives internationales à grande échelle nous disent beaucoup, au-delà des mesures générales de scores, sur les connaissances et les pensées des élèves par rapport aux sciences. Les patterns des réponses données à des items individuels ou bien à des groupes d’items peuvent nous éclairer sur la compréhension conceptuelle des élèves, mais risquent de conduire à des conclusions trop simples et non-valides. On discute les moyens de systématiser et d’analyser les réponses à des items à choix multiple. Enfin, on étudie empiriquement les interactions entre certains caractéristiques des items et les réponses des élèves. On montre qu’il suffit d’une petite modification dans la manière de formuler et/ou formater l’item pour produire un effet substantiel dans le pattern de la réponse. Il s’ensuit que l’interprétation des résultats provenant de ce type d’études doit être basée sur une analyse approfondie des items administrés. Il s’ensuit également que l’information diagnostique peut être une partie constitutive des objectifs de recherche internationaux qu’on cherche à réaliser par ce type d’études à grande échelle. Les exemples présentés d’items et de réponses des élèves sont extraits de la Third International Mathematics and Science Study (TIMSS).


  1. Angell, C. (1996).Elevers fysikkforståelse. En studie basert på utvalgte fysikkoppgaver i TIMSS [Students’ understanding of physics. A study based on selected physics items in TIMSS]. PhD Thesis, University of Oslo.Google Scholar
  2. Angell, C., Kjærnsli, M., & Lie, S. (1999).Hva i all verden skjer i realfagene i videregående skole? National TIMSS report population 3. Oslo: Universitetsforlaget.Google Scholar
  3. Angell, C., Kjaernsli, M., & Lie, S. (2000). Exploring students responses on free-response science items in TIMSS. In D. Shorrocks-Taylor & E.W. Jenkins (Eds.),Learning from others (pp. 159–187). Dordrecht: Kluwer.Google Scholar
  4. Boyes, E., & Stanisstreet, M. (1993). The “greenhouse effect”: Children’s perception of causes, consequences and cures.International Journal of Science Education, 15, 531–552.CrossRefGoogle Scholar
  5. Boyes, E., & Stanisstreet, M. (1994). The ideas of secondary school children concerning ozone layer damage.Global Environmental Change, 4, 311–324.CrossRefGoogle Scholar
  6. Clerk, D., & Rutherford, M. (2000). Language as confounding variable in the diagnosis of misconceptions.International Journal of Science Education, 22, 703–717.CrossRefGoogle Scholar
  7. DiSessa, A.A. (1993). Toward an epistemology of physics.Cognition and Instruction, 10, 105–225.Google Scholar
  8. Driver, R., & Easley, J. (1978). Pupils and paradigms: A review of literature related to concept development in adolescent science students.Studies in Science Education, 5, 61–83.CrossRefGoogle Scholar
  9. Driver, R., Asoko, H., Leach, J., Mortimer, E., & Scott, P. (1994). Constructing scientific knowledge in the classroom.Educational Researcher, 23, 5–12.Google Scholar
  10. Duit, R., & Treagust, D.F. (1995). Students’ conceptions and constructivist teaching approaches. In B.J. Fraser & H.J. Walberg (Eds.),Improving science education. Chicago, IL: The University of Chicago Press.Google Scholar
  11. Fraser, B.J., & Tobin, K.G. (1998).International handbook of science education (2 vols.). Dordrecht: Kluwer.Google Scholar
  12. Gilbert, J.K., Osborne, J., & Fensham, P.J. (1982). Children’s science and its consequences for teaching.Science Education, 66, 623–633.CrossRefGoogle Scholar
  13. Gisselberg, K., Kjærnsli, M., Lie, S., & Weng, P. (1996).Preliminary notes from a Nordic study on item formats in TIMSS. Unpublished manuscript.Google Scholar
  14. Halloun, I.A., & Hestenes, D. (1985). Common sense concepts about motion.American Journal of Physics, 53, 1056–1065.CrossRefGoogle Scholar
  15. Harlen, W. (1999).Effective teaching of science, A review of research. Edinburgh, UK: Scottish Council for Research in Education.Google Scholar
  16. Hennessey, S. (1993). Situated cognition and cognitive apprenticeship: Implications for classroom learning.Studies in Science Education, 22, 1–41.CrossRefGoogle Scholar
  17. Kjærnsli, M., Lie, S., Stokke, K.H., & Turmo, A. (1999).Hva i all verden kan elevene i naturfag? [What do students know in science?]. Oslo: Universitetsforlaget.Google Scholar
  18. Kupier, J. (1994). Students ideas of science concepts: Alternative framework?International Journal of Science Education, 16, 279–292.CrossRefGoogle Scholar
  19. Laws, P.M. (1996). Undergraduate science education: A review of research.Studies in Science Education, 28, 1–85.CrossRefGoogle Scholar
  20. Lie, S., Kjærnsli, M., & Brekke, G. (1997).Hva i all verden skjer i realfagene? National TIMSS report population 2. Oslo: Universitetsforlaget.Google Scholar
  21. Lie, S., Taylor, A., & Harmon, M. (1996). Scoring techniques and criteria. In M.O. Martin & D.L. Kelly (Eds.),Third international mathematics and science study. Technical report: Vol. 1. Design and development (chap. 7, pp. 1–16). Chestnut Hill, MA: Boston College.Google Scholar
  22. Minstrell, J. (1992). Facets of students’ knowledge and relevant instruction. In R. Duit, F. Goldberg, & J. Niedderer (Eds.),Research in physics learning: Theoretical issues and empirical studies. Proceedings of an international workshop held at the University of Bremen (pp. 110–128). Kiel: Institute for Science Education, University of Kiel.Google Scholar
  23. Nielsen, H., & Thomsen, P.V. (1983).Hverdagsforestillinger om fysik [Everyday conceptions in physics]. Århus: University of Århus.Google Scholar
  24. Osborne, R., & Freyberg, P. (1985).Learning in science. The implication of children’s science. Auckland: Heineman.Google Scholar
  25. Pfundt, H., & Duit, R. (1994).Bibliography. Students’ alternative frameworks and science education (4th ed.). Kiel: Institute for Science Education, University of Kiel.Google Scholar
  26. Schoultz, J. (2000). Conceptual knowledge in talk and text: What does it take to understand a science question? InAtt samtala om/i naturvetenskap. Kommunikation, kontext och artefakt. PhD thesis (partly in English), University of Linkoping.Google Scholar
  27. Scott, P.H. (1992). Pathways in learning science: A case study of the development of one student’s ideas relating to the structure of matter. In R. Duit, F. Goldberg, & J. Niedderer (Eds.),Research in physics learning: Theoretical issues and empirical studies. Proceedings of an international workshop held at the University of Bremen (pp. 203–224). Kiel: Institute for Science Education, University of Kiel.Google Scholar
  28. Smith, T.A., Martin, M.O., Mullis, I.V.S., & Kelly, D.L. (2000).Profiles of student achievement in science at the TIMSS international benchmarks: U.S. performance and standards in an international context. Chestnut Hill, MA: Boston College.Google Scholar
  29. Tamir, P. (1990). Justifying the selection of answers in multiple choice items.International Journal of Science Education, 12, 563–573.CrossRefGoogle Scholar
  30. Thorseng, H. (1997).Sporsmål og svar i naturfag [Questions and responses in science]. MSc Thesis, University of Oslo.Google Scholar
  31. Vygotsky, L. (1986).Thought and language. Cambridge MA: MIT Press.Google Scholar

Copyright information

© Instituto Superior de Psicologia Aplicada, Lisbon, Portugal/ Springer Netherlands 2001

Authors and Affiliations

  1. 1.Department of Teacher Education and School DevelopmentUniversity of OsloOsloNorway

Personalised recommendations