The Challenge of Evaluating Students’ Scientific Literacy in a Writing-to-Learn Context

Abstract

This paper reports on the challenge of evaluating students’ scientific literacy in a writing-to-learn context, as illustrated by our experience with an online science-writing project. In this mixed methods study, year 9 students in a case study class (13–14 year olds, n = 26) authored a series of two ‘hybridised’ short stories that merged scientific and narratives genres about the socioscientific issue of biosecurity. In seeking to measure the efficacy of the intervention, we sought evidence of students’ conceptual understanding communicated through their stories. Finding a suitable instrument presented our first challenge. This led to the development of scoring matrices to evaluate students’ derived sense of scientific literacy. Student interviews were also conducted to explore their understanding of concepts related to the biosecurity context. While the results of these analyses showed significant improvements in students’ understanding arising from their participation in the writing tasks, the interviews highlighted a second challenge in evaluating students’ scientific literacy: a disparity between their written and vocalised understandings. The majority of students expressed a deeper level of conceptual understanding during the interviews than they did in their stories. The interviews also revealed alternative conceptions and instances of superficial understanding that were not expressed in their writing. Aside from the methodological challenge of analysing stories quantitatively, these findings suggest that in a writing-to-learn context, evaluating students’ scientific literacy can be difficult. An examination of these artefacts in combination with interviews about students’ written work provided a more comprehensive evaluation of their developing scientific literacy. The implications of this study for our understanding of the derived sense of scientific literacy, as well as implications for classroom practice, are discussed.

Appendices

Appendices

Appendix 1

ᅟ An extract from the part A writing task

Appendix 2

Derived scientific literacy scoring matrix: part A
Criterion 1. The country of origin of the biological incursion.
0	The story does not include the biological incursion’s country of origin.
1	The country of origin is incorrect.
2	The story includes the biological incursion’s correct country of origin.
Criterion 2. How the biological incursion entered Australia.
0	The story does not explain how the biological incursion entered Australia.
1	The story incorrectly explains how the biological incursion entered Australia.
2	The story correctly explains how the biological incursion entered Australia.
Criterion 3. The problems the biological incursion has caused or continues to cause native and/or commercial species or ecosystems (environmental, social and economic impacts).
0	The story does not address any environmental, social or economic impacts of the biological incursion.
1	The story incorrectly or incompletely addresses reasonable environmental, social and economic impacts that pertain to the biological incursion.
2	The story accurately addresses reasonable environmental, social and economic impacts that pertain to the biological incursion.
Criterion 4. The difficulties scientists and farmers face controlling the pest or how the pest was brought under control.
0	The story does not explain any difficulties faced by scientists and/or farmers in controlling the biological incursion or how the pest was brought under control.
1	The story incorrectly or incompletely explains the difficulties faced by scientists and/or farmers in controlling the biological incursion or how the pest was brought under control.
2	The story accurately explains the difficulties faced by scientists and/or farmers in controlling the biological incursion or how the pest was brought under control.
Total score: ……… out of 8

Appendix 3

Derived scientific literacy scoring matrix: part B
Criterion 1. What is avian influenza?
0	The story does not identify what avian influenza is.
1	The story incorrectly identifies what avian influenza is.
2	The story correctly identifies avian influenza as a contagious or viral disease/infection.
Criterion 2. The organisms affected by avian influenza or those at risk of infection.
0	The story does not state the organisms affected by avian influenza or those at risk of infection.
1	The story incorrectly and/or incompletely states the organisms affected by avian influenza or those at risk of infection.
2	The story accurately states the organisms affected by avian influenza or those at risk of infection.
Criterion 3. The problems that an outbreak of avian influenza would cause on a farm and in the wider community (social and economic impacts).
0	The story does not address any social or economic impacts of avian influenza.
1	The story incorrectly or incompletely addresses reasonable social and economic impacts that pertain to avian influenza.
2	The story accurately addresses reasonable social and economic impacts that pertain to avian influenza.
Criterion 4. The difficulties scientists and farmers face controlling avian influenza.
0	The story does not explain any difficulties faced by scientists and/or farmers in controlling avian influenza.
1	The story incorrectly or incompletely explains the difficulties faced by scientists and/or farmers in controlling avian influenza.
2	The story accurately explains the difficulties faced by scientists and/or farmers in controlling avian influenza.
Total score: ……… out of 8