1 Introduction

School curricula can change at a very slow pace. They are controlled by communities, politicians, educators and others, rather than content experts, so naturally innovations take time to be adopted. In contrast, technology and science is advancing at a very rapid rate [1]. Disparities between the lived world of children embedded in artefacts from recent knowledge and the school curriculum can lead to tensions for them and their teachers. For example, 2005 saw the celebration of the centenary of Einstein’s publication of special theory of relativity [2], but this important scientific idea is rarely found in primary school classrooms. One way to bring new knowledge into schools may be to use computers. However, computers are not having the desired impact in education. A recent OECD report [3] shows a weak inverse relationship between computer investments in schools and learning achievement measured by PISA tests. This makes for a wicked problem in education.

2 Literature

Perkins [4] provided the concept of ‘person-plus’ to embody the shared cognition of a learner assisted by their notebook. In our projects, we acknowledge this notebook could be a notebook computer, or any kind of specialised electronic assistive device. This ‘person-plus’ view supports arguments for open book assessment, because it is closer to real-world problem solving. The assistive device improves information retrieval and accuracy, and Perkins argues this leads to higher order cognitive functioning. By extension, a student using modern digital technology can solve problems that are far more complex than traditionally expected. This models human adoption of other mind tools such as language, numbering systems and so on. Moursund [5] extended this thinking to divide problems into those which are more readily solved by humans alone, by computers alone, and those which are best solved by humans working in combination with computers. Increasingly we face the latter, so it makes good sense to selectively school our children to work in this mode where it is useful and appropriate.

Such an approach runs counter to traditional educational practice. The aim of most educational innovations is to teach the content set in the standard curriculum for a class of students, in a better or more engaging way. If computers are to radically transform the curriculum, it appears to be necessary to use them to teach content which is suited to their use. New technology often makes old methods redundant, so to teach handwriting with a computer keyboard would be unlikely to succeed. If our projects aspired to deliver novel content to Year 6 (age 11) students, the criteria for success would have to be different from ‘teaching better’ or ‘more engagement’. The statistic of ‘effect size’ is often Cohen’s d, but can also be a Z-score or an odds-ratio [6]. Effect size is the accepted measure of educational impact (see Fig. 1). Hattie’s [7] meta-study of different classroom interventions established an effect size of 0.4 as the ‘hinge point’ at which a significant impact could be determined.

Fig. 1.
figure 1

Effect size of learning improvement (intervention or age) (after [8]). This also shows the impact of Calculus for Kids

Full size image

Fluck’s transformative use of computers in schools [10] was the focus for this study. The highest effect size for transformative ways of using computers in class reported by Puentedura [9] was 1.6, relating to ‘redefinition’ activities. It is already well known that, in the right circumstances, children can use computers to achieve results that many observers see as ‘beyond their years’. Papert and others were demonstrating this with Logo in the late 70s and Scardamalia and Bereiter with CSILE in the late 80s and 90s. Research teams have frequently shown outstanding effects when using well-designed software in properly supported educational contexts. We undertook this research with the specific intent to seek high effect size learning transformation through computer use.

3 Methodology

Our main conceptual tools were Rogers’ [11] theory of innovation adoption, the non-template problem-solving method of Allen [12], a realistic mathematics education approach [13], and a methodology we are developing on the transformational use of ICT in school education which combines professional software tools with multimedia instructional materials.

In both projects described in this report, the basic design for the research was an intervention method using a cyclic approach, consisting of four stages: produce/modify learning materials, train the teachers, conduct intervention activity in schools, and assess results. The cycle was ongoing, ensuring continuous updating and modifications to meet changes and findings. We recruited schools for the projects from several Australian states, ensuring a wide range of ICSEA (Index of Community Socio-Educational Advantage) scores. The ICSEA is constructed so that the average score is 1000.

Local facilitators were chosen by each participating school and attended training sessions at the University of Tasmania. In subsequent years we brought previously-trained facilitators back for these sessions to provide feedback on their local implementation and induct new facilitators joining the project. This ensured new learning from each round was handed on to the next. This process was used each year to introduce the teaching techniques we developed to newly participating schools. Since we acquired perpetual licences for the software, schools could participate for up to three years. This gave us the opportunity to investigate the same process repeatedly used in the same school over several years to see if they obtained consistent results. For both projects, teachers received teaching materials and software, which will now be described.

The Australian Research Council funded the Calculus for Kids project. It taught Year 6 students to solve real world problems with integral calculus using computer algebra software. In the Australian curriculum, calculus is taught in Years 11 and 12 [14]. The teaching materials for Calculus for Kids comprised multimedia presentations (in PowerPoint); MAPLE workbook files with worked examples and practice questions; and PDF documents containing more demanding practice questions. The software was the computer algebra package MAPLE, installed on each student’s computer workstation. The multimedia presentations included videos explaining concepts e.g. how algebra relates to more regular shapes and actions rarely met in real life, while calculus relates to more real, and less regular, shapes and actions. These explanatory materials gradually made way for operational skills training for the MAPLE program, and then showed how the principles and notation of integral calculus could be implemented by using the software. Topics included graphing functions, equations for parabolas, calculating definite integrals, areas between curves and of regions bounded by several curves using integrals. By the end of the twelfth lesson, students were able to solve real world problems using integral calculus by operating their computers. This learning achievement was assessed in the final thirteenth lesson through a post-test based upon the first year university engineering calculus examination, to which students responded using their computers running MAPLE. An example post-test item can be seen in Fig. 2.

Fig. 2.
figure 2

Post-test item from Calculus for Kids

Full size image

The Ian Potter Foundation funded the Science-ercise project. The teaching materials were delivered as a similar set of multimedia presentations and a printed booklet for each student, together with some practical equipment such as constant velocity model cars and a parabola cannon. The software in this case comprised an Excel notebook file or its equivalent for iOS devices such as iPads. The file contained a worksheet for each of the six activities to simplify calculations. These activities included constant velocity motion, parabolic motion, double-slit optical interference fringes and the twins’ paradox from special relativity. At the completion of the learning sequence, students undertook a short test to measure learning achievement assisted by their computers running the same software. For Science-ercise, the post-test items were modified from Tasmanian Certificate of Education Physics university entrance examinations (Year 11/12). See Fig. 3.

Fig. 3.
figure 3

Post-test item from Science-ercise.

Full size image

4 Results

Both the interventions in this study were designed to elicit learning outcomes from Year 6 students (11–12 years in age) comparable to those normally expected from much older students. The instruments used to validate this level of achievement were post-activity tests containing items from tests taken by such older students.

The classes selected for participation in these two projects were mixed-gender and mixed ability from a range of socio-economic backgrounds. Calculus for Kids was conducted in 26 schools from five states over seven years; and Science-ercise has involved 187 students in 5 Tasmanian schools over 13 months. Table 1 provides demographic data for the projects, establishing the representative nature of the sample populations.

Table 1. Demographic data for the interventions
Full size table

The key question was the extent to which participating students demonstrated high-level learning achievement in the post-tests. Normally at these levels (Year 11/12, first year degree courses), a pass mark is considered to be 50%, so the proportion of students attaining this level of achievement was the key determination to be made (see Table 2).

Table 2. Proportion of students achieving a pass mark
Full size table

The results show that a large proportion of students in the Calculus for Kids project achieved a pass mark on questions about practical uses of integral calculus. The Science-ercise project showed a bare majority of students achieved a pass mark. Anecdotally, we got the impression some schools in the latter project found the material more demanding and in some case were not able to finish the sequence of lessons.

Calculating an effect size for interventions such as these presents several methodological difficulties. Traditionally, effect size is calculated as Cohen’s d based upon the difference in achievement obtained from the same or calibrated pre- and post-tests. In these projects, there were strong pedagogical reasons for omitting any pre-test. Firstly, the topics were outside the prescribed curriculum at these age levels, and therefore there was no reason to expect any knowledge of the topics whatsoever from the participating students. Secondly, the projects were constructed to generate enthusiasm and confidence for students tackling ‘hard’ concepts. This objective would have been compromised by starting the teaching sequence with a test highly likely to confound and disappoint most participants. As it was, the data show the students were drawn from populations with close to average ICSEA measures of social advantage, and the majority of participants achieved a post-test score pass mark.

The instruments used to measure learning achievement before and after the innovation are clearly crucial. Ideally, effect size is calculated by asking students to complete the same achievement assessment before and after the treatment/innovation. However, this is not always possible, desirable or feasible. Nor is it necessary. All that is required is that the measurements map onto a common scale. We chose to map the achievements of the students onto an academic year level scale. For the pre-test measures we used the national calibrated numeracy test results (NAPLAN). The post-test scores were mapped onto the calibrated first year engineering bachelor’s course, and therefore to Year 13 on the common scale (university entrance occurs after Year 12). To calculate the effect size for Calculus for Kids, we initially found the correlation between pre-treatment academic year level (based on numeracy scores on standardised national tests) and the post-treatment year levels. This came to 0.56 with p = 0.048 which is statistically significant. Similar work was done with Science-ercise.

We used the conventional method for calculating the intervention effect size in the absence of a pre-test by mapping participants onto a common scale before and after the learning intervention (treatment) as shown in Fig. 4, and then calculating Cohen’s d. This produced an effect size of 25.53 for Calculus for Kids, and 11.45 for Science-ercise. These are very high effect sizes.

Fig. 4.
figure 4

Effect size from mapping to a learning age scale

Full size image

In our second approach to determine effect sizes, we relied on Glass et al.’s [15] finding that normal learning progress is one standard deviation per year. Therefore, an intervention that produces learning achievements 5–6 years ahead of chronological age, equates to 5–6 standard deviations of progress, corresponding to an effect size of 6.17 in this case. This is well above Hattie’s hinge point for a significant intervention. The Calculus for Kids intervention achieved an effect size greater than 4.0 [16]. Our Science-ercise project achieved similar success [17]. These effect sizes are over ten times the size of Hattie’s hinge point for a significant intervention.

Looking further at effect sizes for non-traditional interventions, Wiliam [18] modified Glass’s conjecture in 2004 by stating that learning achievement of “one third of a grade [was] equivalent to a standardized effect size of 0.21”. Therefore, one academic year level would be equivalent to an effect size of 0.63. It seems this relationship declines with student age. Wiliam described this decline by saying “one year’s growth in achievement typically ranges from around 0.25 to 0.4 standard deviations” and “for the NAEP tests, one year’s growth is only about one fourth of a standard deviation”. We adopted a conservative value of 0.3 SD per year of schooling. Even at 1.85, the effect size for the Calculus for Kids innovation is 4.6 times Hattie’s hinge point of 0.4 (Table 3).

Table 3. Effect size measures for the two interventions.
Full size table

5 Discussion

These projects have provided evidence that children as young as 10 can use computers to demonstrate higher order learning achievement when freed from contemporary curriculum and reporting constraints. The power of this evidence is increased by the geographical and social diversity of the participants, providing generalisation of the results and impetus for radical reform of learning. Policy makers require these practical demonstrations to inform the community and make political decisions about the content of schooling which are palatable and feasible.

It is important to discuss how best to calculate an effect size for these transformative uses of computers in school education. There are important issues of methodology to consider. We have been asked if our Year 6 students really understood integral calculus. Our response is they generally passed the university test designed to assess this specific learning achievement. A counter-argument is they had assistance, in the form of the computer software they used when solving the post-test problems. In return, we might argue the conventional students were allowed to use writing implements unavailable to our participants. It is evident we have no clear way to think about the impact of computers when transforming curriculum, but a clear imperative to do so given their pervasive use in professional and business life. In this paper we have applied three reasonable ways to calculate the effect size of the intervention. Even the lowest calculation provided a quantum almost four time the ‘hinge point’ for an effective innovation.

One way to view our work is to say the operational skills of operating MAPLE to solve definite integrals replaced the more complex operational skills conventionally taught in manipulating integral calculus formulae. Our participants needed to understand conventional mathematics notation, and demonstrated this by placing the correct values and symbols into the equations they constructed using the software. The computer did the work of actually crunching the numbers to evaluate the compiled expressions. To that extent, the students demonstrated the higher order learning achievement of solving real world problems using integral calculus. They were assisted by software able to undertake symbolic manipulation which freed them from the labour of lower order mechanical evaluation. We do not argue they showed higher cognitive function, just higher learning achievement.

We anticipate that project impact over time will be considerable. With younger students proven capable of higher order learning achievement through the sensitive use of computers, curriculum reform can bring current knowledge forward in schooling, leaving room and time for new ideas and socialisation. This will lead to a more advanced workforce capable of greater participation in the knowledge economy. The benefits will therefore be both economic and social.

It is important to note the important flow-on benefits of these projects. We have only demonstrated accelerated learning achievement of 4+ years with primary school students. However, similar processes could be put in place at other educational levels. For instance, first year undergraduates will be able to achieve learning outcomes at the Masters or even initial Ph.D. candidate level. Year 10 students will be empowered to demonstrate understanding from first year degree courses. We seek to engender this quantum leap in learning in areas where humans working with computers can perform better than either alone. Our work with Year 6 students can be a lever for future work at all educational levels.

6 Conclusion

These studies have provided in-principle demonstrations that children as young as ten years old can master advanced concepts through the use of sophisticated software. Participating students were drawn from a wide geographical range and a variety of socio-economic backgrounds, indicating the generality of the findings. However, the topics were strictly mathematical and scientific, so there is a need to widen the subject base to substantiate a general claim the curriculum should be re-thought. To broaden the subject base, we propose further research to build on this foundation.

A follow on study will explore curriculum redefinition as a way forward with important inspiration from the Europe-wide iTEC project that demonstrated higher order thinking in learning through envisioning classrooms of the future and established a bank of new teaching ideas [19].

Such a study will be foundational to a broader demonstration of curriculum transformation using computers. It will focus on the use of computational thinking throughout the curriculum, conceived as in Fig. 5.

Fig. 5.
figure 5

Curriculum redefinition through computational thinking (after [20])

Full size image

As in the projects this report has described, we would continue to work with Year 6 students across Australia, using short effective interventions and specialist application software. While the project will provide starting point activities comprising six lessons for each of eight subjects, schools will participate in guided workshops that will generate specific plans appropriate for their own students. Key elements in each of the interventions will be:

  • Short in duration (typically 5–7 lessons);

  • Students to use a single computer software application for each intervention, generally free of cost to schools;

  • Learning outcome will be a significant departure from the existing curriculum (4–6 years in advance of chronological age);

  • A post-test of learning achievement accompanied by pre- and post-activity attitudinal surveys. For the latter we will modify the Attitudes toward mathematics inventory [21] to make it subject-generic and link with our previous work; also, the Mathematics and technology attitude scale [22, 23] will be similarly adapted.

This broader base, if successful, would provide strong evidence, through effect sizes of four or more, for curriculum redefinition. We can only hope this will be extended by inference from Year 6 to both older and younger students.