Challenge of Transforming Curricula with Computers, High Impact Interventions and Disruption
Conventional educational attainment expectations for school students are generally defined by curriculum documents in each jurisdiction. However, new technologies disrupt societies, so it is pertinent to ask how computers have changed educational expectations. Robert Heinlein put this into perspective in a short story (1957). The protagonist is Holly Jones. She is 15 and a spaceship designer: “I’m very bright in mathematics, which is everything in space engineering, so I’ll get my degree pretty fast. Meanwhile we design ships anyhow. I didn’t tell Miss Brentwood this, as tourists think a girl my age can’t possibly be a spaceship designer.” This demonstrates some early aspirations of how education might change in future where lunar habitats become well established.
Outside education, business process reengineering is the practice of re-designing work, particularly when using new technology to reduce costs (Davenport 1993). Davenport commented on the role of computers in changing business practices: “...Information and IT are rarely sufficient to bring about the process change; most process innovations are enabled by a combination of IT, information and organizational/human resource changes” (p. 95).
Ranking of the companies 1–100
Market value in billion US dollars
Alphabet (Google holding company)
Alibaba (online sales site)
There is therefore a contrast between expectations of educational change and actual change in the commercial world. The commercial world is the one into which school students will grow. Therefore, educational leaders have an imperative to grapple with this growing schism.
This dichotomy provides a significant challenge for educational theorists and researchers. There have been two noted responses by theorists, the TPACK framework and SAMR model. Koehler and Mishra (2008) introduced TPACK as a way of looking at the key role of technology when teaching online. However, this focused on the method of teaching, not the content. Puentedura (2012) went further and theorized that computers would first substitute for common tools (e.g., keyboard replacing a pen), then augment those tools. Later, in a transformation phase, the computer would allow for significant task redesign, leading to redefinition of learning to that which was previously inconceivable. This transformative perspective was predicted by Downes et al. (2002, p. 23) and aligns with Fluck’s three phase model for ICT policy development (2003, p. 139).
The challenge for researchers arises from the need to measure change. Educational research frequently looks at proposed interventions, generally with the intention of improving learning. Two classical kinds of measures are preferred for this kind of work. The first is null hypothesis statistical testing, with a t-test or similar used to confirm a new process makes a difference to learning when compared to previous processes. The significance of such a correlation measure is established by a p-value, which demonstrates the unlikely possibility of the result being accidental or by natural variation.
The second class of measures of improved learning includes effect size. Lopez et al. (2015) stress the importance of citing the effect size, because this dimensionless measure is independent of sample size. Effect size can be calculated in several ways: Cohen’s d, using the pooled standard deviation of the control and experimental groups is most commonly found when conducting t-tests; partial eta squared has mostly been reported in studies using ANOVA; adjusted R2 has been associated with regression analyses; and Hedges’s g, which applies a correction for small samples (Lopez et al. 2015, p. 108; Tamim et al. 2011). Effect sizes for the use of computers in education have been quite modest with “the average effect size ranging between .20 and .47 standard deviations” (Karasavvidis 1999, p. 2). Effect size calculations are predicated on the test scores being distributed in a near-normal way and on the standard deviations for pre- and posttests being similar.
With transformative applications of computers in the SAMR model, educational researchers have a difficulty in identifying a common scale from which to calculate the effect size of a learning innovation. Following the TPACK framework, if the researcher is using a computer to teach the same content in a new way, then a standardized and calibrated assessment of educational achievement can be administered to both the control and experimental groups before and after the intervention, with the Solomon four-group design overcoming external validity weaknesses caused by pretest sensitization of subjects. In such research designs, the implicit assumption is the pre- and posttests will accurately measure student educational achievement at a point in time.
For transformative applications of computers, the radically different educational content may be such a departure from the conventional curriculum that students will have no prior knowledge whatsoever. This does raise an ethical question concerning the emotional impact of a pretest assessment in which virtually all students will likely have a negative experience. The research design parameters violate simple humanity, so methods are sought which give the required validity and yet provide a positive emotional experience to the research subjects.
Using Computers in Assessment
There is another aspect of teaching new content which is impossible to deliver without computers: assessment. If computer use is transformative, it will be essential in assessment (Fluck and Hillier 2017). This also makes the pretest/posttest paradigm difficult. If the learned skills can only be realistically demonstrated through student computer use in the posttest, then protocols would dictate the pretest should also depend upon the same mode of assessment. A complication might be the interference effect of the operational skills required to use the software intended for use as part of the educational intervention to be studied. If operating this particular software is in fact an element of the intervention, then its inclusion in the pretest is contraindicated.
The difference in modality of testing became a matter of national concern in Australia in 2018 when standardized testing in schools was conducted both on paper and on screen. Approximately 20% of school students undertook the testing on screen. Severe doubts were raised about the comparability of the results from the two modes, with the testing agency “acknowledging that year 9 students who sat the writing test online performed better on average than those who completed the traditional pen-and-paper version” (Robinson 2018). The testing agency subsequently released the results using a common mean scale score and reported effect sizes for any differences over time (Australian Curriculum, Assessment and Reporting Authority 2018). This demonstrates national bodies are aware of the modality issues, and map test scores onto a common scale to allow effect sizes to be calculated.
Puentedura (2012) has reported transformative “redefinition” uses of computers in school education with an example effect size of 1.563 (Salomon et al. 1989). This is in the context of an average or hinge effect size of 0.4 for educational interventions (Hattie 2009). The Salomon et al. effect size of 1.563 provided the greatest impact in Hattie’s list of meta-studies “Teacher estimates of student achievement.” Salomon et al. used a research design with three randomly selected parallel groups of students. All subjects were pretested with a group-administered standard test of reading comprehension (p. 622). Posttesting comprised a metacognitive reconstruction task, another version of the reading comprehension test and a delayed essay writing task. The researchers undertook structured equation modeling and did not actually calculate the effect size. This was drawn as Hedges’ g from the meta-study by Pearson et al. (2005). Puentedura’s reliance on this study and meta-analysis was criticized by Hamilton et al. (2016).
The Salomen et al. study is useful because it points to a multiplicity of achievement tests both at the pretest and posttest stage. The research design explicitly used a different version of the reading comprehension test for the posttest. This technique is sometimes used to avoid memorization – where the student recalls the same item on the pretest and responds to the posttest informed by that recollection. Test design can be influenced by the choice of item-response theory or Rasch modeling. Item response theory allows test elements to be analyzed for difficulty, which assists when collating scores. Rasch modeling goes further and allows scores to be mapped onto the same invariant scale. We can presume that Salomen et al. mapped the results of the different reading comprehension tests onto a common scale to allow meaningful comparisons. In a similar way, transformative educational computer-based activities may need a different posttest to calculate an effect size.
One possible way forward in transformative or redefinition applications of computers in education is to look at learning achievement in terms of years of schooling. Normally, an effect size of 1.0 corresponds with a change of one standard deviation (Coe 2002, p. 3). Glass et al. (1981, p. 102) have asserted that an effect size of 1.0 corresponds to the normal learning achievement in a year in elementary school. However, Wiliam (2010) found this relationship varies according to the age of the student, becoming smaller for older year groups. Learning progression reduced from one standard deviation per year in Grade 4 and reduced to 0.27 in Grade 8 (p. 116).
There are perhaps two justifiable ways forward. Where the transformation involves learning material relevant to a higher chronological age than that of the student subjects, Wiliam’s progression could be used to measure the impact in terms of years of advanced standing. This faces a minor problem in that Wiliam’s progressions differed slightly for Mathematics and Reading, so the alignment between age and maturation is not precise. The other possible way forward is to map both pre- and posttest results onto a curriculum-age scale. A curriculum-age scale is one where each item of learning is situated in a particular schooling year and is the way most curricula are presented. If the transformative intervention aspires to teach material 3 years in advance of the chronological age of student subjects, then this would be the point on the curriculum-age scale indicated by test scores. Such an approach allows for the pretest and posttests to be different, since each test points to a different age level on the scale.
The Calculus for Kids project (Chin et al. 2017) epitomized this transformative approach, asking Year 6 (12-year-old) students to learning how to solve real world problems using integral calculus, with MAPLE computer algebra system software as an aide. This skill is not normally taught until Year 12, so the posttest used items from 1st year university examinations of integral calculus (presumed to be Year 13 on the curriculum-age scale). Key to the assessment of learning achievement in this project was student computer use.
As shown previously, the method of calculating effect size varies between studies. Robust research designs rely upon the same pre- and posttest to determine the effect size of an intervention. However, other methods are also used. Tamim et al. (2015) cite a large effect size of 2.066 for a study of tablet use (Huang et al. 2014). Huang et al. (2014) used different pre- and posttests containing 26 and 14 multiple-choice questions, respectively, but validated these through Cronbach’s alpha (α, for reliability) and an expert review group for validity. In consideration of these factors, the researchers resolved to aim for a large study and used Cohen’s d for reporting the effect size.
Traditionally, students have been taught how to find the integral of a mathematical function using a series of rules deduced from first principles. This helps students to understand how to integrate a new function in the future. However, as the catalogue of integral functions grows, the use of poorly memorized results using “first principles” can impede practical calculation. Therefore, professional engineers use a variety of specialist software to “crunch the numbers” although they also need to understand the process and its application to obtain sensible and accurate results. One might argue that reliance on computing equipment in engineering is analogous to the widening use of word processors in newspaper offices in lieu of pens. There is certainly a discussion to be had about the way these technologies redefine the underlying skills or their acquisition.
RQ1: What is the effect on learning achievement of transformational computer use in one mathematical education topic for Year 6 students?
Participating schools were recruited from five Australian states, ensuring a wide range of locations and socio-economic backgrounds. Local facilitators were chosen by each participating school and attended a training session at the University of Tasmania. The training course for facilitators inducted them into higher-level mathematical areas, by taking them through the student learning materials consisting of colorful multimedia animations. In each package, teachers were provided materials for 12 1-h modules to be taught over 6 weeks with links to worked examples in Maple worksheets and extension activities for students to autonomously consolidate learning.
The Calculus for Kids learning module was designed for Year 6 students in their regular classes where everyone could use a computer for each lesson. Schools with both 1:1 laptop classes and computer laboratories were included in the study. Our initial preference was for situations where every student used a personal laptop computer. If this was taken home each day, that promised additional time on task. However, this ideal was not always met, and some classes were able to access the learning module in a school computer laboratory at specific timetabled sessions. Maple was selected as the supportive software because several of the researchers were familiar with it; to ensure students were not disadvantaged in future mathematics learning (it uses conventional notation); and because the study was supported by an industrial partner in the research (Maple 18-2016).
Demographic data for Calculus for Kids
Mean ICSEA (s.d.)
Mean age on the day of the posttest (s.d.)
The ICSEA (Index of Community Socio-Educational Advantage) is a scale of socio-educational advantage that is computed for each school (Australian Curriculum, Assessment and Reporting Authority 2015). The ICSEA is constructed so the median score is 1000 with a standard deviation of 100, and incorporates parental occupation/education, remoteness, and indigenous proportion of students. Lower values represent greater educational disadvantage.
The study used published data on numeracy skills at each school (NAPLAN) as a proxy for aptitude on a curriculum-age scale. This took the place of a specific pretest. The posttest positioned students onto the same curriculum-age scale. It was designed by one of the authors, a nationally awarded teacher of university engineering, and used items from 1st year university engineering calculus examinations.
Between June 2010 and April 2016, 434 students in 23 classes at 19 schools in 5 (of 8) Australian states completed the Calculus for Kids learning module. The module was taught in Tasmania, Queensland, Victoria, South Australia, and New South Wales. The students were selected to be in Year 6, generally the final year of primary school (changes in Queensland meant the Year 7 cohort moved from primary to secondary school during the period of the research). The mean age of students was 11.96 years, with the youngest 9.97 years and the eldest 13.82 years.
Analysis showed there was a small but significant difference in posttest scores for male students (M = 13.30, SD = 0.39) and female students (M = 13.38, SD = 0.33; t(432) = −2.166, p = 0.031, two-tailed). So, the male students appeared to perform less well. However, when the pretest curriculum-age scale levels were subtracted from the posttest curriculum-age scale levels, the gender differential became insignificant: t(432) = −1.38, p = 0.169). There was also a small and positive correlation between school ICSEA value and student posttest curriculum-age scale level (r = 0.168, n = 434, p = 0.000), with higher levels of social-economic status and geographical advantage value associated with higher levels of student achievement. This corresponds with expected findings; the correlation is small though significant.
Student achievements for Calculus for Kids
Year level on curriculum-age scale – pretest
Year level on curriculum-age scale – posttest
These values give an effect size based on Cohen’s d of 22.19. This is extraordinarily high, since an educational intervention is considered significant if it has an effect size greater than 0.25 of a standard deviation using a “rule of thumb” (Tallmadge 1977, p. 34) or 0.4 (Hattie 2009) using Cohen’s d.
The effect size is very large (>0.8) (Cohen 1988). This finding is important, because much research to date has compared the impact of computer use on learning the same content without computers. When contemplating significant curriculum reform built on an assumption of universal computer access, this very large effect size provides evidence in support of a transformative approach.
The outcome from this project has provided evidence that children as young as 10 years old can use computers to demonstrate higher order thinking when freed from contemporary curriculum and reporting constraints. The power of this evidence is increased by the geographical and social diversity of the participants, providing impetus for radical reform of learning. Policy makers require these practical demonstrations of curriculum transformation with computers to inform the community and make political decisions about the content of schooling palatable and feasible (Kozma 2011, p. 27). This is particularly true in the case of disruptive innovation (Cerna 2013, p. 14).
The researchers have been asked if the Year 6 students really understood integral calculus. The response is students generally passed the university test designed to assess this specific learning achievement. A counter-argument is the students had assistance, in the form of the computer software they used when solving the posttest problems. The Calculus for Kids students needed to know, what mathematical tools to use, when to use them, and how to use them. This required understanding of the concepts within integral calculus and their application. It is clear there are difficulties when thinking about the impact of computers when transforming curriculum, but there is a clear imperative to do so given their pervasive use in professional and business life. As with any technological innovation, its use replaces old skills with new ones. In Calculus for Kids, the skills to operate the Maple software replaced the mechanical memorization of algebraic formula manipulations. The consequence was that students were able to solve real world problems using integral calculus, which is indicative of higher order thinking.
The study only demonstrated accelerated learning achievement with elementary school students. However, analogous processes could be trialed at other educational levels. For instance, 1st year undergraduates might be able to achieve learning outcomes at the Masters’ or even initial PhD candidate level. Year 10 students may be empowered to demonstrate understanding from 1st year degree courses. This would be a significant outcome for transformative uses of computers in education.
There is no doubt that school education suffers from plurality of control and the inertia evident in any large system. Both educational leaders (Oates 2009) and students (Barrance and Elwood 2018) have expressed concerns that innovations and assessment reforms are proceeding with undue haste. Society needs to contemplate the growing disjunction between the pervasive use of digital technology in professional life and the lack of transformative curriculum changes.
This raises significant difficulties for educational researchers, where the “gold standard” of effect size for evaluating innovation impact needs readjustment. Transformational and redefinition uses of computers make it possible to teach new concepts, not currently in the age-designated curriculum for student subjects. The conventional use of “effect size” comes under increasing strain in such circumstances. Already there are a variety of statistical calculations leading to an effect size, but transformational computer use makes pre/post testing and the use of control groups very difficult. This entry has argued that many such situations can be addressed by using the concept of a curriculum-age scale. It does leave open the question of transformational learning not covered by the operational curriculum for the relevant, or indeed any other, jurisdictional area. That may be for another discussion!
- Allen D (2001) Learning integral calculus through non-template problem solving. Primus Probl Resour Issues Math Undergrad Stud 11(2):147–160Google Scholar
- Australian Curriculum, Assessment and Reporting Authority (2015) Guide to understanding ICSEA (Index of Community Socio-educational Advantage) values. Australian Curriculum, Assessment and Reporting Authority. http://docs.acara.edu.au/resources/Guide_to_understanding_icsea_values.pdf
- Australian Curriculum, Assessment and Reporting Authority (ACARA) (2018) National assessment program – literacy and numeracy (NAPLAN) 2017 technical report. Australian Curriculum, Assessment and Reporting Authority (ACARA), Sydney. http://www.nap.edu.au/docs/default-source/default-document-library/2017_naplan_technical_report.pdf?sfvrsn=2Google Scholar
- Cerna L (2013) The nature of policy change and implementation: a review of different theoretical approaches. OECD. https://www.oecd.org/edu/ceri/The%20Nature%20of%20Policy%20Change%20and%20Implementation.pdf
- Chin CKH, Fluck A, Chong CL, Penesis I, Ranmuthugala D, Coleman B (2017) Higher order thinking through Calculus for Kids. J Digit Learn Teach Vic 4(1):26–31Google Scholar
- Coe R (2002) It’s the effect size, stupid: what effect size is and why it is important. Paper presented at the Conference of the British Educational Research Association, University of Exeter, England. 12–14 Sept 2002Google Scholar
- Davenport TH (1993) Process innovation. Harvard Business School Press, BostonGoogle Scholar
- Downes T, Fluck A, Gibbons P, Leonard R, Matthews C, Oliver R, Vickers M, Williams M (2002) Making better connections. Commonwealth Department of Education, Science and Training. Online at http://isq3.wikispaces.com/file/view/Making+Better+Connections+%28DEST%29.pdf
- Fluck A (2003) Integration or transformation? A cross-national study of ICT in school education. PhD thesis, University of Tasmania. https://eprints.utas.edu.au/232/
- Fluck A, Hillier M (2017) eExams: strength in diversity. In: Tatnall A, Webb M (eds) Tomorrow’s learning: involving everyone. Springer Nature, Cham, pp 409–417Google Scholar
- Glass GV, McGaw B, Smith ML (1981) Meta-analysis in social research. Sage, LondonGoogle Scholar
- Hattie JAC (2009) Visible learning: a synthesis of over 800 meta-analyses relating to achievement. Routledge, AbingdonGoogle Scholar
- Heinlein R (1957, August) The menace from earth. The Magazine of Fantasy & Science Fiction. http://www.baen.com/Chapters/0743498747/0743498747___2.htm
- Huang Y-M, Liao Y-W, Huang S-H, Chen H-C (2014) A Jigsaw-based cooperative learning approach to improve learning outcomes for mobile situated learning. Educ Technol Soc 17(1):128–140Google Scholar
- Karasavvidis I (1999) Learning to solve correlational problems: a study of the social and material distribution of cognition. PhD thesis, University of Twente, Enschede, The Netherlands. https://ris.utwente.nl/ws/files/6076012/t000000a.pdf
- Koehler MJ, Mishra P (2008) Introducing TPACK. In: The AACTE Committee on Innovation and Technology (ed) Handbook of technological pedagogical content knowledge for educators. Routledge for the American Association of Colleges for Teacher EducationGoogle Scholar
- Kozma RB (2011) A framework for ICT policies to transform education. In: Transforming education: the power of ICT policies. United Nations Educational, Scientific and Cultural Organization, Paris. http://unesdoc.unesco.org/images/0021/002118/211842e.pdf
- Maple (18-2016) Maplesoft, a division of Waterloo Maple Inc., Waterloo, Ontario. www.maplesoft.com
- Pearson P D, Ferdig R, Blomeyer RL, Moran J (2005) The effects of technology on reading performance in the middle-school grades: a meta-analysis with recommendations for policy. Learning Point Associates/North Central Regional Educational Laboratory (NCREL). https://files.eric.ed.gov/fulltext/ED489534.pdf
- Puentedura R (2012) Thinking about change in learning and technology. Presentation given September 25, 2012 at the 1st Global Mobile Learning Conference, Al Ain, UAE. http://www.hippasus.com/rrpweblog/archives/2012/04/10/iPad_Intro.pdf
- Robinson N (2018) The NAPLAN results are in: here’s what you need to know. Australian Broadcasting Corporation News. http://www.abc.net.au/news/2018-08-28/naplan-results-are-in-and-heres-what-you-need-to-know/10169934
- Rogers EM (2003) Diffusion of innovations, 5th edn. Free Press, New YorkGoogle Scholar
- Statistica (2018) The 100 largest companies in the world by market value in 2018 (in billion U.S. dollars). https://www.statista.com/statistics/263264/top-companies-in-the-world-by-market-value/
- Tallmadge GK (1977) The joint dissemination review panel IDEABOOK. U. S. Office of Education, Washington, DCGoogle Scholar
- Tamim RM, Borokhovski E, Pickup D, Bernard RM, El Saadi L (2015) Tablets for teaching and learning: a systematic review and meta-analysis. Commonwealth of LearningGoogle Scholar