Multimodal literacy

The changing nature of the communicative landscape wrought by new technologies is accentuating the need to broaden our definition of literacy. Kress (2003: 1) argues that literacy in the new media age will involve “human, cognitive/affective, cultural and bodily engagement with the world and on the forms and shapes of knowledge”. He proposes a shift away from an alphabetic literacy and to explore the new forms of literacy needed in today’s world. To be considered literate in this day and age is to be able to effectively communicate multimodally. The “multimodal turn” (Jewitt 2009: 4) is a recognition that language is usually co-deployed with other semiotic resources and meaning is made multimodally as a result of the orchestration of these resources. Language has come to be understood “not as some discreetly independent entity, but rather as part of complex sets of interconnecting forms of human semiosis” (Christie 2002: 3). Communication, especially with multimedia and social media, involves not just language, but also the use of multimodal resources, such as images, videos, embodied action, and three-dimensional objects to make meanings in different contexts (Smith et al. 2014). As such, students need to develop complementary competencies not only in traditional domains, such as literacy and numeracy, which remain foundational, but also develop a fluency in multimodal literacy (Lim & Hung, 2016). In this light, the literacy curriculum today cannot remain as a language-focused curriculum. The literacy curriculum must evolve beyond just the teaching and learning of reading in print to reading both in print and on screen; from that of reading of books to that of reading of books and viewing of multimodal texts critically; from writing to writing and typing; from speaking to speaking and representing.

Multimodal literacy (Jewitt and Kress 2003; O’Halloran and Lim 2011; van Leeuwen 2017) is about students learning to view multimodal texts critically and to communicate effectively through multimodal representations. Multimodal literacy focuses on the distinct “epistemological commitment” and functional specialisation of each meaning-making resource (Kress 2003: 55). The unique affordances of the semiotic resources bring about their functional specialisations, for instance, “writing is better for representing events in sequence, and image is better for representing relation of elements in space” (Kress 2003: 46). Multimodal literacy is about understanding the affordances, that is the potentials and limitations, of the different meaning-making resources, as well as how they work together to produce a coherent and cohesive multimodal text.

In order to view multimodal texts critically, students must understand how meanings are made across the semiotic resources and be able to cite textual evidence to support their interpretation of the multimodal texts. In order to communicate effectively through multimodal representations, students must also have an awareness of the affordances of various semiotic resources and how they can work together effectively to make meaning. The importance of developing multimodal literacy in our students is increasing gaining recognition by policy makers and curriculum planners internationally. Singapore, in particular, has introduced the viewing and representing of multimodal texts as new areas of language learning in the Ministry of Education English Language Syllabus as early as in 2010. With the recognition of the importance of multimodal literacy, the next question to be answered is how to effectively develop multimodal literacy in students. With the inclusion of viewing and representing as part of students’ literacy development, there is a corresponding need to build competencies in teachers to be able to teach multimodal literacy. This article describes an instructional approach with content informed by systemic functional theory, and pedagogy aligned to the Learning by Design framework widely used in multiliteracies (Cope and Kalantzis 2015). The aim of the approach is to provide teachers with the meta-language, pedagogical scaffolds, and resources to effectively teach the viewing and representing of multimodal texts.

Systemic functional approach to teaching multimodal literacy

Given that students are to appreciate the affordances of various semiotic resources and how they work together in a multimodal text so as to view critically, a way to develop multimodal literacy is to have students perform analyses of multimodal texts. Through their evaluation, students learn to identify the choices made in the multimodal texts and can explore the ways in which the meanings are made. Within the field of multimodal research, there have been many approaches to the analysis of multimodal discourse developed over the years. Jewitt et al. (2016) survey the landscape of multimodal research and identify the three main approaches to multimodality as systemic functional linguistics (O’Toole 2011; Bateman 2014; O’Halloran and Lim 2014), social semiotics (Kress and van Leeuwen 2001, 2006; Adami 2009; Kress 2010a, 2010b) and conversation analysis (Goodwin 2000; Antaki 2011; Broth and Mondada 2013).

Both the systemic functional linguistics and social semiotics approaches to multimodality are inspired by the work of Michael Halliday (1978) who viewed language as social semiotic and developed systemic functional theory as an approach to study language. The systemic functional approach linguistics approach, described in this article, as systemic functional approach, adopts a genre-based orientation towards multimodality and is organised around the metafunctional meanings, that is the experiential meanings (happenings through processes, participants, and circumstances), interpersonal meanings (engagement and expression of modality), and textual meaning (organisation of parts). The systemic functional approach focuses on the ‘grammar’ of multimodal texts by studying how each semiotic resource contributes to the emergent meaning through ‘system networks’, as well as the interaction and integration of these resources as a multimodal whole. Notwithstanding, the systemic functional approach could be criticised for holding up language as a reference, in which the principles of meaning-making by other semiotic resources are made. This, however, can be useful in the case of teaching multimodal literacy, in the context of introducing English Language teachers to the principles of meaning-making in other semiotic resources by referencing language which they already familiar with.

The social semiotic approach to multimodality focuses on the ‘agency’ and ‘interest’ of the meaning-maker as well as the ‘design’ of multimodal text through studying the ‘affordances’ of each mode, that is the culturally shaped and socially organised set of semiotic resources for meaning -making. The social semiotics approach is driven by the aim to “explore the common principles behind multimodal communication” (Kress and van Leeuwen 2001: 2). Criticisms of the social semiotic approach include that its’ principles could be overgeneralised by not being sufficiently contextually nuanced and that it seem not to address the hybridised nature of contemporary communication (Prior 2005). Notwithstanding, the social semiotic approach has been productively advanced as a theory of learning (Kress and Bezemer 2016) and invites possibilities for application as a pedagogy for the teaching of multimodal literacy which presents opportunities for future study.

The conversation analysis approach to multimodality stems from the study of talk in social interaction. It focuses on the analysis and transcription of videos to study human action, recognising that multimodal resources are situated and made relevant by the participants of the discourse themselves (Mondada 2018). Criticisms of the conversation analysis approach centres on the question of how the ‘relevance’ of the data analysed is determined (Blommaert 2005) as well as a limited focus on how the multimodal ensemble work together to make meaning in combination. Given that a focus of the conversation analysis approach is on human interaction through video analyses and transcriptions, it appears less relevant for the instruction of multimodal literacy in students.

Of the three main approaches, the systemic functional approach is applied to inform the development of the instructional approach described in this article. It is deemed best fit for purpose because the functional linguistic approaches to literacy are familiar to teachers in Singapore, and that they have used genres, described as ‘text-types’, as a way to orientate and organise students’ learning of language. The systemic functional approach to develop multimodal literacy leverages teachers’ existing knowledge of teaching language and references the principles in the introduction of the other semiotic resources. This is more relatable and accessible for teachers, especially since the teaching of multimodal literacy in Singapore is intended to take place in the English Language classroom. The systemic functional approach has also been productive in its applications across educational-related domains such as “educational semiotics, computational multimodal studies, multimodal translation and multimodal literacy” (Jewitt et al. 2016: 55).

While first, and most famously, applied to describe language, it must be emphasised that systemic functional theory is not just a theory of language. Rather, as Halliday (1978) elucidates, systemic functional theory is a theory of meaning; a semiotic theory, that has been most extensively applied to language. The central perspective offered by systemic functional theory is that meaning-making is a result of choice. This choice is “not a conscious decision made in real time but a set of possible alternatives” (Halliday 1994: xiv–xxvi) in the design and composition of a text. The principles extrapolated from the systemic functional theory to the other semiotic resources stem from the premise of metafunctional meanings made through semiotic selections within a system network. The systemic functional approach emphasises explicit teaching of the generic features of these texts and introduces the strategies used to engage viewers. The systemic functional approach also offers a meta-language for students to discuss and describe the modes and meanings in multimodal texts. In the study and analysis of modes beyond language, a meta-language is necessary to “describe meaning in various realms” (Cope and Kalantzis 2000: 24). Unsworth (2014: 13) argues that “teachers and students need this kind of metalanguage for talking about language, images, sound, and so forth, and for their meaning-making interactions”. The meta-language introduced in the systemic functional approach stems from the theoretical view of genre and multimodality. It organises the student’s viewing around the meaning potential in the multimodal texts to achieve specific purposes - that is how the textual features fulfil typical functions. It also orientates the students towards the semiotic choices made in the multimodal texts – that is the media strategies and typical effects of these selections on the engagement of the viewer. With the meta-language, students are empowered to describe and discuss the representations made in the texts. The approach thus aims to equip “students with the understanding to know where to look and what to look out for in a [multimodal] text” (Lim and Tan 2017: 182).

Pedagogical features of the systemic functional approach

The pioneering work in broadening the traditional definition of literacy to a more encompassing understanding is attributed to the New London Group’s work on multiliteracies (Cope and Kalantzis 2000). The ‘multi’ stands for the “enormous and significant differences in contexts and patterns of communication… and the ‘multi-’ of multimodality” (Cope and Kalantzis 2015: 3). The distinction between multiliteracies and multimodal literacy is subtle. It has been observed that multiliteracies, notwithstanding its plural form, along with the new emergent literacies, tend to be “strongly focused on competencies and written lettered representation” (Jewitt 2007: 245). Multimodal literacy, on the other hand, draws attention to the other modes of meaning-making beyond language and focuses on the specific meanings made in the viewing and representing of multimodal text as well as how semiotic resources are co-deployed to package information in the text.

For the teaching of multiliteracies, Cope and Kalantzis (2015) have developed a Learning by Design framework where they introduce the knowledge processes of Experiencing, Conceptualising, Analyzing and Applying in multiliteracies, summarised in Table 1. The pedagogical features of the systemic functional approach, introduced in this article, is aligned to Cope and Kalantzis’ (2015) Learning by Design Framework and adopts the “reflexive pedagogy” described by them. Reflexive pedagogy combines insights and practices from both didactic and authentic pedagogy and is positioned as a pedagogy of communication and knowledge representations. The reflexive pedagogy recognises that “knowledge is not (just) the stuff that ends up in our minds. It is what we do and make. Learning is a consequence of a series of knowledge actions, using multimodal media to externalize our thinking… Learning is also very social, as we rely on the artifacts of collective memory, and work with others in the essentially collaborative task of knowledge making” (Cope and Kalantzis 2015: 39).

Table 1 Learning By Design Framework. Adapted from Cope and Kalantzis (2015)
Full size table

The essence of multimodal literacy is the ability for students to be able to view multimodal texts critically and to make multimodal representations effectively. The pedagogical features of the systemic functional approach, represented in Fig. 1, are to introduce students to the texts by their genres, that is the social purpose of the texts, and guiding them to identify specific features in the text-types, as well to use a corpus of authentic texts, rather than artificially constructed ones. Lessons are designed for students to engage in collaborative learning inquiry as they experience and evaluate the carefully selected multimodal texts that exemplify the genre. Teachers also engage in explicit teaching of the features of the texts as well as the media strategies used to realise specific meanings, such as prominence, power, address, and distance, in the multimodal texts. Through the ‘reflexive pedagogy’ of blending both inquiry-based learning and didactic teaching, an inductive learning approach is achieved, where students are to make meaning from the texts collaboratively even as they are taught the meta-language to describe and discuss multimodal texts. The systemic functional approach also harnesses the use of educational technology (EdTech) to facilitate the collaborative annotation and joint meaning-making of students, as well as support the making of artefacts, through appropriate digital creation platforms. The experience of artefact making is integral to the development of multimodal literacy, as it not only provide students to represent their learning, but also provide an opportunity for students to learn through making. The pedagogical features of the systemic functional approach to develop multimodal literacy in students are also correspondingly related to the dimensions and knowledge processes in the Learning by Design Framework in Table 2.

Fig. 1
figure 1

Pedagogical Features in the Systemic Functional Approach to Teaching Multimodal Literacy

Full size image
Table 2 Learning by Design Framework with Pedagogical Features of Systemic Functional Approach
Full size table

Situated practice

The focus in the dimension of Situated Practice is on the immersion of learners in the everyday world and helping them make connections across the domains of education and daily life. As such, the pedagogical features of the systemic functional approach proposed are that authentic multimodal texts are used and are introduced according to their genre, such as print advertisements, films, and online news. Based on their genres, there are specific features and typical functions that these features serve. Students are also taught to recognise the obligatory and optional features of the various text-types. This foregrounds the communicative purpose of the multimodal texts and how the semiotic choices are organised towards fulfilling the purpose. Authentic text that students encounter in their daily lives are used to strengthen the connection between theory and practice so that students can see the application of what they are learning in the classroom.

Students also learn together with their peers through joint meaning-making. Collaborative learning inquiry can be done though whole class discussion, small group discussions as well as collaborative annotations and discussions enabled through educational technology platforms. Through collective negotiation of meaning, argumentation and citation of evidence to defend their interpretation, students develop a more robust analysis of the multimodal texts they view and a deeper appreciation of the multimodal composition.

Overt instruction

Following the reflexive pedagogy, where useful practices from didactic pedagogy are applied, the explicit teaching of the terms and concepts through the introduction of the meta-language for the specific multimodal text is a feature of the systemic functional approach. The focus in the dimension of overt instruction is for students to “learn to use abstract, generalising terms through drawing distinctions, identifying similarities and differences, and categorizing with labels” (Cope and Kalantzis 2015: 19). As such, explicit teaching provides students with a guided understanding of the textual features and engagement strategies used in the multimodal text to persuade viewers. Overt instruction can take the form of questioning by the teacher to prompt evidence-based interpretations from the students. Through the use of questioning, teachers guide students in making explicit the students’ tacit understanding of the visual texts. Having been taught the terms and concepts, students are empowered to identify textual features, describe the semiotic choices and discuss the meanings made in the multimodal texts as well as cite textual evidence to support their interpretation of the multimodal text.

Critical framing

Critical framing is achieved through inductive learning, where students are guided in the viewing and analysis of multimodal texts so as to identity features and patterns that are later consolidated and explicitly taught by the teachers as concepts. Students work across a selection of teacher-curated multimodal texts, and are guided through an inductive process to identify and relate the textual features to the typical functions they serve, as well as surface and associate the multimodal strategies to the typical effects they realise. The process where students interrogate the texts and “examine cause and effect, structure and function” helps them to “develop chains of reasoning and explain patterns” (Cope and Kalantzis 2015: 20) which is later affirmed and reinforced through the explicit teaching by the teacher. Through this inductive learning, students are guided to view the multimodal text critically and are empowered to interrogate the “world of subjectivity (human agency, interest and intent)” (Cope and Kalantzis 2015: 20).

Transformed practice

The dimension of Transformed Practice is about students “learning by applying experiential, conceptual or critical knowledge” (Cope and Kalantzis 2015: 21). This involves the students making artefacts as a way of learning and a demonstration of their learning. The effective composition of a multimodal representation indicates that students have understood how the semiotic choices work together in organising the text and orientating the viewer to specific communicative intent. The systemic functional approach also harnesses educational technology to facilitate students’ making of multimodal representations. The annotation and analysis of multimodal texts are performed through the use of educational technology platforms, such as those with collaborative input features.

Instructional content in the systemic functional approach

As the instructional content in the systemic functional approach has been described in Lim and Tan (2017) for the genre of print advertisements, and in Lim and Tan (2018) for the genre of film text, the description of the instructional content in this article is brief. The instruction content is the meta-language suggested for the teachers to use for specific genres of texts. In this article, the instructional content for the teaching of print advertisement is represented in Fig. 2. The instructional content is designed to be implemented across a series of lessons, which is reinforced each time with the introduction of a new genre of multimodal texts. In other words, each time a new text-type is introduced, the similar concepts learnt in the previous series are revisited, with additional features pertinent to the specific genre added.

Fig. 2
figure 2

Instructional Content of Lessons for the Viewing of Print Advertisments

Full size image

The instructional content for the teaching of multimodal literacy are organised in the specific lessons on 1) Form, 2) Engagement, 3) Message, and 4) Integration (Fig. 2). Thereafter, the students may continue with the analysis and evaluation of more texts, and begin creating their multimodal representations.

Lesson on form

The lesson on Form is about helping students acquire knowledge of the genre of the text through the textual features and the typical functions they serve. For instance, the teacher may show that in an image, the Main Visual Display is the largest and most prominent part of the visual text, the Focus of Attention stands out the most and the Logo is the graphic representation of company. Linguistic features include Slogan, as a memorable catch phrase, the Call and Visit information, which provides the contact details, as well as features such as the Brand and Product names and the Call to Action (Tan et al. 2012).

Lesson on engagement

The lesson on engagement is about helping students to develop a sensitivity to the multimodal strategies and their typical effects. The teacher may introduce the common media strategies so as to understand why they are used to achieve specific purposes. For instance, in realising address, students learn that the subjects’ type of gaze—looking directly at or away from the viewer—changes the way the viewer interacts with them. In realising power, students also learn to recognise media strategies, such as the use of vertical angles to realise power (Kress and van Leeuwen 2006).

Lesson on message

The lesson on Message is about helping students to understanding literal and inferential representations in the text as well as how the text presents its ideas. Teachers build on what students have learnt in Form and Engagement, and guide them in identifying the types of persuasion used to appeal to the viewer as well as discuss the literal and inferential meanings made in the multimodal texts. Students learn to recognise how appeals are made through authority, reason, and emotion (Halmari and Virtanen 2005). Students also discuss the literal and inferential meanings in the multimodal texts and consider the interest(s) of the text producers as well as the context of production and reception of the multimodal text.

Lesson on integration

In the lesson on integration, students examine the relationship between the language and the visuals in terms of similar or different meanings made by the various semiotic resources. They consider the similar meanings made across language and images to reinforce the message, as well as the apparently contradictory meanings made across language and images to infuse a layer of irony and play in the multimodal texts (Lim and O’Hallloran 2012). Students are led to appreciate the interaction between form and content in multimodal texts, and how the integration of the linguistic and visual elements can lead to a coherent emergent meaning (Lim 2004) and thereby achieving the intended purpose(s).

Trial of the systemic functional approach

Context

The systemic functional approach to the teaching of multimodal literacy was trialled in a secondary school in Singapore. The government (public) secondary school is located in the western part of Singapore, in a typical neighbourhood surrounded by public housing. Three classes of secondary one students (13 year old students) were involved in the trial, with one class serving as the control class. The students from the three classes are considered to be of general average ability by national standards as they are in the Express stream, where students are on a four-year course learning up to the Singapore-Cambridge GCE O-Level examination. Notwithstanding, in the context of this school, a more granular distinction based on their grades from the national assessment (Primary School Leaving Examinations) is made across students from the 3 classes - with class 1E1 comprising students with relatively higher grades; class 1E2 comprising students of slightly lower grades than students from 1E1; and class 1E3 comprising students with slightly lower grades than students from 1E3.

Methods

The researchers worked with two teachers, Clarice and Sharon (pseudonyms), from the school to implement the systemic functional approach to teach multimodal literacy. Clarice taught 1E1, a class of 40 students, and Sharon taught 1E3, a class of 40 students. The control class, 1E2, with 42 students, continued with the regular lesson activity and were not introduced to the systemic functional approach. The research instruments used included 1) pre and post-tests, 2) students’ survey, 3) students’ free response questions, 4) students’ artefacts, and 5) teachers’ reflections. Students from all three classes took a pre-test and a post-test, but only students from the two intervention classes were surveyed and had the students’ artefacts collected by the researchers for analysis.

In the discussion with the researchers, Clarice and Sharon were keen to explore how they could achieve the syllabus outcomes through the use of the systemic functional approach to develop multimodal literacy in their students. The teachers also wanted to better guide their students in identifying and describing the features in multimodal texts so that they would be able to evaluate the texts and support their interpretations with textual evidence. Print advertisements were identified to be the genre of multimodal texts to be taught. This was, in part, motivated by the national English Language examinations, where students’ understanding of print advertisements were assessed.

The researchers worked with Clarice and Sharon to co-design the teaching resources based on the systemic functional approach. The researchers also conducted consultation sessions with the teachers to help them understand the pedagogical features and the instructional content of the lessons. The teachers then conducted the four one-hour lessons, described earlier, for their classes, and assigned the production of students’ artefacts, as an assignment to be completed during the one-week school holidays.

Pre and post tests

For the pre and post-tests, students were to answer four questions based on the print advertisement shown in Fig. 3. They were 1) What is the purpose of the advertisement? 2) Who is the target audience? 3) What is the main message in the advertisement? 4) What are the strategies used to convey the message? The teachers marked the students’ responses and analysed the students’ scores. It was noted that while students generally were able to answer the first two questions well, many students had difficulties with question 3, where they had to infer the main idea in the advertisement, and most students had difficulties with identifying the media strategies used in the advertisements. This was consistent in both pre-test and post-test, although there was improvements in the intervention classes in the post-test. The results of the pre and post-test are in Table 3.

Fig. 3
figure 3

Print Advertisements Used in Pre and Post Tests

Full size image
Table 3 Class Mean Scores from Students’ Pre and Post-Tests
Full size table

While the improvement in the intervention classes was not great, likely due to the short nature of the intervention of only four lessons, there improvement was discernible. What is of interest is the decline in the mean scores of students from the control class in the post-test. While the reason for this is uncertain, the researcher speculated that while efforts have been taken to ensure that the print advertisements were similar and that the questions asked were the same, it was possible that students in all three classes found the print advertisement used in the post-test more challenging to understand.

Students’ survey

The students were also given the following statements in a survey, in which they were to indicate whether they strongly agreed, agreed, disagreed, or strongly disagreed.

  1. 1)

    I enjoyed the lessons on understanding visual texts (advertisements).

  2. 2)

    I am interested to learn more about how advertisements are designed.

  3. 3)

    I pay more attention to advertisements now.

  4. 4)

    I am able to identify the techniques used in visual texts to achieve a variety of purposes.

  5. 5)

    I am able to gather evidences to support my interpretation of the visual texts.

The results of the students’ survey are presented in Table 4. The first three statements are related to the design of their learning experience, that is the pedagogical features in the systemic functional approach. Most students (92.3% from 1E1 and 92.29% from 1E3) indicated that they have enjoyed the lesson series and a slightly smaller group, though still a majority, of students (84.6% from 1E1 and 87.23% from 1E3) reported that the lessons have piqued their interests to learn more about how advertisements are designed. It is noted, however, that the number of students who strongly agreed that they are interested to learn more about how advertisements are designed are much lower at 5.13% from 1E3, as compared to 33.3% from 1E1.

Table 4 Results from the Students’ Survey
Full size table

Most students (84.6% of students from 1E1 and 61.53% of students from 1E3) also indicated that they will pay more attention to advertisements now, although the number of students disagreeing from 1E3 at 38.5%, as compared to 15.4% from 1E1, is notably high. This could suggest that the connection and transfer of what they have learnt to what they experience in their daily lives can still be strengthened. The last two questions are related to the instructional content in the systemic functional approach and provide insights into the students’ development as critical viewers. Most students (94.9% from 1E1 and 92.29% from 1E3) indicated that they have learnt to identify the techniques used in visual texts to achieve a variety of purposes and a slightly smaller group, but still a majority, of students (89.7% from 1E1 and 89.8% from 1E3) reported that they were able to gather evidences to support their interpretation of the visual texts. The results were similar across both intervention classes, with the higher ability students from 1E1 reporting a marginally higher positive response in strongly agreeing and agreeing with the statements. It is useful to note that while most students have reported that they have enjoyed and have learnt through the lessons, and that their application of learning could be observed in the artefacts they designed, there were also a minority of students who disagreed. This suggests that the instructional approach implemented could still be improved. As the survey was anonymous to encourage honest feedback, it was not possible to follow up with the students who disagreed to have a deeper understanding of their reasons, which would otherwise have been illuminating. In later studies, an additional question eliciting for suggestions for improvements from students were added.

In the free response questions at the end of the survey, students were asked what they had learnt. Examples of responses include being able to “uncover deeper meanings”, learning to “identify techniques used to achieve a variety of purpose”, knowing “how to make a good and interesting advertisement”, and that “every single information on the poster is important and contributes greatly to the advertisement”. Students were also asked in what ways their learning would be useful in examinations. Examples of responses included understanding “what the questions are asking and the requirements of the questions”, knowing “how to read an advertisements”, and scoring “higher marks”. Finally, students were asked in what ways their learning would be useful in life. Examples of responses included, being “more discerning”, knowing “what information to look out for and where to look”, and – our favourite response - “I will not be tricked”.

For the production of students’ artefacts, students were tasked to work in groups to produce a print advertisement on the theme of travel. Examples of their products are shown in Fig. 4 below. The students’ design of the travel advertisement showed an application of their learning. For instance, the use of size and positioning to realise the prominence of the globe, and the inclusion of the Call and Visit Information as part of the feature of print advertisements, are evident in Fig. 4.1. The use of reason as an appeal is evident is Fig. 4.2, where again size and positioning of the text box are used effectively to direct the viewers’ attention. Fig. 4.3 shows an advertisement for luggage and uses an image of Justin Bieber, a popular celebrity with youths, as endorsement to realise an appeal to authority. It also uses a secondary appeal to reason by offering discount and a description of the positive attributes of the luggage as being “light and strong”. Overall, the students’ design of the artefacts displayed an understanding of the features and strategies used in print advertisements. In retrospect, what would have been useful, but was not done in the trial, was to have the students also present, either orally or in the written mode, their rationale for their design choices. This was also done in later studies where students completed an additional worksheet explaining their design choices.

Fig. 4
figure 4

Examples of Students’ Artefacts

Full size image

Teachers’ reflections

In the teacher’s written reflections after the lessons, both Clarice and Sharon felt that the systemic functional approach was a useful way to teach visual texts. Clarice shared that “it was really eye-opening for everyone to actually realise that there is so much thought and meaning behind making a simple advertisement.” She felt that “exposing the students to different kinds of advertisements generally strengthened their interest too as people generally response better to visuals, especially with the use of images and colour.” Sharon shared that she had concerns initially with “the terminology (i.e. words like “function”) as well as the application of the techniques and students’ receptiveness to the various systems” and wondered if was “too much of an information overload for the students”, especially for her students in 1E3. However, she noted that “after simplifying the systems and their terms, [she] was confident that the students would be able to understand”.

The teachers also felt that the lessons also involved just the right degree of challenge for the students – overall, they felt that the lessons were neither too easy nor difficult. Sharon observed that “the series of lessons were well-received by the students and based on verbal feedback, [the] majority of students found the lessons stimulating and refreshing.”

In terms of improvement, Clarice felt that “it might be better if the advertisements chosen as well as the questions formed in the worksheets were geared more towards the exam syllabus, as the advertisements [used in the lessons] tend to be slightly more pictorial and some of the tougher questions may not actually come out for exams.” Nonetheless, she noted that “what’s best [was] that the students actually learned life skills. They generally understand that there is more to advertisements than just learning about them for examinations.”

From their reflections, both teachers expressed positive sentiments about the systemic functional approach, both in its utility and effectiveness. While there were initially concerned with the extent of meta-language, they were heartened that the students were able to cope with them during the lessons and that most students found the lessons enjoyable. They also noted that what was taught in the lessons was probably more advanced and beyond what was required in the syllabus and what would be tested in the examinations. However, the teachers also saw the need and value of developing multimodal literacy in students, as part of the life-skills students would need to thrive in the contemporary digital environment.

Conclusion

This article describes the development of a pedagogical approach, that is informed by systemic functional theory, to develop multimodal literacy in secondary school students in Singapore. The pedagogical features of the approach is aligned to the Learning by Design framework, used widely within the field of multiliteracies. The pilot trial to implement the systemic functional approach with two teachers in two classes is also described in the article. The trial was productive in refining the systemic functional approach to develop multimodal literacy in students. The instructional content was worked through with the teachers, who identified and curated relevant lesson materials, such as examples of visual texts that would be of interest to their students. The pedagogical features of the approach were discussed and negotiated with the teachers into what were practicable within the constraints of a tight curriculum. To prevent an overload of terminology, the meta-language to be introduced through the approach was scrutinised and deliberated, and where possible, aligned to terms already used in the teaching of language, to ensure that each new term introduced was necessary and helpful for the students to use for their description and discussion of multimodal texts. Admittedly, it can be difficult to teach secondary school students multimodal grammar. As such, in our discussion and negotiation with the teachers on the extent of meta-language to be introduced and used, we have decided not to bring in visual transitivity at this stage. There was a greater focus on the interpersonal meanings relative to ideational and textual meanings made multimodally. Nonetheless, we have attempted to introduce the main ideas behind salience and information value by discussing the media strategies used to realise them in a print advertisement. The article reports on what has been developed and used to teach students on multimodal literacy in a pilot trial. The work continues and we are exploring how aspects of systems of the multimodal grammar realising ideational and textual meanings can be introduced more effectively in the future.

The positive findings from the trial affirm that multimodal literacy can be taught and that the systemic functional approach described in this article offers a way to design for its learning. Following this pilot trial, the approach has been further refined and implemented in a number of secondary schools in Singapore. With every implementation, iterations were made to instructional strategies and the lesson resources co-created with the teachers involved. Results of the trials with schools, over the years, and well as the description of the approach have been reported in Lim et al. (2015), Lim and Tan (2017, 2018). Starting from the multimodal genre of print advertisements, the approach has been extended to films, and more recently to online news. Workshops have also been conducted to train Singapore teachers in this instructional approach, with lesson resources made available for them to adopt and adapt accordingly. Lim et al. (in press) also describe a networked learning community formed by teachers who have trialled this instructional approach and are united by their common interest in developing multimodal literacy in their students.

The work on developing an approach, informed by systemic functional theory, for the teaching of multimodal literacy, is inspired by Michael Halliday’s clarion call towards an ‘appliable linguistics’ (Mahboob and Knight 2010), where “the value of a theory lies in the use that can be made of it”. (Halliday 2006: 192). In light of the changing communicative landscape which our students today inhabit and the need to equip them with the literacy needed to make meaning from multimodal texts and make multimodal texts, the literacy curriculum needs to be expanded to include multimodal literacy. Teachers, at the frontline of equipping our students in the classrooms, need to be supported with theoretically-grounded instructional strategies to develop multimodal literacy in our students. Advances made in the field of multimodality present fertile understandings that could potentially be cascaded into the classrooms. The systemic functional approach described in this article represents an attempt to translate these concepts into an instructional approach for the teaching of multimodal literacy.

Almost two decades ago, Kress (2000: 161) had declared that “it would be an unforgivable dereliction of the responsibilities of intellectuals if the potentials of representation and communication— of literacy in a very broad and metaphoric sense— offered by current developments were not fully explored, and a concerted attempt made to shape their direction”. As described earlier in this article, much progress has been made in the field of multimodality in recent years. Research in multimodality presents exciting possibilities, with promises to influence pedagogy, inform a forward-looking literacy curriculum and develop multimodal literacy in our students. The development of the systemic functional approach for the teaching of multimodal literacy, described in this article, represents an attempt to contribute to the ongoing conversation on how best to prepare students for this digital age.