Abstract
This chapter introduces the five most prominent approaches to multimodal data analysis: Multimodal (Inter)action Analysis, Mediated Discourse Analysis, Systemic Functional Multimodal Discourse Analysis, Social Semiotics, and Multimodal Conversation Analysis. The chapter begins by discussing the origin of each approach, methods of data collection and analysis, and embedded theoretical foundations. The theoretical differences between the approaches are then examined with a focus on the way each approach treats the individual, the artefact, and the notion of mode during analysis. With the variety of available approaches to multimodal data analysis, it is important that researchers link data collection and analysis to coherent theoretical underpinnings.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Al Zidjaly, N. (2009). Agency as an interactive achievement. Language In Society, 38(2), 177–200.
Baldry, A., & Thibault, P. J. (2006). Multimodal transcription and text analysis: A multimedia toolkit and associated on-line coursebook. London and Oakville: Equinox.
Bernad-Mechó, E. (2017). Metadiscourse and topic introductions in an academic lecture: A multimodal insight. Multimodal Communication, 6(1), 659.
Deppermann, A. (2013). Multimodal interaction from a conversation analytic perspective. Journal of Pragmatics, 46(1), 1–7.
de Saint-Georges, I. (2014). Mediated discourse analysis, “embodied learning” and emerging social and professional identities. In C. Maier & S. Norris (Eds.), Interactions, images and texts (Vol. 11, 1st ed., pp. 347–356). Boston, MA: De Gruyter.
Eggins, S. (2004). Introduction to systemic functional linguistics (2nd ed.). London: A&C Black.
Forceville, C., & Urios-Aparisi, E. (2009). Multimodal metaphor. Berlin: Walter de Gruyter.
Geenen, J. (2017). Show and (sometimes) tell: Identity construction and the affordances of video-conferencing. Multimodal Communication, 6(1), 1–18.
Goffman, E. (1959). The presentation of self in everyday life. Gloucester, MA: Peter Smith Pub Incorporated.
Goffman, E. (1981). Forms of talk. Philadelphia, PA: University of Pennsylvania Press.
Goodwin, C. (1979). The interactive construction of a sentence in natural conversation. In G. Psathas (Ed.), Everyday language: Studies in Ethnomethodology (pp. 97–121). New York: Irvington Publishers.
Goodwin, C. (2000). Action and embodiment within situated human interaction. Journal of Pragmatics, 32(10), 1489–1522.
Gumperz, J. J. (1982). Discourse strategies. Cambridge: Cambridge University Press.
Haddington, P., Mondada, L., & Nevile, M. (2013). Being mobile: Interaction on the move. In P. Haddington, L. Mondada, & M. Nevile (Eds.), Interaction and mobility, language and the body in motion. Berlin, Boston: De Gruyter.
Halliday, M. A. K. (1978). Language as a social semiotic: The social interpretation of language and meaning. Baltimore, MD: University Park Press.
Hamilton, H. E. (1994). Conversations with an Alzheimer’s Patient: An interactional sociolinguistic study. Cambridge: Cambridge University Press.
Hart, C. (2016). The visual basis of linguistic meaning and its implications for critical discourse studies: Integrating cognitive linguistic and multimodal methods. Discourse & Society, 27, 335–350.
Jewitt, C. (2006). Technology, literacy and learning: A multimodal approach. New York: Psychology Press.
Jewitt, C. (2013). Multimodal methods for researching digital technologies. In The SAGE handbook of digital technology research. Thousand Oaks, CA: Sage.
Kress, G. (1999). Multimodality. In B. Cope & M. Kalantzis (Eds.), Multiliteracies: Literacy learning and the design of social futures. New York, NY: Routledge.
Kress, G., & van Leeuwen, T. (1996). Reading images: The grammar of visual design. New York, NY: Routledge.
Kress, G., & van Leeuwen, T. (2001). Multimodal discourse: The modes and media of contemporary communication. London: Edward Arnold.
Kress, G., & van Leeuwen, T. (2002). Colour as a semiotic mode: Notes for a grammar of colour. Visual Communication, 1(3), 343–368.
Kress, G., & van Leeuwen, T. (2006). The semiotic landscape. In P. Griffiths, A. J. Merrison, & A. Bloomer (Eds.), Language in use—A reader (pp. 344–350). London: Routledge.
van Leeuwen, T. (2005). Introducing social semiotics. New York, NY: Psychology Press.
Machin, D., & Mayr, A. (2012). How to do critical discourse analysis: A multimodal introduction. London: SAGE.
Matelau, T. (2014). Vertical identity production and Maori identity. In S. Norris & C. Maier (Eds.), Interactions, texts and images: A reader in multimodality (Vol. 1, 1st ed., pp. 255–266). New York: Mouton de Gruyter.
Mondada, L. (2006). Participants’ online analysis and multimodal practices: Projecting the end of the turn and the closing of the sequence. Discourse Studies, 8(1), 117–129.
Mondada, L. (2009). Emergent focused interactions in public places: A systematic analysis of the multimodal achievement of a common interactional space. Journal of Pragmatics, 41(10), 1977–1997.
Mondada, L. (2013). The conversation analytic approach to data collection. In J. Sidnell & T. Stivers (Eds.), The handbook of conversation analysis (pp. 210–228). Oxford: Wiley-Blackwell.
Norris, S. (2004). Analyzing multimodal interaction: A methodological framework. London: Routledge.
Norris, S. (2011). Identity in interaction: Introducing multimodal interaction analysis. London: Mouton de Gruyter.
Norris, S. (2013). What is a mode? Smell, olfactory perception, and the notion of mode in multimodal mediated theory. Multimodal Communication, 2(2), 155–169.
Norris, S. (2014). Learning tacit classroom participation. WCTLA. Procedia Social and Behavioral Sciences. Amsterdam: Elsevier.
Norris, S. (forthcoming). Systematically working with multimodal data: Research methods in multimodal discourse analysis. Hoboken, NJ: John Wiley and Sons.
Norris, S., & Pirini, J. P. (2017). Communicating knowledge, getting attention, and negotiating disagreement via videoconferencing technology: A multimodal analysis. Journal of Organizational Knowledge Communication, 3(1), 23–48.
Norris, S. (2006). Multiparty interaction: A multimodal perspective on relevance. Discourse Studies, 8(3), 401–421.
O’ Toole, M. (2004). Opera Ludentes: The Sydney opera house at work and play. In K. O’Halloran (Ed.), Multimodal discourse analysis: Systemic functional perspectives (pp. 11–27). London: Continuum.
O’Halloran, K. (1998). Classroom discourse in mathematics: A multisemiotic analysis. Linguistics and Education, 10(3), 359–388.
O’Halloran, K. (2004). Multimodal discourse analysis: Systemic functional perspectives. London: Continuum.
O’Halloran, K. (2008). Systemic functional-multimodal discourse analysis (SF-MDA): Constructing ideational meaning using language and visual imagery. Visual Communication, 7(4), 443–475.
O’Halloran, K., Marissa, K. L. E., & Tan, S. (2014). Multimodal analytics: Software and visualization techniques for analyzing and interpreting multimodal data. In The Routledge handbook of multimodal analysis (pp. 386–396). London: Routledge.
O’Halloran, K. L., Tan, S., Wignell, P., Bateman, J. A., Pham, D.-S., Grossman, M., & Moere, A. V. (2016). Interpreting text and image relations in violent extremist discourse: A mixed methods approach for big data analytics. Terrorism and Political Violence, 1–21.
Pietikäinen, S., Lane, P., Salo, H., & Laihiala-Kankainen, S. (2011). Frozen actions in the arctic linguistic landscape: A nexus analysis of language processes in visual space. International Journal of Multilingualism, 8(4), 277–298.
Pirini, J. (2014a). Introduction to multimodal (inter)action analysis. In S. Norris & C. Maier (Eds.), Interactions, texts and images: A reader in multimodality (Vol. 1, 1st ed., pp. 77–92). New York: Mouton de Gruyter.
Pirini, J. (2014b). Producing shared attention/awareness in high school tutoring. Multimodal Communication, 3(2), 163–179.
Pirini, J. (2016). Intersubjectivity and materiality: A multimodal perspective. Multimodal Communication, 5(1), 1–14.
Pirini, J. (2017). Agency and Co-production: A multimodal perspective. Multimodal Communication, 6(2), 1–20.
Pirini, J., Norris, S., Geenen, J., & Matelau, T. (2014). Studying social actors: Some thoughts on ethics. In S. Norris & C. Maier (Eds.), Interactions, texts and images: A reader in multimodality (Vol. 1, 1st ed., pp. 233–243). New York: Mouton de Gruyter.
Sacks, H. (1967). The search for help: No one to turn to. In E. Schneidman (Ed.), Essays in self-destruction. New York: Science House.
Sacks, H. (1972). An initial investigation of the usability of conversational data for doing sociology. In Studies in social interaction. New York: Free Press.
Sacks, H., & Schegloff, E. A. (2002). Home position. Gesture, 2(2), 133–146.
Sacks, H., Schegloff, E. A., & Jefferson, G. (1974). A Simplest systematics for the organization of turn-taking for conversation. Language, 50(4), 696–735.
Schegloff, E. A. (1967). The first five seconds: The order of conversational openings. Berkeley: Department of Sociology, University of California.
Schiffrin, D. (1987). Discourse markers. Cambridge: Cambridge University Press.
Scollon, R. (1998). Mediated discourse as social interaction: A study of news discourse. London and New York: Longman.
Scollon, R. (2001). Mediated discourse: The nexus of practice. London and New York: Routledge.
Scollon, R., & Scollon, S. B. K. (2003). Discourses in place: Language in the material world. London: Routledge.
Scollon, R., & Scollon, S. B. K. (2004). Nexus analysis: Discourse and the emerging internet. New York, NY: Psychology Press.
Sidnell, J. (2012). Basic conversation analytic methods. In J. Sidnell & T. Stivers (Eds.), The handbook of conversation analysis (pp. 77–100). Oxford: Wiley-Blackwell.
Tannen, D. (1984). Coherence in spoken and written discourse. Norwood: ABLEX Pub. Corp.
Tapio, E. (2014). The Marginalisation of finely tuned semiotic practices and misunderstandings in relation to (signed) languages and deafness. Multimodal Communication, 3(2), 245.
Vygotsky, L. S. (1978). Mind in society. Cambridge: Harvard University Press.
Wertsch, J. V. (1998). Mind as action. New York: Oxford University Press.
Zemel, A., & Koschmann, T. (2014). “Put your fingers right in here”: Learnability and instructed experience. Discourse Studies, 16(2), 163–183.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Copyright information
© 2018 The Author(s)
About this chapter
Cite this chapter
Pirini, J., Matelau-Doherty, T., Norris, S. (2018). Multimodal Analysis. In: Phakiti, A., De Costa, P., Plonsky, L., Starfield, S. (eds) The Palgrave Handbook of Applied Linguistics Research Methodology. Palgrave Macmillan, London. https://doi.org/10.1057/978-1-137-59900-1_28
Download citation
DOI: https://doi.org/10.1057/978-1-137-59900-1_28
Publisher Name: Palgrave Macmillan, London
Print ISBN: 978-1-137-59899-8
Online ISBN: 978-1-137-59900-1
eBook Packages: Social SciencesSocial Sciences (R0)