Multimodal Analysis

  • Jesse PiriniEmail author
  • Tui Matelau-Doherty
  • Sigrid Norris


This chapter introduces the five most prominent approaches to multimodal data analysis: Multimodal (Inter)action Analysis, Mediated Discourse Analysis, Systemic Functional Multimodal Discourse Analysis, Social Semiotics, and Multimodal Conversation Analysis. The chapter begins by discussing the origin of each approach, methods of data collection and analysis, and embedded theoretical foundations. The theoretical differences between the approaches are then examined with a focus on the way each approach treats the individual, the artefact, and the notion of mode during analysis. With the variety of available approaches to multimodal data analysis, it is important that researchers link data collection and analysis to coherent theoretical underpinnings.


Multimodal Discourse Analysis Multimodal (Inter)action Analysis Social Semiotics SF-MDA Mediated Discourse Analysis 


  1. Al Zidjaly, N. (2009). Agency as an interactive achievement. Language In Society, 38(2), 177–200.CrossRefGoogle Scholar
  2. Baldry, A., & Thibault, P. J. (2006). Multimodal transcription and text analysis: A multimedia toolkit and associated on-line coursebook. London and Oakville: Equinox.Google Scholar
  3. Bernad-Mechó, E. (2017). Metadiscourse and topic introductions in an academic lecture: A multimodal insight. Multimodal Communication, 6(1), 659.CrossRefGoogle Scholar
  4. Deppermann, A. (2013). Multimodal interaction from a conversation analytic perspective. Journal of Pragmatics, 46(1), 1–7.CrossRefGoogle Scholar
  5. de Saint-Georges, I. (2014). Mediated discourse analysis, “embodied learning” and emerging social and professional identities. In C. Maier & S. Norris (Eds.), Interactions, images and texts (Vol. 11, 1st ed., pp. 347–356). Boston, MA: De Gruyter.Google Scholar
  6. Eggins, S. (2004). Introduction to systemic functional linguistics (2nd ed.). London: A&C Black.Google Scholar
  7. Forceville, C., & Urios-Aparisi, E. (2009). Multimodal metaphor. Berlin: Walter de Gruyter.CrossRefGoogle Scholar
  8. Geenen, J. (2017). Show and (sometimes) tell: Identity construction and the affordances of video-conferencing. Multimodal Communication, 6(1), 1–18.CrossRefGoogle Scholar
  9. Goffman, E. (1959). The presentation of self in everyday life. Gloucester, MA: Peter Smith Pub Incorporated.Google Scholar
  10. Goffman, E. (1981). Forms of talk. Philadelphia, PA: University of Pennsylvania Press.Google Scholar
  11. Goodwin, C. (1979). The interactive construction of a sentence in natural conversation. In G. Psathas (Ed.), Everyday language: Studies in Ethnomethodology (pp. 97–121). New York: Irvington Publishers.Google Scholar
  12. Goodwin, C. (2000). Action and embodiment within situated human interaction. Journal of Pragmatics, 32(10), 1489–1522.CrossRefGoogle Scholar
  13. Gumperz, J. J. (1982). Discourse strategies. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  14. Haddington, P., Mondada, L., & Nevile, M. (2013). Being mobile: Interaction on the move. In P. Haddington, L. Mondada, & M. Nevile (Eds.), Interaction and mobility, language and the body in motion. Berlin, Boston: De Gruyter.CrossRefGoogle Scholar
  15. Halliday, M. A. K. (1978). Language as a social semiotic: The social interpretation of language and meaning. Baltimore, MD: University Park Press.Google Scholar
  16. Hamilton, H. E. (1994). Conversations with an Alzheimer’s Patient: An interactional sociolinguistic study. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  17. Hart, C. (2016). The visual basis of linguistic meaning and its implications for critical discourse studies: Integrating cognitive linguistic and multimodal methods. Discourse & Society, 27, 335–350.CrossRefGoogle Scholar
  18. Jewitt, C. (2006). Technology, literacy and learning: A multimodal approach. New York: Psychology Press.Google Scholar
  19. Jewitt, C. (2013). Multimodal methods for researching digital technologies. In The SAGE handbook of digital technology research. Thousand Oaks, CA: Sage.Google Scholar
  20. Kress, G. (1999). Multimodality. In B. Cope & M. Kalantzis (Eds.), Multiliteracies: Literacy learning and the design of social futures. New York, NY: Routledge.Google Scholar
  21. Kress, G., & van Leeuwen, T. (1996). Reading images: The grammar of visual design. New York, NY: Routledge.Google Scholar
  22. Kress, G., & van Leeuwen, T. (2001). Multimodal discourse: The modes and media of contemporary communication. London: Edward Arnold.Google Scholar
  23. Kress, G., & van Leeuwen, T. (2002). Colour as a semiotic mode: Notes for a grammar of colour. Visual Communication, 1(3), 343–368.CrossRefGoogle Scholar
  24. Kress, G., & van Leeuwen, T. (2006). The semiotic landscape. In P. Griffiths, A. J. Merrison, & A. Bloomer (Eds.), Language in use—A reader (pp. 344–350). London: Routledge.Google Scholar
  25. van Leeuwen, T. (2005). Introducing social semiotics. New York, NY: Psychology Press.Google Scholar
  26. Machin, D., & Mayr, A. (2012). How to do critical discourse analysis: A multimodal introduction. London: SAGE.Google Scholar
  27. Matelau, T. (2014). Vertical identity production and Maori identity. In S. Norris & C. Maier (Eds.), Interactions, texts and images: A reader in multimodality (Vol. 1, 1st ed., pp. 255–266). New York: Mouton de Gruyter.Google Scholar
  28. Mondada, L. (2006). Participants’ online analysis and multimodal practices: Projecting the end of the turn and the closing of the sequence. Discourse Studies, 8(1), 117–129.CrossRefGoogle Scholar
  29. Mondada, L. (2009). Emergent focused interactions in public places: A systematic analysis of the multimodal achievement of a common interactional space. Journal of Pragmatics, 41(10), 1977–1997.CrossRefGoogle Scholar
  30. Mondada, L. (2013). The conversation analytic approach to data collection. In J. Sidnell & T. Stivers (Eds.), The handbook of conversation analysis (pp. 210–228). Oxford: Wiley-Blackwell.Google Scholar
  31. Norris, S. (2004). Analyzing multimodal interaction: A methodological framework. London: Routledge.CrossRefGoogle Scholar
  32. Norris, S. (2011). Identity in interaction: Introducing multimodal interaction analysis. London: Mouton de Gruyter.Google Scholar
  33. Norris, S. (2013). What is a mode? Smell, olfactory perception, and the notion of mode in multimodal mediated theory. Multimodal Communication, 2(2), 155–169.Google Scholar
  34. Norris, S. (2014). Learning tacit classroom participation. WCTLA. Procedia Social and Behavioral Sciences. Amsterdam: Elsevier.CrossRefGoogle Scholar
  35. Norris, S. (forthcoming). Systematically working with multimodal data: Research methods in multimodal discourse analysis. Hoboken, NJ: John Wiley and Sons.Google Scholar
  36. Norris, S., & Pirini, J. P. (2017). Communicating knowledge, getting attention, and negotiating disagreement via videoconferencing technology: A multimodal analysis. Journal of Organizational Knowledge Communication, 3(1), 23–48.CrossRefGoogle Scholar
  37. Norris, S. (2006). Multiparty interaction: A multimodal perspective on relevance. Discourse Studies, 8(3), 401–421.CrossRefGoogle Scholar
  38. O’ Toole, M. (2004). Opera Ludentes: The Sydney opera house at work and play. In K. O’Halloran (Ed.), Multimodal discourse analysis: Systemic functional perspectives (pp. 11–27). London: Continuum.Google Scholar
  39. O’Halloran, K. (1998). Classroom discourse in mathematics: A multisemiotic analysis. Linguistics and Education, 10(3), 359–388.CrossRefGoogle Scholar
  40. O’Halloran, K. (2004). Multimodal discourse analysis: Systemic functional perspectives. London: Continuum.Google Scholar
  41. O’Halloran, K. (2008). Systemic functional-multimodal discourse analysis (SF-MDA): Constructing ideational meaning using language and visual imagery. Visual Communication, 7(4), 443–475.CrossRefGoogle Scholar
  42. O’Halloran, K., Marissa, K. L. E., & Tan, S. (2014). Multimodal analytics: Software and visualization techniques for analyzing and interpreting multimodal data. In The Routledge handbook of multimodal analysis (pp. 386–396). London: Routledge.Google Scholar
  43. O’Halloran, K. L., Tan, S., Wignell, P., Bateman, J. A., Pham, D.-S., Grossman, M., & Moere, A. V. (2016). Interpreting text and image relations in violent extremist discourse: A mixed methods approach for big data analytics. Terrorism and Political Violence, 1–21.Google Scholar
  44. Pietikäinen, S., Lane, P., Salo, H., & Laihiala-Kankainen, S. (2011). Frozen actions in the arctic linguistic landscape: A nexus analysis of language processes in visual space. International Journal of Multilingualism, 8(4), 277–298.CrossRefGoogle Scholar
  45. Pirini, J. (2014a). Introduction to multimodal (inter)action analysis. In S. Norris & C. Maier (Eds.), Interactions, texts and images: A reader in multimodality (Vol. 1, 1st ed., pp. 77–92). New York: Mouton de Gruyter.Google Scholar
  46. Pirini, J. (2014b). Producing shared attention/awareness in high school tutoring. Multimodal Communication, 3(2), 163–179.CrossRefGoogle Scholar
  47. Pirini, J. (2016). Intersubjectivity and materiality: A multimodal perspective. Multimodal Communication, 5(1), 1–14.CrossRefGoogle Scholar
  48. Pirini, J. (2017). Agency and Co-production: A multimodal perspective. Multimodal Communication, 6(2), 1–20.CrossRefGoogle Scholar
  49. Pirini, J., Norris, S., Geenen, J., & Matelau, T. (2014). Studying social actors: Some thoughts on ethics. In S. Norris & C. Maier (Eds.), Interactions, texts and images: A reader in multimodality (Vol. 1, 1st ed., pp. 233–243). New York: Mouton de Gruyter.Google Scholar
  50. Sacks, H. (1967). The search for help: No one to turn to. In E. Schneidman (Ed.), Essays in self-destruction. New York: Science House.Google Scholar
  51. Sacks, H. (1972). An initial investigation of the usability of conversational data for doing sociology. In Studies in social interaction. New York: Free Press.Google Scholar
  52. Sacks, H., & Schegloff, E. A. (2002). Home position. Gesture, 2(2), 133–146.CrossRefGoogle Scholar
  53. Sacks, H., Schegloff, E. A., & Jefferson, G. (1974). A Simplest systematics for the organization of turn-taking for conversation. Language, 50(4), 696–735.CrossRefGoogle Scholar
  54. Schegloff, E. A. (1967). The first five seconds: The order of conversational openings. Berkeley: Department of Sociology, University of California.Google Scholar
  55. Schiffrin, D. (1987). Discourse markers. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  56. Scollon, R. (1998). Mediated discourse as social interaction: A study of news discourse. London and New York: Longman.Google Scholar
  57. Scollon, R. (2001). Mediated discourse: The nexus of practice. London and New York: Routledge.Google Scholar
  58. Scollon, R., & Scollon, S. B. K. (2003). Discourses in place: Language in the material world. London: Routledge.CrossRefGoogle Scholar
  59. Scollon, R., & Scollon, S. B. K. (2004). Nexus analysis: Discourse and the emerging internet. New York, NY: Psychology Press.CrossRefGoogle Scholar
  60. Sidnell, J. (2012). Basic conversation analytic methods. In J. Sidnell & T. Stivers (Eds.), The handbook of conversation analysis (pp. 77–100). Oxford: Wiley-Blackwell.CrossRefGoogle Scholar
  61. Tannen, D. (1984). Coherence in spoken and written discourse. Norwood: ABLEX Pub. Corp.Google Scholar
  62. Tapio, E. (2014). The Marginalisation of finely tuned semiotic practices and misunderstandings in relation to (signed) languages and deafness. Multimodal Communication, 3(2), 245.CrossRefGoogle Scholar
  63. Vygotsky, L. S. (1978). Mind in society. Cambridge: Harvard University Press.Google Scholar
  64. Wertsch, J. V. (1998). Mind as action. New York: Oxford University Press.Google Scholar
  65. Zemel, A., & Koschmann, T. (2014). “Put your fingers right in here”: Learnability and instructed experience. Discourse Studies, 16(2), 163–183.CrossRefGoogle Scholar

Copyright information

© The Author(s) 2018

Authors and Affiliations

  • Jesse Pirini
    • 1
    Email author
  • Tui Matelau-Doherty
    • 2
  • Sigrid Norris
    • 2
  1. 1.School of ManagementVictoria University of WellingtonWellingtonNew Zealand
  2. 2.AUT Multimodal Research CentreAuckland University of TechnologyAucklandNew Zealand

Personalised recommendations