Multimodality in Memes: A Cyberpragmatic Approach

  • Francisco YusEmail author


In this chapter, a corpus of 100 instances of a specific type of meme, image macro (the one typically made up of one top and one bottom stretch of text and a picture in-between), is analyzed in its multimodal quality, specifically in search of different categories that these memes might fit into depending on the relationship existing between picture and text and its impact on the quality of the eventual interpretation. An underlying assumption in the chapter, broadly within a cyberpragmatic framework (F. Yus, Cyberpragmatics: Internet-mediated communication in context, John Benjamins, 2011), is that different text-picture combinations will have an impact on eventual relevance by yielding different balances of cognitive effects and mental effort, the latter sometimes compensated for by an offset of additional cognitive effects in the shape of implications.



This chapter is part of the research project PROMETEO 2016/052, titled “Humor gender: Observatory of identity of women and men through humor”, funded by the Generalitat Valenciana, Conselleria d’Educació, Investigació, Cultura i Sport.


  1. Barthes, R. (1977). Image. Music. Text. London: Fontana.Google Scholar
  2. Carston, R. (2002). Thoughts and utterances. Oxford: Blackwell.CrossRefGoogle Scholar
  3. Chan, E. (2011). Integrating visual and verbal meaning in multimodal text comprehension: Towards a model of intermodal relations. In S. Dreyfus, S. Hood, & S. Stenglin (Eds.), Semiotic margins: Meaning in multimodalities (pp. 144–167). London: Continuum.Google Scholar
  4. Cohn, N. (2013). Beyond word balloons and thought bubbles: The integration of text and image. Semiotica, (197), 35–63.
  5. Conradie, M., Brokensha, S., & Pretorius, M. (2012). No small irony: A discourse analysis of Zapiro’s 2010 World Cup cartoons. Language Matters, 43(1), 39–59. Scholar
  6. Dawkins, R. (1976). The selfish gene. Oxford: Oxford University Press.Google Scholar
  7. Dynel, M. (2016). I has seen image macros!’ Advice animal memes as visual-verbal jokes. International Journal of Communication, 10, 660–688.Google Scholar
  8. El Refaie, E., & Hörschelmann, K. (2010). Young people’s readings of a political cartoon and the concept of multimodal literacy. Discourse: Studies in the Cultural Politics of Education, 31(2), 195–207. Scholar
  9. Forceville, C. (2014). Relevance theory as a model for analyzing visual and multimodal communication. In D. Machin (Ed.), Visual communication (pp. 51–70). Berlin: Mouton de Gruyter.Google Scholar
  10. Forceville, C., & Clark, B. (2014). Can pictures have explicatures? Linguagem em (Dis)curso (special issue on relevance theory), 14(3), 451–472. Scholar
  11. Gill, T. (2002). Visual and verbal playmates: An exploration of visual and verbal modalities in children’s picture books. Unpublished BA (Hons) thesis, University of Sydney.Google Scholar
  12. Jewitt, C. (2016). Multimodal analysis. In A. Georgakopoulou & T. Spilioti (Eds.), Handbook of language and digital communication (pp. 69–84). Abingdon: Routledge.Google Scholar
  13. Kardaş, T. (2012). No laughing matter: Visualizing Turkey’s Ergenekon in political cartoons. Middle East Critique, 21(2), 203–223. Scholar
  14. Kress, G., & van Leeuwen, T. (2006). Reading images: The grammar of visual design (2nd ed.). London: Routledge.CrossRefGoogle Scholar
  15. Laineste, L., & Voolaid, P. (2016). Laughing across borders: Intertextuality of internet memes. European Journal of Humour Research, 4(4), 26–49. Scholar
  16. Lim Fei, V. (2004). Developing an integrative multi-semiotic model. In K. O’Halloran (Ed.), Multimodal discourse analysis: Systemic functional perspectives (pp. 220–246). London: Continuum.Google Scholar
  17. Martinec, R., & Salway, A. (2005). A system for image-text relations in new (and old) media. Visual Communication, 4(3), 337–371. Scholar
  18. McCloud, S. (1994). Understanding comics: The invisible art. New York: Harper Collins.Google Scholar
  19. Nissenbaum, A., & Shifman, L. (2017). Internet memes as contested cultural capital: The case of 4chan’s /b/ board. New Media & Society, 19(4), 483–501. Scholar
  20. Salway, A., & Martinec, R. (2002). Some ideas for modelling image-text combinations. Guildford: University of Surrey, Department of Computing.Google Scholar
  21. Sarapik, V. (2009). Picture, text, and imagetext: Textual polylogy. Semiotica, 174(1/4), 277–308. Scholar
  22. Segev, E., Nissenbaum, A., Stolero, N., & Shifman, L. (2015). Families and networks of Internet memes: The relationship between cohesiveness, uniqueness, and quiddity concreteness. Journal of Computer-Mediated Communication, 20, 417–433. Scholar
  23. Shifman, L. (2014). Memes in digital culture. Cambridge, MA: The MIT Press.Google Scholar
  24. Sperber, D., & Wilson, D. (1995). Relevance: Communication and cognition (2nd ed.). Oxford: Blackwell.Google Scholar
  25. Stöckl, H. (2004). In between modes: Language and image in printed media. In E. Ventola, C. Charles, & M. Kaltenbacher (Eds.), Perspectives on multimodality (pp. 9–30). Amsterdam: John Benjamins.CrossRefGoogle Scholar
  26. Taecharungroj, V., & Nueangjamnong, P. (2014). The effect of humour on virality: The study of Internet memes on social media. Paper presented at 7th International Forum on Public Relations and Advertising Media Impacts on Culture and Social Communication, Bangkok, August.Google Scholar
  27. Trifonas, P. P. (2015). Text and images. In P. P. Trifonas (Ed.), International handbook of semiotics (pp. 1139–1152). Berlin: Springer. Scholar
  28. Tsakona, V. (2009). Language and image interaction in cartoons: Towards a multimodal theory of humor. Journal of Pragmatics, 41, 1171–1188. Scholar
  29. van Leeuwen, T. (2006). Word and image. In K. Brown (Ed.), Encyclopedia of language and linguistics (2nd ed.). Amsterdam: Elsevier.Google Scholar
  30. van Leeuwen, T. (2011). Multimodality. In J. Simpson (Ed.), The Routledge handbook of applied linguistics (pp. 668–682). Abingdon: Routledge.Google Scholar
  31. van Leeuwen, T. (2015). Multimodality. In D. Tannen, H. E. Hamilton, & D. Schiffrin (Eds.), The handbook of discourse analysis (2nd ed., pp. 447–465). Oxford: Wiley Blackwell.Google Scholar
  32. Wharton, T. (2009). Pragmatics and non-verbal communication. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  33. Wilson, D., & Carston, R. (2006). Metaphor, relevance and the ‘emergent property’ issue. Mind & Language, 21(3), 404–433. Scholar
  34. Wu, S. (2014). A multimodal analysis of image-text relations in picture books. Theory and Practice in Language Studies, 4(7), 1415–1420.Google Scholar
  35. Yus, F. (1997). La teoría de la relevancia y la estrategia humorística de la incongruencia-resolución. Pragmalingüística, 3–4, 497–508.Google Scholar
  36. Yus, F. (2010). Relevance theory. In B. Heine & H. Narrog (Eds.), The Oxford handbook of linguistic analysis (pp. 679–701). Oxford: Oxford University Press.Google Scholar
  37. Yus, F. (2011). Cyberpragmatics: Internet-mediated communication in context. Amsterdam: John Benjamins.CrossRefGoogle Scholar
  38. Yus, F. (2013a). An inference-centered analysis of jokes: The Intersecting Circles Model of humorous communication. In L. Ruiz Gurillo & B. Alvarado (Eds.), Irony and humor: Highlights and genres (pp. 59–82). Amsterdam: John Benjamins.CrossRefGoogle Scholar
  39. Yus, F. (2013b). Analyzing jokes with the Intersecting Circles Model of humorous communication. Lodz Papers in Pragmatics, 9(1), 3–24. Scholar
  40. Yus, F. (2016). Humour and relevance. Amsterdam: John Benjamins.CrossRefGoogle Scholar
  41. Yus, F. (2017). Incongruity-resolution cases in jokes. Lingua, 197, 103–122. Scholar

Copyright information

© The Author(s) 2019

Authors and Affiliations

  1. 1.Department of English StudiesUniversity of AlicanteAlicanteSpain

Personalised recommendations