Multimedia Tools and Applications

, Volume 78, Issue 10, pp 14029–14043 | Cite as

Multimedia analysis and design: a conceptual framework

  • Vito RobertoEmail author
  • Elio Toppano


The paper proposes a conceptual model inspired to semiotic theories, to be applied to the analysis and design of multimedia. We introduce a meta-model with four levels of semantic aggregation. There results a framework of concepts, relations and processes accounting for the multiple meanings that arise from a multimedia text. We explore the effectiveness of the model by considering four commercial clips by the brand Lancôme. Our analysis confirms that the framework is well suited to the analysis, indexing, design of narrative multimedia.


Multimedia Analysis Semiotic Design Commercials Advertisements 



  1. 1.
    Baley B, Konstan J, Carlis J (2001) Supporting multimedia designers: towards more effective design tools. In: Proc. Modeling Multimedia Information and Systems (MMM 2001), 267–286Google Scholar
  2. 2.
    Bardzell J (2009) Interaction criticism and aesthetics. In: Proc. CHI2009, Boston. 2357–2366Google Scholar
  3. 3.
    Bateman JA (2007) Towards a grande paradigmatique of film: Christian Metz reloaded. Semiotica 167(1/4):13–64Google Scholar
  4. 4.
    Bianchi C (2011) Semiotic approaches to advertising texts and strategies: narrative, passion, marketing. Semiotica 183(1/4):243–271Google Scholar
  5. 5.
    Bolchini D, Chatterji R, Speroni M (2009) Developing heuristics for the semiotics inspection of websites, In: Proc. 27th ACM Int. Conf. on Design of Communication. 67–72Google Scholar
  6. 6.
    Bouvier G, Machin D (2013) How advertisers use sound and music to communicate ideas, attitudes and identities: a multimodal critical discourse approach. In: Pennock Spek B, Del Saz Rubio MM (eds) The multimodal analysis of television advertising. University of Valencia Press, ValenciaGoogle Scholar
  7. 7.
    Chandler D (2002) Semiotics: The Basics. Routledge, London and New YorkGoogle Scholar
  8. 8.
    Cockton G (2006) Designing worth is worth designing. Proc. NordiCHI Conference, ACM Press. 165–174Google Scholar
  9. 9.
    Colombo C, Del Bimbo A, Pala P (2001) Retrieval of commercials by semantic content: The semiotic perspective. Multimedia Tools and Applications, Vol. 13, Kluwer Academic Publishers 93–118Google Scholar
  10. 10.
    Davis M (1994) Knowledge representation for video. In: Proc. 12th National Conf. on Artificial Intelligence (AAAI-94), Seattle, Washington, AAAI Press. 120–127Google Scholar
  11. 11.
    De Saussure F (1922) Cours de Linguistique Generale. (In French). Payot, ParisGoogle Scholar
  12. 12.
    Desmet PMA, Pohlmeyer AE (2013) Positive design: an introduction to design for subjective well-being. Int J Design 7(3):5–19Google Scholar
  13. 13.
    Eugeni R (2011) Media experiences and practices of analysis. For a critical pragmatics of media. In: Int. Workshop Practicing Theory, Univ. AmsterdamGoogle Scholar
  14. 14.
    Floch JM (2001) Semiotics, Marketing and Communication: Beneath the Signs, the Strategies. Palgrave, HampshireGoogle Scholar
  15. 15.
    Fogg B (2003) Persuasive technology: using computers to change what we think and do. Morgan Kaufmann, San FranciscoGoogle Scholar
  16. 16.
    Friedman B, Kahn PH, Borning A (2006) Value sensitive design and information systems. In: Zhang P, Galletta D (eds) Human-computer interaction and management information systems: foundations. Sharpe Inc, New York, pp 348–372Google Scholar
  17. 17.
    Gabrielson A, Lindstrom E (2001) The influence of musical structure on emotional expression. In: Juslin P, Sloboda J (eds) Music and emotion theory and research. Oxford University Press, New York, pp 223–248Google Scholar
  18. 18.
    Greimas AJ (1984) Semiotique figurative et semiotique plastique. In: Actes semiotiques. Documents, N.60 (in French)Google Scholar
  19. 19.
    Greimas AJ, Courtes J (1982) Semiotics and language. An analytical dictionary. In: Indiana University press. USA, BloomingtonGoogle Scholar
  20. 20.
    Harrison C (2003) Visual social semiotics: understanding how still images make meaning. Tech Commun 50(1):46–60Google Scholar
  21. 21.
    Hassenzahl M (2010) Experience design: technology for all the right reasons. Morgan & Claypool, San RafaelGoogle Scholar
  22. 22.
    Hebert L (2008) Tools for Text and Image Analysis: An Introduction to Applied Semiotics. URL: Accessed 30 Dec 2018
  23. 23.
    Islam MN, Bouwman H (2016) Towards user-intuitive web interface sign design and evaluation, International Journal of Human-Computer Studies, Vol.86, C, Academic Press, 121–137Google Scholar
  24. 24.
    Kress G, van Leeuwen T (2006) Reading Images: The Grammar of Visual Design. Routledge, LondonGoogle Scholar
  25. 25.
    Kujala S, Vaananen-Vainio-Mattila K (2009) Value of information systems and products: understanding the users’ perspective and values. J Inform Technol Theory Appl, JITTA 9(4):23–39Google Scholar
  26. 26.
    Lauer C (2009) Contending with terms: ‘multimodal’ and ‘multimedia’ in the academic and public spheres. Comput Compos 26:225–239Google Scholar
  27. 27.
    Lombardo V, Damiano R (2010) Narrative annotation and editing of video. In: Proc. ICIDS’10, Third Joint Conference on Interactive Digital Storytelling. 62–73Google Scholar
  28. 28.
    Mancini C, Buckingham Shum SJ (2001) Cognitive coherence relations and hypertext: from cinematic patterns to scholarly discourse. In: Proc. HypertextGoogle Scholar
  29. 29.
    Mann WC, Thompson S (1988) Rhetorical structure theory: toward a functional theory of text organization. Text 8(3):243–281Google Scholar
  30. 30.
    McQuarrie EF, Mick DG (1999) Visual rhetoric in advertising: text-interpretive, experimental, and reader-response analyses. J Consum Res 26(1):37–54Google Scholar
  31. 31.
    Montero S, Diaz P, Aedo I (2003) A Framework for the analysis and comparison of hypermedia design methods. In: Proc. Appl. Informatics 2003, Innsbruck, A. 1053–1058Google Scholar
  32. 32.
    Nadin M (1988) Interface design: a semiotic paradigm. Semiotica 69:269–302Google Scholar
  33. 33.
    O’Halloran KL (2011) Multimodal Discourse Analysis. In: K. Hyland, B. Paltridge (Eds.), Companion to Discourse. London and New York: Continuum, 120–137Google Scholar
  34. 34.
    Purchase HC, Naumann D (2001) A semiotic model of multimedia. Theory and eval- uation. In: Design and Management of Multimedia Information Systems: Opportunities and Challenges, Idea Group Publishing, 1–24Google Scholar
  35. 35.
    Rogers Y (2004) New theoretical approaches for human-computer interaction. Annu Rev Inf Sci Technol 13(1):87–143Google Scholar
  36. 36.
    Scharfe H (2007) Grand principles of narratology. In: I.Bas, C.Freeman (Eds.), Chal- lenging the Boundaries, Rodopi, Amsterdam, NL, pp. 95–110Google Scholar
  37. 37.
    Sieckenius de Souza C (2001) Semiotic Approaches to User Interface Design, Editorial, Knowledge-Based Systems, Vol.14, 8, Elsevier. 415–418Google Scholar
  38. 38.
    Sieckenius de Souza C (2004) The semiotic engineering of human-computer inter- action. The MIT Press, CambridgeGoogle Scholar
  39. 39.
    SieckeniusdeSouza C (2013) Viewpoint:Semioticperspectivesoninteractivelanguages for life on the screen. J Vis Lang Comput 24(3):218–221Google Scholar
  40. 40.
    Snoek CGM, Worring M (2005) Multimodal video indexing: a review of the state- of-the-art. Multimed Tools Appl 25:5–35Google Scholar
  41. 41.
    Toppano E, Roberto V (2009) Semiotic design and analysis of hypermedia. In: Proc. 20th ACM Conference on Hypertext and Hypermedia, HT2009, Turin, I. 367–368Google Scholar
  42. 42.
    Toppano E, Toppano A (2014) The role of sound in brand communication. In: Proc. of the Int. Conference on e-Commerce, Lisbon, P. Computer Science and Information Systems Series, IADIS Press. 167–174Google Scholar
  43. 43.
    Tresor by Lancôme, Ines Sastre (2007) URL: Accessed 30 Dec 2018
  44. 44.
    Tresor by Lancôme, Isabella Rossellini (1990) URL: Accessed 30 Dec 2018
  45. 45.
    Tresor by Lancôme, Kate Winslet (2009) URL: Accessed 30 Dec 2018
  46. 46.
    Tresor by Lancôme, Penelope Cruz (2010) URL: . Accessed 30 Dec 2018
  47. 47.
    Tromp N, Hekkert P, Verbeek PP (2011) Design for socially responsible behavior: a classification of influence based on intended user experience. Des Issues 27(3):3–19Google Scholar
  48. 48.
    Tseng C (2013) Analysing characters' interactions in filmic texts: a functional semiotic approach. Soc Semiot 23(5):587–605Google Scholar
  49. 49.
    Verbeek PP (2006) Materializing morality: design ethics and technological mediation. Science. Technol Human Val 31(3):361–380Google Scholar
  50. 50.
    Wingstedt J, Brandstrom S, Berg J (2010) Narrative music, visuals and meaning in film. Vis Commun 9(2):193–210Google Scholar
  51. 51.
    Wouken A, Carr L, Wills G, Hall W (2002) Rethinking web design models: require- ments for addressing the content. Tech. Rep. N.ECSTR-IAM03-002, Univ. SouthamptonGoogle Scholar
  52. 52.
    Wright P, Wallace J, McCarthy J (2008) Aesthetics and experience-centered design. ACM Trans. on Computer-Human Interaction, Vol.15, 4, Art. 18Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Dipartim. Scienze Matematiche, Informatiche e Fisiche (DMIF)Università di UdineUdineItaly
  2. 2.Dipartim. Scienze Matematiche, Informatiche e Fisiche (DMIF)Università di UdineUdineItaly

Personalised recommendations