Multimedia Tools and Applications

, Volume 78, Issue 10, pp 13613–13648 | Cite as

FANTASIA: a framework for advanced natural tools and applications in social, interactive approaches

  • Antonio OrigliaEmail author
  • Francesco Cutugno
  • Antonio Rodà
  • Piero Cosi
  • Claudio Zmarich


With the recent availability of industry-grade, high-performing engines for video games production, researchers in different fields have been exploiting the advanced technologies offered by these artefacts to improve the quality of the interactive experiences they design. While these engines provide excellent and easy-to-use tools to design interfaces and complex rule-based systems to control the experience, there are some aspects of Human-Computer Interaction (HCI) research they do not support in the same way because of their original mission and related design patterns pointing at a different primary target audience. In particular, the more research in HCI evolves towards natural, socially engaging approaches, the more there is the need to rapidly design and deploy software architectures to support these new paradigms. Topics such as knowledge representation, probabilistic reasoning and voice synthesis demand space as possible instruments within this new ideal design environment. In this work, we propose a framework, named FANTASIA, designed to integrate a set of chosen modules (a graph database, a dialogue manager, a game engine and a voice synthesis engine) and support rapid design and implementation of interactive applications for HCI studies. We will present a number of different case studies to exemplify how the proposed tools can be deployed to develop very different kinds of interactive applications and we will discuss ongoing and future work to further extend the framework we propose.


Human computer interaction Game engines Application development tools 



Antonio Origlia’s work is funded by the Italian PRIN project Cultural Heritage Resources Orienting Multimodal Experience (CHROME) #B52F15000450001.


  1. 1.
    André C, Ghio A, Cavé C, Teston B (2007) PERCEVAL: a computer-driven system for experimentation on auditory and visual perception. arXiv:0705.4415
  2. 2.
    Byun TM, Tiede M (2017) Perception-production relations in later development of american english rhotics. PloS One 12(2):e0172,022CrossRefGoogle Scholar
  3. 3.
    Caselli MC, Casadio P (1995) Il primo vocabolario del bambino. Franco Angeli, MilanoGoogle Scholar
  4. 4.
    Cera V, Origlia A, Cutugno F, Campi M (2018) Semantically annotated 3d material supporting the design of natural user interfaces for architectural heritage. In: Proceedings of the AVI-CH workshop (to appear)Google Scholar
  5. 5.
    Cosi P, Paci G, Sommavilla G, Tesser F (2016) Mivoq-ptts-a revolutionary new way of thinking tts. In: Proceedings of interspeech, pp 3888–3889Google Scholar
  6. 6.
    Di Maro M, Valentino M, Riccio A, Origlia A (2017) Graph databases for designing high-performance speech recognition grammars. In: IWCS 2017—12th international conference on computational semantics—short papersGoogle Scholar
  7. 7.
    Dietze F, Karoff J, Valdez AC, Ziefle M, Greven C, Schroeder U (2016) An open-source object-graph-mapping framework for neo4j and scala: Renesca. In: International conference on availability, reliability, and security. Springer, pp 204–218Google Scholar
  8. 8.
    Drakopoulos G, Kanavos A, Makris C, Megalooikonomou V (2015) On converting community detection algorithms for fuzzy graphs in neo4j. In: Proceedings of the 5th International Workshop on Combinations of Intelligent Methods and Applications, CIMAGoogle Scholar
  9. 9.
    González J, Escobar J, Sánchez H, De la Hoz J, Beltrán J (2017) 2d and 3d virtual interactive laboratories of physics on unity platform. In: Journal of physics: conference series, vol 935. IOP Publishing, p 012069Google Scholar
  10. 10.
    Hornecker E, Stifter M (2006) Learning from interactive museum installations about interaction design for public settings. In: Proceedings of the 18th Australia conference on computer-human interaction: design: activities, artefacts and environments. ACM, pp 135–142Google Scholar
  11. 11.
    Irwansyah F, Yusuf Y, Farida I, Ramdhani M (2018) Augmented reality (ar) technology on the android operating system in chemistry learning. In: IOP Conference series: materials science and engineering, vol 288. IOP Publishing, p 012068Google Scholar
  12. 12.
    Jiménez P, Diez JV, Ordieres-Mere J (2016) Hoshin kanri visualization with neo4j. empowering leaders to operationalize lean structural networks. Procedia CIRP 55:284–289CrossRefGoogle Scholar
  13. 13.
    Kersten TP, Tschirschwitz F, Deggim S (2017) Development of a virtual museum including a 4d presentation of building history in virtual reality. Int Archives Photogrammetry Remote Sens Spatial Inform Sci 42:361CrossRefGoogle Scholar
  14. 14.
    Kopp S, Gesellensetter L, Krämer NC, Wachsmuth I (2005) A conversational agent as museum guide–design and evaluation of a real-world application. In: International workshop on intelligent virtual agents. Springer, pp 329–343Google Scholar
  15. 15.
    Kuhl PK (2004) Early language acquisition: cracking the speech code. Nature Rev Neuroscience 5(11):831–843CrossRefGoogle Scholar
  16. 16.
    Lison P, Kennington C (2016) Opendial: a toolkit for developing spoken dialogue systems with probabilistic rules. ACL 2016:67Google Scholar
  17. 17.
    Martinie C, Navarre D, Palanque P, Barboni E, Canny A (2018) Toucan: an ide supporting the development of effective interactive java applications. In: Proceedings of the ACM SIGCHI symposium on engineering interactive computing systems. ACM, p 4Google Scholar
  18. 18.
    McKeown G, Valstar MF, Cowie R, Pantic M (2010) The SEMAINE corpus of emotionally coloured character interactions. In: Proceedings of ICME, pp 1079–1084Google Scholar
  19. 19.
    Niewiadomski R, Bevacqua E, Mancini M, Pelachaud C (2009) Greta: an interactive expressive eca system. In: Proceedings of the 8th international conference on autonomous agents and multiagent systems-Volume 2, 1399–1400. International Foundation for Autonomous Agents and Multiagent SystemsGoogle Scholar
  20. 20.
    Origlia A, Cosi P, Rodà A, Zmarich C (2017) A dialogue-based software architecture for gamified discrimination tests. In: Proceedings of the first workshop on games-human interaction @ CHItalyGoogle Scholar
  21. 21.
    Origlia A, Paci G, Cutugno F (2017) MWN-E: a graph database to merge morpho-syntactic and phonological data for italian. In: Proceedings of SubsidiaGoogle Scholar
  22. 22.
    Origlia A, Rodà A, Zmarich C, Cosi P, Nigris S, Colavolpe B, Brai I (2018) Gamified discrimination tests for speech therapy applications. In: Proceedings of the annual conference of the Italian association of speech science (AISV) (to appear)Google Scholar
  23. 23.
    Origlia A, Rossi A, Chiacchio ML, Cutugno F (2016) Cultural heritage presentations with a humanoid robot using implicit feedback. In: Proceedings of the AVI-CH workshopGoogle Scholar
  24. 24.
    Origlia A, Savy R, Poggi I, Cutugno F, Alfano I, D’Errico F, Vincze L, Cataldo V (2018) An audiovisual corpus of guided tours in cultural sites: data collection protocols in the chrome project. In: Proceedings of the AVI-CH workshop (to appear)Google Scholar
  25. 25.
    Petersen T (1990) Developing a new thesaurus for art and architecture. Library Trends 38(4):644–658Google Scholar
  26. 26.
    Pianta E, Bentivogli L, Girardi C (2002) Developing an aligned multilingual database. In: Proceedings of the 1st international conference on global wordnetGoogle Scholar
  27. 27.
    Polka L, Jusczyk PW, Rvachew S (1995) Methods for studying speech perception in infants and children, Speech perception and linguistic experience: Issues in cross-language research, 49–89Google Scholar
  28. 28.
    Qiu W, Yuille A (2016) Unrealcv: connecting computer vision to unreal engine. In: European conference on computer vision. Springer, pp 909–916Google Scholar
  29. 29.
    Schmid S (1999) Fonetica e fonologia dell’italiano Paravia scriptoriumGoogle Scholar
  30. 30.
    Shah S, Dey D, Lovett C, Kapoor A (2018) Airsim: high-fidelity visual and physical simulation for autonomous vehicles. In: Field and service robotics. Springer, pp 621–635Google Scholar
  31. 31.
    Shrinivasan YB, Zhang Y (2017) CELIO: an application development framework for interactive spaces. arXiv:1710.01772
  32. 32.
    Squire K, Jenkins H, Holland W, Miller H, Alice O, Philip Tan K, Todd K (2003) Design principles of next-generation digital gaming for education, Educ Technol, 17–23Google Scholar
  33. 33.
    Tallal P (1976) Rapid auditory processing in normal and disordered language development. J Speech Language Hear Res 19(3):561–571CrossRefGoogle Scholar
  34. 34.
    Thiebaux M, Marsella S, Marshall AN, Kallmann M (2008) Smartbody: behavior realization for embodied conversational agents. In: Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems-Volume 1, 151–158. International Foundation for Autonomous Agents and Multiagent SystemsGoogle Scholar
  35. 35.
    De la Torre F, Hodgins J, Bargteil A, Martin X, Macey J, Collado A, Beltran P (2008) Guide to the carnegie mellon university multimodal activity (cmu-mmac) database, Robotics Institute, 135Google Scholar
  36. 36.
    Traum D, Aggarwal P, Artstein R, Foutz S, Gerten J, Katsamanis A, Leuski A, Noren D, Swartout W (2012) Ada and grace: direct interaction with museum visitors. In: Intelligent virtual agents. Springer, pp 245–251Google Scholar
  37. 37.
    Tsao FM, Liu HM, Kuhl PK (2004) Speech perception in infancy predicts language development in the second year of life: a longitudinal study. Child Development 75(4):1067–1084CrossRefGoogle Scholar
  38. 38.
    Valentino M, Origlia A, Cutugno F (2017) Multimodal speech and gestures fusion for small groups. In: Proceedings of the workshop on ”designing, implementing and evaluating mid-air gestures and speech-based interaction” @ CHItaly 2017 [online]Google Scholar
  39. 39.
    Webber J (2012) A programmatic introduction to neo4j. In: Proceedings of the 3rd annual conference on systems, programming, and applications: software for humanity. ACM, pp 217–218Google Scholar
  40. 40.
    Zmarich C, Bonifacio S (2005) Phonetic inventories in Italian children aged 18-27 months: a longitudinal study. In: INTERSPEECH, pp 757–760Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.URBAN/ECO Research CenterUniversity of Naples “Federico II”NapoliItaly
  2. 2.Department of Electrical Engineering and Information TechnologyUniversity of Naples “Federico II”NapoliItaly
  3. 3.Department of Information EngineeringUniversity of PaduaPadovaItaly
  4. 4.Institute of Cognitive Sciences and Technology (CNR-ISTC)PadovaItaly

Personalised recommendations