An ASAP Realizer-Unity3D Bridge for Virtual and Mixed Reality Applications

  • Jan KolkmeierEmail author
  • Merijn Bruijnes
  • Dennis Reidsma
  • Dirk Heylen
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10498)


Modern game engines such as Unity make prototyping and developing experiences for virtual and mixed reality contexts increasingly accessible and efficient, and their value has long been recognized by the scientific community as well. However, these game engines do not have the capabilities to control virtual embodied characters, situated in such environments, with the same expressiveness, flexibility, and generalizability, as offered by modern BML realizers that generate synchronized multimodal behavior from Behavior Markup Language (BML). We implemented a Unity embodiment bridge to the Articulated Social Agents Platform (ASAP) to combine the benefits of a modern game engine and a modern BML realizer. The challenges and solutions we report can help others integrate other game engines with BML realizers, and we end with a glimpse at future challenges and features of our implementation.


BML SAIBA Virtual agents Unity Virtual reality Mixed reality 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bruijnes, M.: Believable suspect agents: response and interpersonal style selection for an artificial suspect. Ph.D. thesis, University of Twente, Enschede, October 2016. sIKS dissertation series no. 2016–39.
  2. 2.
    Hartholt, A., Traum, D., Marsella, S.C., Shapiro, A., Stratou, G., Leuski, A., Morency, L.P., Gratch, J.: All together now: introducing the virtual human toolkit. In: 13th International Conference on Intelligent Virtual Agents, Edinburgh, UK.
  3. 3.
    Kopp, S., Krenn, B., Marsella, S., Marshall, A.N., Pelachaud, C., Pirker, H., Thórisson, K.R., Vilhjálmsson, H.: Towards a common framework for multimodal generation: the behavior markup language. In: Gratch, J., Young, M., Aylett, R., Ballin, D., Olivier, P. (eds.) IVA 2006. LNCS, vol. 4133, pp. 205–217. Springer, Heidelberg (2006). doi: 10.1007/11821830_17CrossRefGoogle Scholar
  4. 4.
    Kopp, S., van Welbergen, H., Yaghoubzadeh, R., Buschmeier, H.: An architecture for fluid real-time conversational agents: integrating incremental output generation and input processing. Journal on Multimodal User Interfaces 8(1), 97–108 (2014)Google Scholar
  5. 5.
    Niewiadomski, R., Bevacqua, E., Mancini, M., Pelachaud, C.: Greta: an interactive expressive eca system. In: Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems-Volume 2, pp. 1399–1400. International Foundation for Autonomous Agents and Multiagent Systems (2009)Google Scholar
  6. 6.
    Reidsma, D., Charisi, V., Davison, D., Wijnen, F., van der Meij, J., Evers, V., Cameron, D., Fernando, S., Moore, R., Prescott, T., Mazzei, D., Pieroni, M., Cominelli, L., Garofalo, R., De Rossi, D., Vouloutsi, V., Zucca, R., Grechuta, K., Blancas, M., Verschure, P.: The EASEL project: towards educational human-robot symbiotic interaction. In: Lepora, N.F.F., Mura, A., Mangan, M., Verschure, P.F.M.J.F.M.J., Desmulliez, M., Prescott, T.J.J. (eds.) Living Machines 2016. LNCS, vol. 9793, pp. 297–306. Springer, Cham (2016). doi: 10.1007/978-3-319-42417-0_27CrossRefGoogle Scholar
  7. 7.
    Reidsma, D., Van Welbergen, H.: Multimodal plan representation for adaptable bml scheduling. In: Autonomous Agents and Multi-Agent Systems. Citeseer (2013)Google Scholar
  8. 8.
    Reidsma, D., van Welbergen, H.: AsapRealizer in practice-a modular and extensible architecture for a bml realizer. Entertainment Computing 4(3), 157–169 (2013)CrossRefGoogle Scholar
  9. 9.
    Shapiro, A.: Building a character animation system. In: Allbeck, J.M., Faloutsos, P. (eds.) MIG 2011. LNCS, vol. 7060, pp. 98–109. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-25090-3_9CrossRefGoogle Scholar
  10. 10.
    van Welbergen, H., Yaghoubzadeh, R., Kopp, S.: AsapRealizer 2.0: The next steps in fluent behavior realization for ECAs. In: Bickmore, T., Marsella, S., Sidner, C. (eds.) IVA 2014. LNCS, vol. 8637, pp. 449–462. Springer, Cham (2014). doi: 10.1007/978-3-319-09767-1_56CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Jan Kolkmeier
    • 1
    Email author
  • Merijn Bruijnes
    • 1
  • Dennis Reidsma
    • 1
  • Dirk Heylen
    • 1
  1. 1.Human Media InteractionUniversity of TwenteEnschedeThe Netherlands

Personalised recommendations