Serious Games: Customizing the Audio-Visual Interface

  • Bill Kapralos
  • Robert Shewaga
  • Gary Ng
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8526)


Serious games are gaining in popularity within a wide range of educational and training applications given their ability to engage and motivate learners in the educational process. Recent hardware and computational advancements are providing developers the opportunity to develop applications that employ a high level of fidelity (realism) and novel interaction techniques. However, despite these great advances in hardware and computational power, real-time high fidelity rendering of complex virtual environments (found in many serious games) across all modalities is still not feasible. Perceptual-based rendering exploits various aspects of the multi-modal perceptual system to reduce computational requirements without any resulting perceptual effects on the resulting scene. A series of human-based experiments demonstrated a potentially strong effect of sound on visual fidelity perception, and task performance. However, the resulting effects were subjective whereby the influence of sound was dependent on various individual factors including musical listening preferences. This suggests the importance of customizing (individualizing) a serious game’s virtual environment with respect to audio-visual fidelity, background sounds, etc. In this paper details regarding this series of audio-visual experiments will be provided followed by a description of current work that is examining the customization of a serious game’s virtual environment by each user through the use of a game-based calibration method.


Serious games virtual simulation audio-visual interaction audio-visual fidelity calibration 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Robison, R.A., Liu, C.Y., Apuzzo, M.L.J.: Man, mind, and machine: The past and future of virtual reality simulation in neurologic surgery. World Nuerosurgery 76(5), 419–430 (2011)CrossRefGoogle Scholar
  2. 2.
    Farmer, E., von Rooij, J., Riemersma, J., Joma, P., Morall, J.: Handbook of simulator based training. Ashgate Publishing Limited, Surrey (1999)Google Scholar
  3. 3.
    Hulusic, V., Harvey, C., Debattista, K., Tsingos, N., Walker, S., Howard, D., Chalmers, A.: Acoustic rendering and auditory-visual cross-modal perception and interaction. Computer Graphics Forum 31(1), 102–131 (2012)CrossRefGoogle Scholar
  4. 4.
    Norman, G., Dore, K., Grierson, L.: The minimal relationship between simulation fidelity and transfer of learning. Medical Education 46(7), 636–647 (2012)CrossRefGoogle Scholar
  5. 5.
    Blascovich, J., Bailenson, J.: Infinite Reality. Harper Collins, New York (2011)Google Scholar
  6. 6.
    Mastoropoulou, G., Debattista, K., Chalmers, A., Troscianco, T.: The influence of sound effects on the perceived smoothness of rendered animations. In: Proceedings of the Symposium on Applied Perception in Graphics and Visualization 2005, La Coruna, Spain, August 26-28, pp. 9–15 (2005)Google Scholar
  7. 7.
    Larsson, P., Vstjll, D., Kleiner, M.: On the quality of experience: A multi-modal approach to perceptual ego-motion and sensed presence in virtual environments. In: Proceedings of the First International Speech Communications Association Tutorial and Research Workshop on Auditory Quality of Systems, Akademie Mont-Cenis, Germany (2003)Google Scholar
  8. 8.
    Carlile, S.: Virtual auditory space: Generation and application. R. G. Landes, Austin TX (1996)CrossRefGoogle Scholar
  9. 9.
    Mania, K., Wooldridge, D., Coxon, M., Robinson, A.: The effect of visual and interaction fidelity on spatial cognition in immersive virtual environments? IEEE Transactions on Visualization and Computer Graphics 12(3), 396–404 (2006)CrossRefGoogle Scholar
  10. 10.
    Storms, S.L., Zyda, M.J.: Interactions in perceived quality of auditory-visual displays. Presence: Teleoperators and Virtual Environments 9(6), 557–580 (2000)CrossRefGoogle Scholar
  11. 11.
    Hulusic, V., Debattista, K., Aggarwal, V., Chalmers, A.: Maintaining frame rate perception in interactive environments by exploiting audio-visual cross-modal interaction. The Visual Computer 27(1), 57–66 (2011)CrossRefGoogle Scholar
  12. 12.
    Bonneel, N., Suied, C., Viaud-Delmon, I., Drettakis, G.: Bimodal perception of audio-visual material properties for virtual environments. ACM Transactions on Applied Perception 7(1), 1–16 (2010)CrossRefGoogle Scholar
  13. 13.
    Shams, L., Kim, R.: Crossmodal influences on visual perception. Physics of Life Reviews 7(3), 295–298 (2010)CrossRefGoogle Scholar
  14. 14.
    Rojas, D., Kapralos, B., Cristancho, S., Collins, K., Conati, C., Dubrowski, A.: The effect of background sound on visual fidelity perception. In: Proceedings of ACM Audio Mostly 2011 (Extended Abstracts), Coimbra, Portugal, September 7-9, pp. 1–7 (2011)Google Scholar
  15. 15.
    Rojas, D., Kapralos, B., Cristancho, S., Collins, K., Hogue, A., Conati, C., Dubrowski, A.: Developing effective serious games: The effect of background sound on visual fidelity perception with varying texture resolution. Studies in Health Technology and Informatics 173, 386–392 (2013)Google Scholar
  16. 16.
    Rojas, D., Kapralos, B., Hogue, A., Collins, K., Nacke, L., Cristancho, S., Conati, C., Dubrowski, A.: The effect of sound on visual fidelity perception in stereoscopic 3-d. IEEE Transactions on System Man and Cybernetics Part B 43(6), 1572–1583 (2013)Google Scholar
  17. 17.
    Rojas, D., Kapralos, B., Collins, K., Dubrowski, A.: The effect of contextual sound cues on visual fidelity perception. Studies in Health Technology and Informatics (to appear, 2014)Google Scholar
  18. 18.
    Cowan, B., Rojas, D., Kapralos, B., Collins, K., Dubrowski, A.: Spatial sound and its effect on visual quality perception and task performance within a virtual environment. In: Proceedings of the 21st International Congress on Acoustics, Montreal, Canada, June 2-7, pp. 1–7 (2013)Google Scholar
  19. 19.
    Kapralos, B., Moussa, F., Dubrowski, A.: An overview of virtual simulations and serious games for surgical education and training. In: Brooks, A.L., Braham, S., Jain, L.C. (eds.) Technologies of Inclusive Well-Being. SCI, vol. 536, pp. 289–306. Springer, Heidelberg (2014)CrossRefGoogle Scholar
  20. 20.
    Kerr, D.S.: The refractor/phoropter an important tool in vision correction (December 7, 2010),
  21. 21.
    Tawadrous, M., Hogue, A., Kapralos, B., Collins, K.: An interactive in-game approach to user adjustment of stereoscopic 3D settings. In: Proceedings of Stereoscopic Displays and Applications XXIV, San Francisco, CA, USA, February 3-7, pp. 1–7 (2007)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Bill Kapralos
    • 1
  • Robert Shewaga
    • 1
  • Gary Ng
    • 1
  1. 1.Faculty of Business and Information TechnologyUniversity of Ontario Institute of TechnologyOshawaCanada

Personalised recommendations