Multimedia Tools and Applications

, Volume 55, Issue 3, pp 353–378 | Cite as

The role of mismatches in the sensory feedback provided to indicate selection within a virtual environment

  • Raquel Viciana-AbadEmail author
  • Arcadio Reyes-Lecuona
  • Matthieu Poyade
  • José Escolano


It is generally understood that virtual reality simulations have a high computational cost. Hence, they rarely can reduce completely all the incoherence within the cross-modal sensory outputs provided. The main research approaches to date have consisted in technically reducing possible mismatches, however minimal research has been conducted so as to analyse their influence on human capabilities. Thus, the objective of this study is to provide further insights to the designers of virtual reality about the negative influence of simulation lags and interesting design implications. To clearly show this, we have investigated the importance of coherent sensory feedback by incorporating time delays and spatial misalignments in the feedback provided by the simulation as a response to participant´s actions to mimic computationally expensive environments. We have also evaluated these misalignments considering two typical interaction setups. In particular, the sensory mismatches influence has been assessed in human factors, such as the sense of presence, task performance and delay perception. Our experimental results indicate that the closer the interaction conditions are to real configurations the higher the sensory requirements are regarding accuracy. The implications of this study offer the designer guidelines to prioritise the reduction of those mismatches in the sensory cues provided depending on the simulations goals.


Co-location Delay Feedback Mismatches Performance Presence Selection 



The authors wish to thank the participants in the study for their collaboration and comments. This work has been partially supported by DIANA group (University of Málaga) and by the University of Jaén through project UJA2009/12/12.


  1. 1.
    Allison RS, Harris LR, Jenkin M, Jasiobedzka U, Zacher JE (2001) Tolerance of temporal delay in virtual environments. In Proc. of IEEE Virtual Reality, pp 247–254. doi:
  2. 2.
    Ardito C, Costabile MF, Angeli A, Pittarello F (2007) Navigation help in 3D worlds: some empirical evidences on use of sound. Multimed Tools Appl 33:201–216. doi: 10.1007/s11042-006-0060-0 CrossRefGoogle Scholar
  3. 3.
    Arsenault R, Ware C (2000) Eye-hand co-ordination with force feedback. In Proc. of the SIGCHI conf. on Human factors in computing systems, pp 408–414Google Scholar
  4. 4.
    Bailenson JN, Yee N (2008) Virtual interpersonal touch: haptic interaction and copresence in collaborative virtual environments. Multimed Tools Appl 37:5–14. doi: 10.1007/s11042-007-0171-2 CrossRefGoogle Scholar
  5. 5.
    Begault DR (2000) 3-D sound for virtual reality and multimedia. Facsimile reprint of 1994 edition, Preface to the NASA Technical Memorandum version, Ames Research Center, Moffett Field, CA, USAGoogle Scholar
  6. 6.
    Boring EG (1929) A history of experimental psychology. Appleton Century Crofts, New YorkGoogle Scholar
  7. 7.
    Boukerche A, Shirmohammadi S, Hossain A (2006) Moderating simulation lag in haptic virtual environments. In Proc. of the 39th Annual Symposium on Simulation, pp 269–277, IEEE Computer Society, Washington, USA. doi: 10.1109/ANSS.2006.31
  8. 8.
    Burke JL, Prewett MS, Gray AA, Yang L, Stilson, FRB, Coovert MD, Elliot LR, Redden E (2006) Comparing the effects of visual-auditory and visual-tactile feedback on user performance: a meta-analysis. In Proc. Multimodal interfaces, pp 108–117. ACM, NY, USAGoogle Scholar
  9. 9.
    Burns E, Razzaque S, Whitton M, McCallus M, Panter A, Brooks FP (2005) The hand is slower than the eye: a quantitative exploration of visual dominance over proprioception. In Proc. of IEEE Virtual Reality, pp 3–10Google Scholar
  10. 10.
    Buttazzo G, Lipari G, Abeni L, Caccamo M (2005) Soft real-time systems: predictability vs. efficiency (Series in Computer Science). Springer, BerlinzbMATHGoogle Scholar
  11. 11.
    Congedo M, Lécuyer A, Gentaz E (2006) The influence of spatial delocation on perceptual integration of vision and touch. Presence-Teleoper Virtual Env 15:353–357CrossRefGoogle Scholar
  12. 12.
    De Boeck J, Raymaekers C, Coninx K (2005) Are existing metaphors in virtual environments suitable for haptic interaction. In Proc. of Virtual Reality, pp 261–268. Laval, FranceGoogle Scholar
  13. 13.
    Durlach PJ, Fowlkes J, Metevier CJ (2005) Effect of variations in sensory feedback on performance in a virtual reaching task. Presence-Teleoper Virtual Env 14:450–462CrossRefGoogle Scholar
  14. 14.
    Ehmann S, Lin MC (2001) Accurate and fast proximity queries between polyhedral using convex surface decomposition. In Proc. of Eurographics, 20, pp 500–510Google Scholar
  15. 15.
    Fujisaki W, Shimojo S, Kashino M, Nishida S (2004) Recalibration of audiovisual simultaneity. Nat Neurosci 7:773–778CrossRefGoogle Scholar
  16. 16.
    Harris CS (1965) Perceptual adaptation to inverted, reversed, and displaced vision. Psychol Rev 72:419–444CrossRefGoogle Scholar
  17. 17.
    Hirsh IJ, Sherrick CE (1961) Perceived order in different sense modalities. J Exp Psychol 62:423–432CrossRefGoogle Scholar
  18. 18.
    Ijsselsteijn WA, deKort YAW, Haans A (2006) Is this my hand I see before me? The rubber hand illusion in reality, virtual reality, and mixed reality. Presence-Teleoper Virtual Env 15:455–464. doi: 10.1162/pres.15.4.455 CrossRefGoogle Scholar
  19. 19.
    Jay C, Hubbold R (2005) Delayed visual and haptic feedback in a reciprocal tapping task. In Proc. of World Haptics, pp 655–656. doi:
  20. 20.
    Jay C, Hubbold R (2006) Quantifying the effects of latency on sensory feedback in distributed virtual environments. In Proc. of Virtual Images Seminar, pp 9–16. Paris, FranceGoogle Scholar
  21. 21.
    Keetels M, Vroomen J (2007) No effect of auditory-visual spatial disparity on temporal recalibration. Exp Brain Res 182:559–565. doi: 10.1007/s00221-007-1012-2 CrossRefGoogle Scholar
  22. 22.
    Larsson P, Västfjäll D, Kleiner, M (2001) Ecological acoustics and the multi-modal perception of rooms: real and unreal experiences of auditory-visual virtual environments. In Proc. of the Conf. on Auditory Display, pp 245–249, Espoo, FinlandGoogle Scholar
  23. 23.
    Lécuyer A, Burkhardt JM, Coquillart S, Coiffet P (2001) “Boundary of illusion”: an experiment of sensory integration with a pseudo-haptic system. In Proc. of IEEE Virtual Reality, pp 115–122Google Scholar
  24. 24.
    Lee I, Choi S (2007) Discrimination of virtual environments under visual and haptic rendering delays, frontiers in the convergence of bioscience and information technologies, pp 554–562. doi:
  25. 25.
    Levitin DJ, MacLean K, Mathews M, Chu L, Jensen E (2000) The perception of cross-modal simultaneity. In AIP Conf. Proc. 517, pp 323, doi: 10.1063/1.1291270
  26. 26.
    Loomis JM (1992) Distal attribution and presence. Presence-Teleoper Virtual Env 1:113–119Google Scholar
  27. 27.
    MacKenzie IS, Ware C (1993) Lag as a determinant of human performance in interactive systems. In Proc. of the INTERACT ’93 and CHI ’93, pp 488–493, NY, USA. doi:
  28. 28.
    Morein-Zamir S, Soto-Faraco S, Kingstone A (2003) Auditory capture of vision: examining temporal ventriloquism. Cogn Brain Res 17:154–163. doi: 10.1016/S0926-6410(03)00089-2 CrossRefGoogle Scholar
  29. 29.
    Munhall KG, Gribble P, Sacco L, Ward M (1996) Temporal constraints on the McGurk effect. Percept Psychophys 58:351–362CrossRefGoogle Scholar
  30. 30.
    Navarra J, Soto-Faraco S, Spence C (2007) Adaptation to audiotactile asynchrony. Neurosci Lett 413:72–76. doi: 10.1016/j.neulet.2006.11.027 CrossRefGoogle Scholar
  31. 31.
    Pellegrini RS (2001) Quality assessment of auditory virtual environments. In Proceedings of ICAD, Espoo, FinlandGoogle Scholar
  32. 32.
    Radeau M, Bertelson P (1977) Adaptation to auditory-visual discordance and ventriloquism in semirealistic situations. Percept Psychophys 22:137–146CrossRefGoogle Scholar
  33. 33.
    Rossetti Y, Koga K, Mano T (1993) Prismatic displacement of vision induces transient changes in the timing of eye-hand coordination. Percept Psychophys 54:355–364CrossRefGoogle Scholar
  34. 34.
    Rothrock L, Barron K, Simpson T, Frecker M, Ligetti C, Barton R (2006) Applying the proximity compatibility and the control-display compatibility principles to engineering design interfaces. Hum Factors Ergon Manuf 16:61–81. doi: 10.1002/hfm.v16:1 CrossRefGoogle Scholar
  35. 35.
    Ryu S, Kim H, Park J, Kwon Y, Jeong C (2007) Collaborative object-oriented visualization environment. Multimed Tools Appl 3:209–234. doi: 10.1007/s11042-006-0066-7 CrossRefGoogle Scholar
  36. 36.
    Simpson TW, Barron K, Rothrock L, Frecker M, Barton RR, Ligetti C (2007) Impact of response delay and training on user performance with text-based and graphical user interfaces for engineering design. Res Eng Des 18:49–65. doi: 10.1007/s00163-007-0033-y CrossRefGoogle Scholar
  37. 37.
    Slater M, Usoh M, Steed A (1994) Depth of presence in virtual environments. Presence-Teleoper Virtual Env 3:113–119Google Scholar
  38. 38.
    Sousa Santos B, Dias P, Pimentel A, Baggerman J, Ferreira C, Silva S, Madeira J (2009) Head-mounted display versus desktop for 3D navigation in virtual reality: a user study. Multimed Tools Appl 4:161–181. doi: 10.1007/s11042-008-0223-2 CrossRefGoogle Scholar
  39. 39.
    Spence C, Shore DI, Klein RM (2001) Multisensory prior entry. J Exp Psychol Gen 130:799–832CrossRefGoogle Scholar
  40. 40.
    Swapp D, Pawar V, Loscos C (2006) Interaction with co-located haptic feedback in virtual reality. Virtual Real 10:24–30. doi: 10.1007/s10055-006-0027-5 CrossRefGoogle Scholar
  41. 41.
    Szameitat AJ, Rummel J, Szameitat DP, Sterr A (2009) Behavioral and emotional consequences of brief delays in human–computer interaction. Int J Hum Comput Stud 67:561–570. doi: 10.1016/j.ijhcs.2009.02.004 CrossRefGoogle Scholar
  42. 42.
    Theile G (1993) Trends and activities in the development of multichannel sound systems. In Proc. of AES International Conf., pp 180–187Google Scholar
  43. 43.
    Vatakis A, Spence C (2006) Audiovisual synchrony perception for music, speech, and object actions. Brain Res 1111:134–142. doi: 10.1016/j.brainres.2006.05.078 CrossRefGoogle Scholar
  44. 44.
    Viciana-Abad R, Reyes-Lecuona A (2008) Effects of Co-location and crossmodal interaction between haptic, auditory and visual cues in presence. Haptics: Perception, Devices and Scenarios, LNCS 5024: 832–837, Berlin: Springer. doi: 10.1007/978-3-540-69057-3
  45. 45.
    Vroomen J, Keetels M, de Gelder B, Bertelson P (2004) Recalibration of temporal order perception by exposure to audio-visual asynchrony. Cogn Brain Res 2:32–35. doi: 10.1016/j.cogbrainres.2004.07.003 CrossRefGoogle Scholar
  46. 46.
    Watson B, Walker N, Ribarsky B, Spaulding V (1999) Managing temporal detail in virtual environments: relating system responsiveness to feedback. In Proc. of CHI ’99, pp 280–281, NY, USA. doi:
  47. 47.
    Watson B, Walker N, Woytiuk P, Ribarsky W (2003) Maintaining usability during 3D placement despite delay. In Proc. of IEEE Virtual Reality Conf., pp 133–140. doi:
  48. 48.
    Witmer BG, Jerome CJ, Singer MJ (2005) The factor structure of the presence questionnaire. Presence-Teleoper Virtual Env 14:298–331. doi: 10.1162/105474605323384654 CrossRefGoogle Scholar
  49. 49.
    Woszczyk W (1993) Quality assessment of multichannel sound recordings. In Proc. of AES Int. Conf., pp 197–218Google Scholar
  50. 50.
    Yang U, Jounghyun KG (2004) Increasing the effective egocentric field of view with proprioceptive and tactile feedback. In Proc. of IEEE Virtual Reality, pp 27–34Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  • Raquel Viciana-Abad
    • 1
    Email author
  • Arcadio Reyes-Lecuona
    • 2
  • Matthieu Poyade
    • 2
  • José Escolano
    • 1
  1. 1.Department of Telecommunication EngineeringUniversity of JaénJaénSpain
  2. 2.Department of Electronic TechnologyUniversity of MálagaMálagaSpain

Personalised recommendations