Abstract
In this chapter, we explain the present status of studies for media synchronization in networked multisensory applications with haptics. We also specify the characteristics of haptic media and different features of haptic media from other media such as olfactory, auditory, and visual media. By using such other media together with haptic media, we can get higher realistic sensation when we use the applications for various purposes such as remote education, entertainment, and networked games. When multisensory media streams are transmitted over a network like the Internet, the temporal relationships among the media streams may be disturbed owing to the network delay, delay jitter, and packet loss. Thus, the quality of experience (QoE) may seriously be degraded. To solve this problem, we need to carry out media synchronization control. To achieve a high quality of media synchronization, a number of media synchronization algorithms have been proposed so far. Especially, in networked multisensory applications, we need to take account of human perception of media synchronization errors in the algorithms. This is because the requirements of media synchronization quality depend on types of media. Some algorithms such as Virtual-Time Rendering (VTR) take human perception of the errors into account. For example, VTR tries to accomplish the synchronization by changing the buffering time of each media stream dynamically according to the network delay jitter with several threshold values about the errors. In the algorithms, instead of synchronizing the output timings of media streams exactly, ranges of human perception such as the allowable range, in which users feel that the synchronization error is allowable, the imperceptible range, in which users cannot perceive the error, and the operation range, which is narrower than the imperceptible range and should be kept usually, are taken into account for the sake of high synchronization quality. In this chapter, we explain the algorithms taking account of human perception of the media synchronization errors and enhance other algorithms such as the group (or inter-destination) synchronization control and the adaptive ∆-causality control algorithms for simultaneous output-timing control among multiple terminals by taking account of the perception. It is indispensable to clarify the ranges by QoE assessment in networked multisensory applications. In this chapter, we further make a survey of studies on the assessment. Finally, we discuss the future directions of media synchronization in networked multisensory applications with haptics.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Srinivasan, M.A., Basdogan, C.: Haptics in virtual environments: taxonomy, research status, and challenges. J. Comput. Graph. 21(4), 393–404 (1997)
Kovacs, P.T., Murray, N., Rozinaj, G., Sulema, Y., Rybarova, R.: Application of immersive technologies for education: state of the art. In: Proceedings of 9th International Conference on Interactive Mobile Communication Technologies and Learning (IMCL), pp. 283–288, Nov 2015
Huang, P., Ishibashi, Y.: QoS control and QoE assessment in multi-sensory communications with haptics. IEICE Trans. Commun., Special Section on Quality of Communication Services Improving Quality of Life, E96-B(2), 392–403 (2013)
Ghinea, G., Ademoye, O.: Olfaction-enhanced multimedia: perspectives and challenges. Multimed. Tools Appl. 55(3), 601–626 (2011)
Murray, N., Lee, B., Qiao, Y., Muntean, G.: Olfaction-enhanced multimedia: a survey application domains, displays, and research challenges. ACM Comput. Surv. 48(4), 1–34 (2016)
Murray, N., Qiao, Y., Lee, B., Muntean, G.: Subjective evaluation of olfactory and visual media synchronization. In: Proceedings of ACM Multimedia Systems, pp. 162–171, Mar 2013
Sithu M., Arima, R., Ishibashi, Y.: Experiment on object recognition with haptic, olfactory, and audio senses in virtual environment. IEICE Technical Report, CQ2015, Mar 2016
Ishibashi, Y., Hoshino, S., Zeng, Q., Fukushima, N., Sugawara, S.: QoE assessment of fairness in networked game with olfaction: Influence of time it takes for smell to reach player. Springer’s Multimed. Syst. J. (MMSJ), Special Issue on Network and Systems Support for Games 20(5), 621–631 (2014)
Huang, P., Ishibashi, Y., Fukushima, N., Sugawara, S.: QoE assessment of olfactory media in remote ikebana with haptic media. In: Proceedings of IEEE 2012 International Communications Quality and Reliability (CQR) Workshop, May 2012
Sithu, M., Ishibashi, Y., Huang, P., Fukushima, N.: Influences of network delay on quality of experience for soft objects in networked real-time game with haptic sense. Int. J. Commun. Netw. Syst. Sci. (IJCNS) 8(11), 440–455 (2015)
Sithu, M., Ishibashi, Y., Fukushima, N.: Effects of dynamic local lag control on sound synchronization and interactivity in joint musical performance. ITE Trans Media Technol. Appl., Special Section on Multimed. Transmission System and Services 2(4), 299–309 (2014)
Gunn, C., Hutchins, M., Steveson, D., Adcock, M., Youngblood, P.: Using collaborative haptics in remote surgical training. In: Proceedings of the First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, pp. 1–2, Mar 2005
Mizuno, H., Ishibashi, Y., Sugawara, S.: A comparison between transmission methods of sense of force in a remote haptic instruction system. In: Proceedings of IEEE ICME, pp. 1202–1205, June/July 2009
Zeng, Q., Iwata, K., Ishibashi, Y., Fukushima, N., Sugawara, S.: Influence of network delay in networked ensemble with haptic media, sound, and video. In: National Convention of IPSJ, 3Y-5, Mar 2012
Lun, M., Friedman, D., Sankaranarayanan, G., King, H., Fodero, K., Leuschke, R., Hannaford, B.: The RAVEN: design and validation of a telesurgery system. Int. J. Robot. Res. 1(1), 1–16 (2009)
ITU E. 800, Terms and definitions related to quality of service and network performance including dependability, Aug 1994
ITU-T Rec. P. 10/G. 100 Amendment 1, New appendix I—definition of quality of experience (QoE). Int. Telecommun. Union, Jan 2007
Blakowski, G., Steinmetz, R.: A media synchronization survey: reference model, specification, and case studies. IEEE J. Sel. Areas Commun. 14(1), 5–35 (1996)
Huang, Z., Nahrstedt, K., Steinmetz, R.: Evolution of temporal multimedia synchronization principles: a historical viewpoint. ACM Trans. Multimed. Comput. Commun. Appl. 9(1), 34 (2013)
Sithu, M., Ishibashi, Y.: Media synchronization control in multimedia communication. IGI Global Emerg. Res. Netw. Multimed. Commun. Syst. Chapter 2, 25–61 (2015)
Ishibashi, Y., Tasaka, S., Hasegawa, T.: The Virtual-Time Rendering algorithm for haptic media synchronization in networked virtual environments. In: Proceedings of the 16th International Workshop on Communications Quality and Reliability (CQR), pp. 213–217, May 2002
Ishibashi, Y., Kanbara, T., Tasaka, S.: Inter-stream synchronization between haptic media and voice in collaborative virtual environments. In: Proceedings of ACM Multimedia, pp. 604–611, Oct 2004
Sannomiya, H., Osada, J., Ishibashi, Y., Fukushima, N., Sugawara, S.: Inter-stream synchronization control with group synchronization algorithm. In: Proceedings of the 2nd IEEE Global Conference on Consumer Electronics (GCCE), pp. 520–524, Oct 2013
Ishibashi, Y., Tasaka, S.: A synchronization mechanism for continuous media in multimedia communications. In: Proceedings of IEEE INFOCOM, pp. 1010–1019, Apr 1995
Sithu, M., Ishibashi, Y., Fukushima, N.: Effects of dynamic local lag control on sound synchronization and interactivity in joint musical performance. ITE Trans Media Technol. Appl., Special Section on Multimedia Transmission System and Services 2(4), 299–309 (2014)
Ishibashi, Y., Tasaka, S.: A distributed control scheme for group synchronization in multicast communications. In: Proceedings of International Symposium on Communications (ISCOM), pp. 317–323, Nov 1999
Ishibashi, Y., Hashimoto, Y., Ikedo, T., Sugawara, S.: Adaptive ∆-causality control with adaptive dead-reckoning in networked games. In: Proceedings of the 13th Annual Workshop on Network and Systems Support for Games (NetGames), pp. 75–80, Sep 2007
Marshall, A., Wai, Y.: Haptic virtual environment performance over IP networks: a case study. In: Proceedings of IEEE International Symposium on Distributed Simulation and Real-Time Applications, pp. 181–189, Oct 2003
El Saddik, A.: The potential of haptics technologies. IEEE Instrum. Meas. Mag. 10(1), 10–17 (2007)
Lin, M., Salisbury, K.: Haptic rendering: beyond visual computing. IEEE Comput. Graph. Appl. 24(2), 22–23 (2004)
Salisbury, K., Conti, F., Barbgli, F.: Haptic rendering: Introductory concepts. IEEE Comput. Graph. Appl. 24(2), 24–32 (2004)
Weller, R., Zachmann, G.: User performance in complex bimanual haptic manipulation with 3DoFs vs. 6DoFs. In: Proceedings of IEEE Haptics, Mar 2012
Sato, M.: Development of string-based force display: SPIDAR. In: Proceedings of the 8th International Conference on Virtual System and Multi Media (VSMM), pp. 1034–1039, Sept 2002
Novint Technologies, Inc., Haptic device abstraction layer programmer’s guide, version 1.1.9 Beta, Sept 2007
Gosselin, F., Riwan, A.: Design of virtuose 3D: a new haptic interface for teleoperation and virtual reality. In: Proceedings of ICAR, pp. 205–212, Aug 2001
Vanderlinde, R.Q., Lammertse, P., Frederiksen, E., Ruiter, B.: The hapticmaster: a new high-performance haptic interface. In: Proceedings of EuroHaptics, pp. 1–5 (2002)
Immersion Corporation: CyberGrasp. http://www.immersion.com/3d/products/cybergrasp.php
Yoo,Y., Sung, M., Kim, N., Jun, K.: An experimental study on the performance of haptic data transmission in networked haptic collaboration. In: Proceedings of International Conference on Advanced Communication Technology, pp. 657–662, Feb 2007
Osada, J., Ishibashi, Y., Fukushima, N., Sugawara, S.: QoE comparison of haptic control schemes in remote instruction system with haptic media, video, and voice. In: Proceedings of International Technical Conference on Circuits/Systems, Computers and Communications (ITC-CSCC), July 2012
Khademian, B., Hashtrudi-Zaad, K.: Performance issues in collaborative haptic training. In: Proceedings of IEEE International Conference on Robotics and Automation (ICRA), pp. 3275–3263, Apr 2007
Willaert, B., Corteville, B., Reynaerts, D., Brussel, H., Poorten, E.: Bounded environment passivity of the classical position-force teleoperation controller. In: Proceedings of IEEE/RSJ IROS, Oct 2009
Fujimoto, T., Ishibashi, Y., Sugawara, S.: Influences of inter-stream synchronization error on collaborative work in haptic and visual environments. In: Proceedings of Haptics, pp. 113–119, Mar 2008
Tatematsu, A., Ishibashi, Y., Fukushima, N., Sugawara, S.: QoE assessment in tele-operation with 3D video and haptic media. In: Proceedings of IEEE ICME Workshop on Hot Topics in Multimedia, July 2011
Zeng, Q., Ishibashi, Y., Fukushima, N., Sugawara, S., Psannis, K.E.: Influences of inter-stream synchronization errors among haptic media, sound, and video on quality of experience in networked ensemble. In: Proceedings of IEEE GCCE, pp. 466–470, Oct 2013
Ishibashi, Y., Kaneoka, H.: Faieness among game players in networked haptic environments: influence of network latency. In: Proceedings of IEEE ICME, July 2005
Ishibashi, Y., Tasaka, S.: Causality and media synchronization control for networked multimedia games: centralized versus distributed. In: Proceedings of NetGames, pp. 42–51, May 2003
Hoshino, S., Ishibashi, Y., Fukushima, N., Sugawara, S.: QoE assessment in olfactory and haptic media transmission: Influence of inter-stream synchronization error. In: Proceedings of IEEE CQR, May 2011
Nakano, S., Maeda, Y., Ishibashi, Y., Fukushima, N., Huang, P., Psannis, K.E.: Influence of network delay on fairness between players in networked game with olfactory and haptic senses. IEICE Technical Report, CQ2014-94, Jan 2015
Tasaka, S. Ishibashi, Y.: Mutually compensatory property of multimedia QoS. In: Conference Record IEEE ICC, pp. 1105–1111, Apr/May 2002
Tasaka, S., Ito, Y.: Psychometric analysis of the mutually compensatory property of multimedia QoS. In: Conference Record IEEE ICC, pp. 1880–1886, May 2003
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Definitions
- Quality of Service (QoS)
-
QoS is the quality of service which is provided from a layer to its upper layer in network layer model and it is defined as how much the service is faithful to the ideal situation.
- Quality of Experience (QoE)
-
QoE is also called the user-level QoS. QoE is the quality which is perceived subjectively and/or experienced objectively by end-users.
- Human perception
-
Human perception is the mental process in which human beings perceive or recognize an object or idea. In this chapter we quantitatively express human perception of synchronization error as ranges in which users can feel/perceive/allow the synchronization error.
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this chapter
Cite this chapter
Huang, P., Sithu, M., Ishibashi, Y. (2018). Media Synchronization in Networked Multisensory Applications with Haptics. In: Montagud, M., Cesar, P., Boronat, F., Jansen, J. (eds) MediaSync. Springer, Cham. https://doi.org/10.1007/978-3-319-65840-7_11
Download citation
DOI: https://doi.org/10.1007/978-3-319-65840-7_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-65839-1
Online ISBN: 978-3-319-65840-7
eBook Packages: Computer ScienceComputer Science (R0)