Keywords

1 Introduction

Since the 1980s when portable music players became popular, we have been to enjoy pre-recorded music with sophisticated and lightweight players in an easy and familiar way. These players have small speakers or need external headphones that are not advantageous for creating a high-powered deep bass sound, although musical experiences in live performances or movie theaters provide a strong presence of sound as a physical phenomenon with sensible vibrations. Such experiences are realized by a powerful output of volume using large-scale devices, such as large speakers or woofers, that are not suitable for personal use.

Humans perceive various stimuli as being multimodally combined. Sub-channel stimuli can change the perception of the main stimulus, as can be seen in McGurk effect [5]. Consequently, we considered that the sound and music experience of live performance or theater with real mega volume is not perceived by the sense of hearing alone. We especially focused on the sense of touch on sound vibration; as Merchel [6] discussed in relation to the influence of the listener’s experience of whole-body vibrations, the natural feeling of sound and music is expected to provide vibrations to a greater or lesser extent.

To achieve the experience of realistic sound and music of massive volume but without expensive audio equipment, we have developed a wearable interface with a vibration-motor array that produces vibrotactile expressions in synchronization with the sound of music. Multiple vibrators provide a localized illusion of both sound and vibration. Furthermore, we focused on an illusion of the physical penetration of a sound source through the user’s body based on fictive physical presence by the differences in starting times among the vibration motors. Not just massive volume at live concerts, localized vibration would aid in creating a realistic experience of immersive opera and parades with actors’ movements. The physical penetration is also expected to emphasize the presence of virtual objects passing through the user’s body in 3D theaters. As described above, the surrounding vibrations reproduce these physical experiences.

Thus, the localized vibration and illusion of physical penetration would also provide a novel music experience. Moreover, the sound localization would become easier not only based on normal differences of level and time (ILD: interaural level difference and ITD: interaural time difference) among multiple vibrations but also based on the illusion of the physical penetration from the ongoing sound.

As an application of this configuration, the use of the vibration-motor array as a monitor combined with sound feedback would help a player in a live performance, especially for directional and expressive assignments while the wearable device does not restrict the user’s actions. As another application, the proposed method is also expected to give an impact on people with hearing difficulties. The vibration can individually express the rhythm of music even without any sound.

2 Related Works

First, we refer to studies that have suggested the importance of vibration to the experience of music. Merchel and Altinsoy [5] discussed the effect of whole-body vibrations on the listener’s experience of listening to concert recordings with a box-type chair with a large vibrating system in it. Their results showed that music with vibration, –though only from the buttocks,– was judged better than simple sound alone. Chang et al. [1] suggested that audio-based vibrotactile feedback on a mobile phone is preferred to simple audio and is recognized as better sound. Thus, it is presumed that vibration improves the musical experience, especially with a small device and low volume. Yoshida et al. [11] provided vibration stimuli in the user’s palm using an electromagnetic coil device driven by low-frequency components of the sound. Their evaluation showed that the different types of vibrations elevated the listener’s sensitivity.

There are researches on educational systems in which vibration is used as a tool for learning music. Miura and Sugimoto [7] aimed to give each person a vibration of a particular rhythm of the assigned part to learn on an instrument in ensemble. They showed that a vibration device could become an effective tool for rhythm learning. Similar to their application, our proposed system is also expected to be adopted not only for rhythm learning but also for understanding sound localization to develop musical experiences, though our main purpose is not education itself.

Second, there are discussions on various fixed/wearable systems of haptic expressions for realistic sensation of sound and musical experience. Lindeman et al. designed VR immersive devices with multiple haptic feedbacks covering the user’s upper body [8, 9]. You et al. [12] developed a sound-specific vibration-array interface according to a sound stream using 16 oscillators. They found the shoulder was the most effective place to fix the attachment among the wrist, forearm, upper arm, shoulder or neck to increase immersiveness. There is also a palm-sized mobile haptic device with dual-band stimuli to produce low- and high-frequency bands for musical experiences [2]. The dual-band stimuli showed predominance in making a good impression; however, the user’s action is limited while holding the device in the hand. In contrast, our wearable system does not restrict the action of the user. According to their dual-channel feedbacks, we applied different frequency bands to express various types of music. Sakuragi et al. [10] reduced vibration motors to a minimum setup with actuators only at the clavicle to efficiently provide wide vibrations with as far-reaching stimuli as possible. For more extended experiences, we focused on the vibration-motor array on not only the clavicle but also other places on the user’s body.

The above three music devices [2, 10, 12] cover a narrow surface of the body from only one direction except Lindeman’s VR garment [8], which was not for a musical experience. Even the 4x4 table vibration array has difficulty in generating distinct localizations because it covers only a narrow surface. The palm-sized device and the simple placement on the clavicle also cannot provide vibrations on a wide or surrounding surface. For an immersive experience and localization of the sound from various directions, we considered that a wider surface and arrayed multiple vibrations should be prepared. Accordingly, we adopted multiple vibration motors placed in various directions from the center of the body. Also, the three above studies used small vibrators that made it difficult to create strong and slow vibrations like real loudspeakers. In contrast, we adopted two types of vibration motors: low-frequency and strong vibrators for the low frequency band and a small brushless motor for the midrange frequency band.

Chair-type systems provide vibration stimuli with sound for a similar purpose. Karam et al. [3] suggested a chair-type haptic system fusing eight channels. The system expresses various vibrations of multiple musical instruments in music using haptic devices fixed on various parts of the body. There is also a chair-type vibration machine, Bodysonic [4], to extend the sense of hearing music. The chair-type device is considered to cover the user’s body; however, the strong sound experience in a live performance provides the music experience with sound from the front to the back.

Differently to Karam et al., we focus on haptic vibrations from the front, back, left, and right to elevate sound localization and realistic sensation as though the user were surrounded with musical sound. The novel point of our proposed system is the differences in starting times of the vibrations among multiple vibration motors to create the illusion of physical penetration as though the sound had a physical presence.

3 Physical Experience System of Sound and Music

3.1 Summary of System Structure

In order to expand the sound and experience of using portable headphones/earphones, we implemented a synchronized vibration-motor array system as follows. The system consists of (1) a sound analysis unit and (2) a control unit of a vibration-motor array (see Fig. 1). First, the sound-analysis unit analyzes the signal of ongoing music. Second, the control unit of the vibration-motor array (1) distributes the level of each vibration motor and (2) calculates the time delay for each motor based on the analyzed data and parameter settings. Finally, the original hardware device of the vibration-motor array produces vibrations in synchronization with the sounds of the music.

Fig. 1.
figure 1

System structure

Full size image

3.2 Hardware Structure and Vibration-motor-array Device

The hardware of the system consists of a laptop PC, a set of headphones, an original wearable set of the vibration-motor array, an AVR microcomputer (Arduino UNO), and an Nch-independent MOSFET.

The proposed vibration-motor array device is configured as a wearable belt set for the chest with four vibration motors and a choker with four vibration motors (Figs. 2 and 3). Two dual-axis vibration motors of 6000 RPM (revolutions per minute), which are assigned to the low-frequency power of the sound, are also placed on the upper rubber band of the chest belt in the vicinity of the third and fourth ribs. Two of the same 6000 RPM motors are placed on the upper rubber band of the belt on the left and right sides of the back 15 cm from the center of the spine. On the choker, two micro-brushless vibration motors of 13000 RPM, which are assigned to the middle-frequency power of sound, are placed on the left and right of the front of neck near the clavicles. Two of the same motors are also placed on the left and right at the back of the neck. The left and right pairs of micro-brushless vibration motors are placed at 8 cm intervals.

The AVR microcomputer performs pulse width modulation (PWM) controls for each of eight channels through the Nch-independent MOSFET in synchronization with the volume levels of each frequency band based on the data sent by the sound-analysis unit. The left motors are operated by reflecting the level of sound source from the left channel. The right motors reflect the right channel. Accordingly, the difference in the intensity of vibration between the left and right channels is automatically reproduced, including several-millisecond differences, such as the interaural time difference between the left and right ears. Therefore, localized sound can be felt emphatically by the differences in the intensity of vibrations between the left and right sides.

Fig. 2.
figure 2

Vibration motor array device

Full size image
Fig. 3.
figure 3

Fixed wearable suit with vibration motor array device

Full size image
Fig. 4.
figure 4

Sound-analysis unit

Full size image

3.3 Sound-Analysis Unit

The sampling rate of sound processing is 44100 Hz. The analysis window size is 1024 samples. The FFT analysis of the sound at every 43 Hz derived the levels for following frequency bands: the low-frequency band is from 43 to 86 Hz, and the midrange frequency band is from 258 to 301 Hz. The FFT analysis is processed for each channel (left and right). The four values of the levels, the left/right and low/midrange frequency bands, are converted into 0 to 255 levels to control each vibration motor. The music sound is output by headphones connected to the PC at the same time as the vibration stimuli (see Fig. 4). The system is basically using a sound file, but it can also operate real-time processes of external input with the audio interface system. Accordingly, the system is expected to be applied to improvements of realistic sensation in both interactive sounds during game play and real-time audio in watching TV.

Fig. 5.
figure 5

Vibration-motor controls with delay for physical penetration

Full size image

3.4 Vibration-Motor Control Unit

There are two methods of vibration-motor controls. The first is the direct conversion of the frequency power to each motor without special delay except the interaural time difference. The second is the unique method of time-lag among the motors that was adopted.

Vibration-motor control without additional delay: First, the sound-analysis unit sends the power levels of each frequency band of both the left and the right channels and converts them into levels from 0 to 255. Second, the vibration-motor control unit operates the PWM controls for each vibration motor. Thus, the eight motors are controlled to correspond to the sound-analysis result.

Vibration-motor controls with designed delay for generating illusion of physical penetration by sound: The vibration-motor control unit produces an illusion of physical penetration of sound by time lags in starting the vibrations among the motors. The processing frame rate is set to 120 fps. The delay time is set from 0 to 1000 ms every 100 ms. That is, the second vibration motor starts its operation after 0–1000 ms when the first vibration motor begins its vibrations (Fig. 5). When there is physical penetration by a real object through the human body, it is assumed that the first shock will be perceived on the front surface and the second shock will be perceived on the back surface after the passage through the flesh. Although sound and music do not have any physical presence, the assumed stimuli with the differences in starting times of vibrations are expected to create a physical penetration of sound.

To make the user perceive vibrations by an eccentric rotor, it takes \(x-t\) ms (x is the time of the length of the sound, and t is the starting time among the vibration motors), which is a longer period than short-time pulse signals. Accordingly, there is an overlap between the timing of the front and back vibrations. Moreover, the overlapped timing is expected to create the feeling of passage through the body, as we aimed physical penetration.

4 Experiment

4.1 Evaluation of the Effect of Creating Realistic Sensations with Physical Vibrations in Synchronization with the Ongoing Music

Purpose of experiment: We tried to verify the effectiveness of vibration stimuli for creating realistic sensations of the ongoing sound of the music.

Hypothesis: The vibration stimuli covering the user’s upper body will elevate the realistic sense of the sound’s presence and power.

Participants: Twenty-five university students aged from 19 to 22 years old.

Conditions: We prepared 16 conditions according to two factors: (A) four different volumes of sound (s.none, low, middle, high) and (B) four different strengths of vibration (v.none, weak, medium, strong). The volume of sound for each condition was set as follows. The “s.none” means without any sound. The “low” is the lowest sound that the audio interface can output. The “middle” is 15 dB higher than low. The “high” is 5 dB higher than middle. The strengths of vibration for each condition were as follows. The “v.none” means without any vibration. The “medium” is the linear-mapped strength of the vibration synchronized with the volume levels of the sound converted from 0 to 255 levels. The “weak” is 0.5 times the power of medium. The “strong” is 1.5 times the power of medium. A branching table of the conditions is shown in Table 1. The 16 conditions with two factors were randomly arranged for counterbalance.

Table 1. Conditions in the experiment of synchronized vibrations with music
Full size table

Procedures and Instructions: The participants were instructed to listen to the music for 40 s (rock music played for 20 s and then changed into ballad music for 20 s) through the headphones and our proposed vibration-motor array device for each condition. The time difference between the motors was not generated in this experiment. The combined conditions of sound volume and vibration using each level in each factor were performed in the repeated measurement. When the 40 s of music finished, the participants were instructed to evaluate the statements that followed. The other 15 conditions were repeatedly conducted with both listening and evaluation.

Evaluation Items: The participants made an evaluation using a five-point rating scale of the relevance (5: very relevant, 4: somewhat relevant, 3: even, 2: somewhat irrelevant, 1: irrelevant) of the following statements.

Q1 :

You felt you were in a live performance.

Q2 :

You felt the sound source was close to you.

Q3 :

You felt the sound surround you.

Q4 :

You felt the whole space vibrating from the sound.

Q5 :

You felt the sound had spread.

Q6 :

You felt the sound had moved.

Q7 :

You felt the volume was high.

Q8 :

It was easy to feel changes in the strength of the sound

Q9 :

You didn’t feel anything unnatural about the vibration.

Q10 :

The vibration was consistent with the sound.

Q11 :

It was easy to listen to.

Q12 :

The sound was hollow.

Q13 :

You felt a change at exciting points in the music.

Q14 :

You felt power in the music.

Q15 :

You felt high.

Q16 :

Your body moved unconsciously.

Q17 :

You felt a sense of overdose on music.

Q18 :

You received a warm impression.

Q19 :

You felt comfortable with the sound.

Q20 :

You felt comfortable with the vibration.

Q21 :

You felt comfortable with the harmony of sound and vibration.

Fig. 6.
figure 6

MOS results in the experiment of synchronized physical vibrations with music

Full size image

Results: The results of the mean opinion scores (MOS) are shown in Fig. 6. The results of the analyses of variance (ANOVA) with repeated measurement among the conditions (p < .05) are shown in Table 2.

The subjective evaluations showed significance for both factors except questions 11, 12, and 16. No significant result by vibration factor was found without sound. There were no significant differences in the conditions without vibration in the result of the evaluation questions 9, 10, 12, 16, and 21. Almost all of the evaluation items, except those above, showed significant differences between the with-sound and without-sound conditions at every vibration level. Besides the conditions with or without sound/vibration, there were significant differences among different sound intensities. In addition, there were interactions and simple effects between conditions with and without vibration at each volume. There were also significant differences in high-medium and high-weak strengths of the vibration. However, there was no significant difference in the vibration strength between medium and weak when the volume was in the middle for Q11 and Q19.

4.2 Evaluation of the Time Difference Among Vibration Motors for the Illusion of Physical Penetration of Sound

Purpose of experiment: We tried to verify (1) whether the new experience of physical penetration of sound could be created by the emphatic time lags of the starting times between multiple vibration motors, such as forward to backward, and (2) whether the proposed time improves the realistic sensation and the sense of sound image localization.

Table 2. Two-factor ANOVAs in the experiment of synchronized vibrations
Full size table

Hypotheses: The emphatic time lags of starting times between forward and backward vibration motors (1) generate the feeling of physical penetration of the sounds, (2) improve the realistic sensation and (3) make clear the sense of sound image localization.

Participants: Twenty-five university students aged from 19 to 22 years old.

Conditions: We prepared 11 conditions of within-subject design according to a factor of time lag between forward and backward vibration motors: from 0 to 1000 ms every 100 ms. The conditions are shown in Table 3. The 11 conditions were randomly arranged for counterbalance.

Procedures and Instructions: The participants were instructed to listen to the sound of a bass drum for 30 s with headphones and the vibration-motor array device. The forward vibration motor was synchronized with the real-time sound, and the backward vibration motor was operated after the specified time lags for each condition. Accordingly, the backward vibration motors produced stimuli with one of the 11 different time lags. After each 30-second experience, the participants evaluated the impression for the statements of evaluation items that follow.

Evaluation Items: The participants made an evaluation using a five-point rating scale of the relevance (5: very relevant, 4: somewhat relevant, 3: even, 2: somewhat irrelevant, 1: irrelevant) of the following statements.

Q1 :

The backward vibration was later than the forward vibration.

Q2 :

The forward vibration was later than the sound.

Q3 :

The backward vibration was later than the sound.

Q4 :

You felt as though the sound passed through your body from forward to backward.

Q5 :

You felt the sound was just like the original sound of a bass drum.

Q6 :

You felt the bass drum was played in front of you.

Q7 :

The sound generated from the bass drum hit the front of your body.

Q8 :

The sound generated from the bass drum passed through your body.

Q9 :

You felt that the bass drum was close to you.

Q10 :

You felt a punch of the sound.

Q11 :

The vibration was natural in connection with the sound.

Q12 :

You felt the sound cover you.

Q13 :

The vibration was natural.

Q14 :

The sound passed through the flesh of your body.

Q15 :

The sound passed through the bones of your body.

Q16 :

You felt comfortable with the sound.

Q17 :

You felt comfortable with the vibration.

Q18 :

You felt comfortable with the harmony of the sound and the vibration.

Q19 :

You felt comfortable with the feeling of the sound passing through your body.

Results: The results of the MOSs and ANOVA with repeated measurements among the conditions (p<.05) are shown in Fig. 7 and Table 4. The evaluations except Q2 and Q7 showed significance. In particular, there were significant results in the 100–200 ms and 400–1000 ms lags. The most important results are Q13 and Q14, which showed the highest values in 200 ms lags as evaluations for hypothesis 1).

Table 3. Conditions in the experiment of physical penetration
Full size table
Fig. 7.
figure 7

MOS results in the experiment of physical penetration

Full size image

5 Discussion

5.1 The Effects of Vibration on the Realistic Sensation of Sound

From the results of two factorial analyses of variance, the sound with vibration provided a stronger feeling of immersion and realistic sensation than simple sound by itself. Therefore it is suggested that our proposed method of providing vibration from various directions can elevate the illusion of the physical presence of the sound. Moreover, we conjecture that the realistic sensation of the sound is improved and that the loudness of the sound is strongly perceived, especially when the volume of the sound is low.

The harmony between the vibration and the sound may stimulate comfortable feelings; however, if the vibration was too strong, this reduced the level of comfort. Moreover, there was no negative influence on the perception of the sound quality from the vibration. These results confirmed the hypothesis in Sect. 4.1. For further application for people with hearing difficulties, we should evaluate the musical perception using vibration without sound.

5.2 Illusion of the Physical Penetration of Sound

At first, from the results for Q2, there were no perceptional time lags from the musical sound stimuli and the vibration, while the vibration motors required a few milliseconds for perceivable stimuli from the start of the rotation of the motor.

Table 4. One-factor ANOVAs in the experiment of physical penetration
Full size table

Next, from the results of the factorial analyses of variance, the time lag around 100–200 ms from front to back could provide a feeling as though (1) the sound passed through the user’s body without any unnatural or incongruent feeling and (2) a bass drum were being played in front of the user.

Moreover, the participants could perceive the difference in the front and back vibrations even with only 100 ms of time lag. The significant difference in the feeling of physical penetration was confirmed in the statements regarding the penetration of both flesh and bone. Based on the results that the MOS for Q14 was higher than that of Q15, it is possible that the penetration of flesh is stronger than that of bone. This indicates that humans can feel an illusion of the physical penetration of sound.

From these results, the hypotheses in Sect. 4.2 were confirmed. We consider that the illusion of physical penetration of sound elicits both a realistic sensation and localization of the sound with an impression of physical presence. The advantage of multiple sound sources with each different direction of the physical penetration should be discussed in future work.

6 Conclusions

In this paper, we proposed a wearable system with a vibration-motor array in synchronization with sound to create an emphasized experience of music that included both a strong bass sound and the illusion of the physical presence of the sound source. The multiple vibration motors expressed not only strength and localization but also fictive penetration through the user’s body. The vibration motors were placed surrounding the user’s neck, chest, and back.

We evaluated the effects of (1) the combination of vibration with sound in the musical experience and (2) the differences in starting time between the front and back vibrations on creating the illusion of physical penetration. From the results, it is confirmed that (1) the surrounding vibration in synchronization with the sound of music produced both a realistic sensation and a punch of the musical experience and that (2) the appropriate time lags between the front and back vibration motors created an illusion of physical penetration of the sound; at the same time, sound localization and realistic sensation were also elevated.

In future work, we should consider the possibility of the proposed device for musical experience of people with hearing difficulties. It is also suggested that the proposed vibration-motor array could be combined with the visual presentation of sound through head-mounted displays.