1.1 Scope and Goals

Musical haptics is an emerging interdisciplinary field investigating touch and proprioception in music scenarios from the perspectives of haptic engineering, human–computer interaction (HCI), applied psychology, musical acoustics, aesthetics, and music performance.

The goals of musical haptics research may be summarized as: (i) to understand the role of haptic interaction in music experience and instrumental performance, and (ii) to create new musical devices yielding meaningful haptic feedback.

1.2 Haptic Cues in Music Practice and Fruition

Whenever an acoustic or electroacoustic musical instrument produces sound, that comes from its vibrating components (e.g., the reed and air column in a clarinet, or the strings and soundboard of a piano). While performing on such instruments, the haptic channel is involved in a complex action–perception loop: The player physically interacts with the instrument, on the one hand, to generate sound by injecting energy in the form of forces, velocities, and displacements (e.g., striking the keys of a keyboard, or bowing, plucking, and pressing the strings of a violin), and on the other hand receiving and perceiving the instrument’s physical response (e.g., the instrument’s body vibration, the kinematic of keys being depressed, the resistance and vibration of strings). One could therefore assume that the haptic channel supports performance control (e.g., timing, intonation) as well as expressivity (e.g., timbre, emotion). In particular, skilled performers are known to establish a very intimate, rich haptic exchange with their instruments, resulting in truly embodied interaction that is hard to find in other human–machine contexts. Through training-based learning of haptic cues and auditory–tactile interactions, musicians develop highly precise auditory–motor skills [7, 28]. They then form a base of highly demanding users who expect top quality interaction (i.e., extensive control, consistent response, and maximum efficiency) with their instruments–tools that extends beyond mere performance goals to emotional and aesthetical outcomes.

In addition to what described above, both the performers and the audience are reached by vibration conveyed through air and solid media such as the floor and the seats of a concert hall. Those vibratory cues may then contribute to the perception of music (e.g., its perceived quality) and of instrumental performance (e.g., in an ensemble, a player could be able to monitor others’ performances also through such cues).

Music fruition and performance therefore present a well-defined framework in which to study basic psychophysical, perceptual, and biomechanical aspects of touch and proprioception, all of which may inform the design of novel haptic musical devices. There is now a growing body of scientific studies of music performance and perception from which to inform research in musical haptics, including topics and methods from the fields of psychophysics [19], biomechanics [11], music education [29], psycholinguistics [32], and artificial intelligence [20].

1.3 Musical Devices and Haptic Feedback

While current digital musical instruments (DMIs) usually offer touch-mediated interaction, they fall short of providing a natural physical experience to the performer. With a few exceptions, they lack haptic cues other than those intrinsically provided by their (passive) mechanics, if any (e.g., the kinematics of a digital piano keyboard)—in other words, their behavior is the same whether they are turned on or off. Such missing link between sound production and active haptic feedback, summed to the fact that even sophisticated sound synthesis cannot (yet?) compete with the complexity and liveliness of acoustically generated sound, generally makes the experience of performing on DMIs less rewarding and rich than playing traditional instruments. Try asking a professional pianist, especially a classically trained one, to play a digital piano and watch out! However, one could argue that establishing a rich haptic exchange between musicians and their digital tools would enhance performance control, expressivity, and user experience, while the music listening experience would be improved by conveying audio-related vibratory cues to the listener. Indeed, a recently renewed interest in advancing haptic interaction design for everyday intelligent interfaces—shared across the HCI and engineering communities, as well as the consumer electronics industry—promotes the idea that haptics has the potential to greatly improve usability, engagement, learnability, and the overall experience of the user, moreover with minimal or no requirements for constant visual attention [15, 17]. For example, haptic feedback is already used to improve robotic control in surgical teleoperation [27] and to increase realism and immersion in virtual reality applications [30].

With regard to applications, haptic musical interfaces may provide feedback on the performance itself or on various musical processes (e.g., representing a score). In addition to enhancing performance control and expressivity, they have a high potential as tools for music tuition, for providing guidance in (intrinsically noisy) large ensembles and remote performance scenarios, and for facilitating access to music practice and fruition for persons affected by somatosensory, visual, and even hearing impairments [6, 13, 21]. A notable example is: The virtuoso and profoundly deaf percussionist Evelyn Glennie explained her use of vibrotactile cues in musical performance, to the point of recognizing the pitch, based on where the vibrations are felt on her body [10]. A further potential application of programmable haptic feedback in musical interfaces is to offer a way of prototyping the mechanical response of components found in traditional instruments (e.g., the kinematics and vibratory behavior of a piano keyboard), thus saving time and lowering production costs, as opposed to traditional hardware development.

Some efforts were made in recent years to define a systematic approach for the design of haptic DMIs and to assess their utility [3, 9, 23]. Some of the developed prototypes simulate the haptic behavior of existing acoustic or electroacoustic instruments, while others implement new paradigms not necessarily linked to traditional instruments. Early examples of haptic musical interfaces consist in piano-like keyboards with computer-driven mechanical feedback for simulating touch responses of various keyboard instruments (e.g., harpsichord, organ, piano) [4, 8]. More recently, a haptic system using magneto-rheological technology was developed that could reproduce the dynamic behavior of piano keyboards [16]. A vibrotactile feedback system for open-air music controllers, based on an actuated ring or a feet stimulator, was proposed in [31]. Haptic DMIs inspired by traditional instruments (violin, woodwinds, monochord, and slide whistle) are described in [2, 18, 22]. In [26], actuators were used on acoustic and electroacoustic instruments to feed mechanical energy back and induce or dampen resonances.

Only a few commercial examples of haptic musical devices are currently found. The Yamaha AvantGrandFootnote 1 series of digital pianos embed vibration transducers simulating the effect of vibrating strings and soundboard, and pedal depression. The system can be turned on or off, and vibration intensity adjusted. The Ultrasonic Audio SyntactFootnote 2 is a midair musical interface that performs hand-gesture analysis by means of a camera, and provides tactile feedback at the hand through an array of ultrasonic transducers. The Soundbrenner PulseFootnote 3 is a wearable vibrotactile metronome. The Loflet BassletFootnote 4 and SubpacFootnote 5 are wearable low-frequency vibration transducers (tactile subwoofers), respectively, in the form of a bracelet and a vest, whose goal is to enhance the music listening experience.

1.4 Challenges

Research in musical haptics faces several challenges, some of which are common to haptic engineering and HCI in general.

From a technology viewpoint, the use of sensors and actuators can be especially problematic because haptic musical interfaces should generally be compact and unobtrusive (to allow for seamless interaction), efficient in terms of power (so they can be compatible with current consumer electronics industrial processes), and offer high fidelity/accuracy (to enable sensing subtle gestures and rendering complex haptic cues). Musical haptics would then gain from further developments in sensing and actuator technology in those directions.

From the perspective of HCI and psychophysics, the details of how the haptic modality is actually involved and exploited while performing with traditional musical instruments or while listening to music are still largely unknown. More psychophysical evidence and behavioral evidence are needed to establish the biomechanics of touch and how haptic cues affect measurable performance parameters such as accuracy in timing, intonation, and dynamics, as well as to better understand the role of vibration in idiosyncratic perceptions of sound/instrument quality by performers and music/sound aesthetics by listeners.

What is more, haptic musical interfaces are interactive systems that require rigorous user experience evaluation to help define optimal configurations between perceptual effects and limitations on the one hand, and technological solutions on the other [5, 12, 33]. Despite the fact that several evaluation frameworks have been proposed [14, 24, 34], the evaluation of digital musical devices and related user experience currently suffers from a lack of commonly accepted goals, criteria, and methods [1, 25].

1.5 Outline

The first part of the book presents theoretical and empirical work in musical haptics with particular emphasis on biomechanical, psychophysical, and behavioral aspects of music performance and music perception. Chapter 2 redefines, with an original perspective, the biomechanics of the musician–instrument interaction as a tight dynamic coupling, rather than the mere interaction of two separate entities. Chapter 3 introduces basic concepts and functions related to the anatomy and physiology of the human somatosensory system with special focus on the perception of touch, pressure, vibration, and movement. Chapter 4 reports experiments investigating vibrotactile perception in finger-pressing tasks and while performing on the piano. Chapter 5 examines the role of vibrotactile cues on the perception of sound/instrument quality from the perspective of the musician, based on recent psycholinguistic and psychophysical evidence from violin and piano studies. Chapter 6 reports an experiment that uses quantitative and qualitative HCI evaluation methods to assess how various types of haptic feedback on a DMI affect aspects of functionality, usability, and user experience. Chapter 7 considers a music listening scenario for different musical genres and tests how body vibrations—generated from the original audio signal using a variety of approaches—influence the musical experience of the listener.

The second part of the volume presents design examples, applications, and evaluations of haptic musical interfaces. Chapter 8 describes an advanced hardware–software system for real-time rendering of physically modeled virtual instruments that can be played with force feedback, and its use as a creative artistic tool. Chapter 9 examines hardware and computing solutions for the development of haptic force-feedback DMIs through a case study of music compositions for the Laptop Orchestra of Louisiana. Chapter 10 proposes and evaluates the design of a taxonomy of vibrotactile cues and a stimulation system consisting in wearable garments for providing information similar to a score during music performance. Chapter 11 reports a series of experiments investigating the design and evaluation of vibrotactile stimulation for learning rhythm skills of varying complexity, with a special emphasis on multi-limb coordination. Chapter 12 evaluates the use of touchscreen interfaces augmented with audio-driven vibrotactile cues in music production, focusing on performance, user experience, and the cross-modal effect of audio loudness on tactile intensity. Chapter 13 illustrates common vibrotactile actuators technology and provides three examples of audio-haptic interfaces iteratively designed through validation procedures that tested their accuracy in measuring user gesture and in delivering vibrotactile cues.

A glossary at the end of the book provides descriptions (including related abbreviations) of concepts and tools that are frequently mentioned throughout the volume, offering a useful background for those less acquainted with haptic and music technology.