The Design of the Body:Suit:Score, a Full-Body Vibrotactile Musical Score

Abstract

We present the body:suit:score, a vibrotactile musical score that displays instructions and signals directly on musicians’ bodies using an array of vibrating motors. Initially inspired by the third author’s need for a new kind of score in his compositional practice, the body:suit:score facilitates the exploration of novel musical experiences and performer-audience relationships that are impossible or impractical without such a score, such as allowing the performers to freely move about the performance space amidst the audience. The body:suit:score has been designed through a close collaborative process between artists and technicians. A number of notable design elements and functionalities have emerged in response to this process, and in particular due to the strong requirements for usability and maintainability of the system.

1 Introduction

The impetus for the body:suit:score project arose from Bhagwati’s research-creation into new forms of score for novel performance situations, in particular the scoring for and coordination of musicians moving through real space. Previous projects had included situational scores for musicians moving in three dimensions within a large stairwell (“Racines Éphémères” (Vienna 2008)) and isolated musicians moving through a cityscape, hearing each other solely via a constantly reconfiguring wireless audio network (“Nexus” (Montréal 2010)). All these projects, however, had required the musicians to memorize elaborate rule systems or to react intuitively to musical cue. These limitations seriously curtailed the kind of complex, situative coordination that we have come to rely on and expect in a stationary music performance with a score and a conductor. The initial motivation was thus to create a workable interface for moving musicians that could embody both the role of the score and that of the conductor.

After some initial tests with portable head-up displays (e.g. Google Glasses) it soon became clear that, for reasons of safety alone, such a score must be non-visual. Two sensory modalities were subsequently pursued in parallel at matralab: the elaborate audio score [1] and the vibrotactile body:suit:score. For the body:suit:score, several composers (Julian Klein, Adam Basanta, Csenge Koloszvari) were invited to accompany the development process, and it was decided early on–for research purposes and according to aesthetic preferences–that each would pursue the creative possibilities of a different approach to the system: its use to convey abstract symbols, continuous parametric changes, and as a replacement for the sense of hearing.

The system described here represents three years of development building on the earlier prototype body:suit:score presented in [2]. Since the beginning of the project, the implementation of the system has gone through two major iterations, with the second and current version designed in response to specific challenges presented by the original prototype, the demands of composers working with the system, and the pieces they envisioned using it for. We will discuss the particular design constraints and requirements of a vibrotactile musical score that have emerged from our collaborative research process, and describe the hardware and software and capabilities of the system that have resulted from these requirements and the various approaches taken by composers when using the body:suit:score.

Fig. 1.

The original body:suit:score as presented in [2]

2 Early Work

The earliest prototypes of the body:suit:score were based on hardware and software adapted from the Ilinx project [3], a multi-sensory installation where participants experienced full-body vibrotactile sensations together with light and sound. Our initial prototypes came to define many of the essential capabilities of the body:suit:score, and the structure of their implementation is strongly reflected in the current iteration (see Fig. 2). In the original version of the system, vibrating motors were activated by motor control boards distributed across the body. Each Arduino-compatible control board activated six motors¹. The control boards were connected by Ethernet cables to a Beaglebone Black (BBB) single board computer (SBC). This central hub was responsible for receiving and parsing wireless Open Sound Control (OSC) based communication from a Max/MSP program dubbed the conductor program. Once parsed by the SBC, the messages were forwarded to the control boards in a more compact format using serial peripheral interface (SPI). The SPI stream along with power from a single central battery were carried over the Ethernet cables to each control board. The control boards and motors were sewn directly to the garments and electrically connected using silver low-resistance conductive thread with a polyester core (Fig. 1). The spacing of the motors on the suit was determined based on evaluations of the minimum distance at which two motors could be reliably discriminated: motors on the torso were spaced for maximum discrimination, whereas motors on the arms and legs were spaced closer together to allow sensations of continuous movement to be employed.

Fig. 2.

The conceptual structure of the body:suit:score system from a technological perspective. The conductor program sends messages to control boards that activate vibrating motors.

Several etudes were composed for these early body:suit:scores exploring a variety of approaches, documented in [2]. The work described in [4] was also carried out at this time, evaluating the ease with which different vibrotactile patterns can be recognized. A repertoire of these patterns were used in several pieces, beginning to form the basis of a tactile musical score language.

As well as these contributions, the early versions of the suit had important shortcomings and usability issues. The large SBC and central battery were identified as major sources of weight and bulk, and the use of Ethernet cables to connect these to the motor control boards meant that the suit could not be donned by musicians without assistance. The added bulk was also problematic during performance, as it impeded the wearers ability to move freely.

Difficulties were also encountered with the fit of the garments. A variety of fabrics were tested to try to improve the contact between the motors and the wearer’s skin and provide good transmission of vibration from the suit to the body. More stretchy materials such as high-spandex knit fabrics were able to provide better contact with the motors by conforming to the wearer’s body shape. In addition, early prototypes were tailored to the musicians who would wear them, providing an excellent fit. Unfortunately, these solutions resulted in other trade-offs. The conductive thread connecting the control boards to the motors had to be embroidered by hand when using these stretchy materials, introducing a significant cost in labour and reducing the repeatability of the manufacturing process. Tailoring the suits to individual musicians made each garment less versatile, providing a poorer fit when worn by other individuals.

As mentioned, the motors and control boards were initially sewn directly to the garments using conductive thread; while this provided an excellent and highly reliable mechanical and electrical connection, it also made the suits more difficult to maintain and wash. If electrical components failed for any reason, replacing them required sitches to be torn and re-sewn, causing significant wear on the garments in the process. While these issues were not immediately problematic, as repairs by hand and careful surface cleaning were still possible, our concern for the longevity of the garments motivated us to consider ways to improve the maintainability of the suits.

It was eventually found that the wide motor spacing on the torso was needlessly limiting. In part due to limitations on the number of motors that could be powered by the central battery, it was decided to widely space the motors on the back of the torso with the intention that these would mainly be used to convey symbolic information. Continuous sensations, however, were difficult to achieve, so composers wanting to utilize this kind of sensation were forced to use mainly the motors on the legs. If a large amount of information needed to be conveyed, this created a major bandwidth bottleneck.

The original communication protocol for the body:suit:score was borrowed from the Ilinx project [3]. This protocol was based around a linear amplitude envelope consisting of a rising stage, a sustaining stage, and a falling stage. The duration in milliseconds of each envelope stage, as well as the amplitude² of the sustain stage, could all be individually specified (see Fig. 6). The envelope would always start and finish with zero amplitude.

In symbol-based approaches to the system, the concept of tactons, “structured, abstract messages that can be used to communicate messages non-visually” [6], have provided an especially important framework for working with the suit. In our research we have developed a repertoire of tactons for the body:suit:score that can be used as symbols in a growing tactile language for conveying musical instructions. Using the original control protocol for the suit, a tacton could be displayed by sending several messages sequentially in order to engage and disengage a sequence of actuators in a certain pattern. Unfortunately, because a single tacton was composed of multiple separate messages sent at different times in distinct UDP packets, this introduced multiple opportunities for a tacton or other complex pattern being rendered on the suit to become corrupted by a lost UDP packet. This had the potential to seriously undermine performers’ confidence in the system, as it introduced the possibility that at any time they might have to guess at the meaning of half of a tacton conveying a crucial musical instruction. We recognized this as a serious issue that had to be eliminated.

In addition to the amplitude envelope, the control messages also specified which motor to target. This was accomplished using an integer in the OSC address that referred to one of the control boards, and a real number between zero and one that referred to which of the six motors controlled by that board to target. The use of a real number was chosen to allow for the possibility that control boards in the future might include more than six motors. The numbering of the control boards and their motors was hard coded into the firmware of each control board so that the motors were ordered in a consistent manner from left-to-right and top-to-bottom. Unfortunately, when developing new iterations on the design, or if a control board had to be replaced, these hard coded variables were sometimes not properly updated, meaning that the numbering of the motors sometimes became messy and difficult to work with.

Finally, the aforementioned reliability issues also revealed a major challenge when working with the suits in rehearsal: in case there seemed to be a problem, it was very difficult to quickly troubleshoot the suits since the person operating the conductor program would have to communicate constantly with the musician wearing the suit to see if the messages being sent to the suit were having the expected effect. It was often difficult to explain to the musician exactly what to expect, as well as for the musician to explain exactly what they were feeling. There was clearly a need for more direct feedback from the suit that could be understood by people other than the wearer, such as a software status monitoring system or visual feedback. On the other hand, if visual feedback were added to the suit it was agreed that it should be possible to disable it, in case a composer would prefer a more neutral look during a performance.

Based on the successes and shortcomings of the original body:suit:score system, a set of requirements were developed for a new version:

• The suit must be lightweight and easy to put on and take off without expertise or assistance.

• Every suit must be adjustable to provide an excellent fit, as though tailored to the wearer, without requiring every wearer to have a custom made garment specific to their body shape.

• The suit must provide comfortable but firm contact between the wearer and the vibrating motors.

• The suit should allow total freedom of movement.

• All non-washable electronic components should be removable to facilitate maintenance and care of the garments.

• The spacing of the motors should permit sensations of continuous movement to be produced anywhere on the body.

• The communication protocol should allow complex tactons to be encoded in as few UDP packets as possible, to avoid their corruption by dropped packets.

• Every effort should be made to increase the reliability of the wireless communications.

• The communication protocol should be easy to compose and read, and the numbering of the motors should be adjustable on the fly in case a control board must be replaced.

• Direct feedback should be available from the suits to other people than the wearer to facilitate troubleshooting in rehearsal.

• Visual feedback should have the possibility of being disabled to allow composers some control over the appearance of the body:suit:score in performance.

Fig. 3.

The current version of the body:suit:score with the LEDs removed

3 Current Hardware

In response to the requirements that arose while working with the original version of the body:suit:score, the design of the system was revised. The most significant change was to eliminate the single board computer and central battery, opting instead for the motor control boards to have individual batteries and WiFi communication capabilities (Fig. 3). This decision resulted in a number of immediate design improvements: Ethernet cables were no longer necessary to connect each control board to a central hub, thus significantly reducing the bulkiness of the system, improving the freedom of movement of the wearer, and making it possible for musicians to put on and take off the suit without assistance; use of distributed power allowed additional control boards to be added to the system easily as long as there was a space to put them, whereas the central battery had previously imposed an upper limit on the number of control boards and motors that could be used on one suit; by distributing wireless communications directly to the control boards, several links in the chain of communication were removed, resulting in more reliable transmission of data all the way to the actuators, as well as somewhat simplified troubleshooting; and finally, the use of a more powerful 32-bit microcontroller (ESP8266) allowed for a less compact but much more readable communications protocol to be used all the way from the conductor program to the control boards, without any need for a more compact intermediate format.

To provide a balance between fit, motor contact, and manufacturability, the textile aspect of the garments was designed to combine the use of stretchy and non-stretch fabrics. The parts of the garments with motors were designed to use a medium weight linen, allowing the conductive thread to be layed automatically with a CNC embroidery machine. For other parts of the garments, such as the sides of the vests, stretchy high-spandex knit fabric was used to allow the garment to fit snugly. On the vests an adjustable fit was also achieved by using lace or ribbon on the sides of the vest to cinch down any slack in the fit of the garment (similar to the way a shoelace works). Using these enhancements, the current design of the garments allows them to fit a wide variety of body shapes while always providing very good contact with the motors.

To improve the maintainability of the garments, the motors and control boards were made to be removable from the suit. This was achieved using a variety of connectors (Fig. 4): the motors are now attached to the suit using custom-made conductive hook-and-loop tape connectors with two contacts separated by an insulating strip; the control boards are connected using non-coated nickel snap-on buttons. In both cases, the electronics are sewn with conductive thread onto an intermediary fabric patch. Matching connectors are sewn onto the patches and the suit, allowing any of the electronic components to be attached or removed at will. This allows the garments to be machine washed, as well as making replacement of faulty or damaged electronic components relatively trivial (as long as a replacement patch is available).

Using the same hook-and-loop connectors as the motors, LEDs were added to the suit to provide visual feedback coupled with the vibration intensity of the motors. The LEDs are simply placed in a parallel circuit with the PWM signal driving the motors to achieve this coupling. This has made troubleshooting the suits significantly easier, since it makes it easy to see when a motor is behaving as expected. The LEDs have also been used in several performances, and audiences have reported that this improved their understanding of how the suit works. Should a composer prefer a more neutral look, however, the LEDs can simply be removed from the suits.

Fig. 4.

(From left to right) Control boards are attached with metallic snap-on buttons. The fit of the vests can be adjusted by cinching down slack in the fit. Motors and LEDs are attached with conductive hook-and-loop tape. The small circular patches pictured here are LEDs, which coincide in position with the ERM motors that are on the opposite side of the garment (close to the wearer’s body).

The number of control boards and motors on each suit was nearly doubled to provide for increased flexibility when designing vibrotactile patterns (Fig. 5). On the torso, more control boards were added, doubling the density of motors on the back and adding motors to the front of the torso with the same increased density³. Similarly, motors were added to the back of the legs where previously they had only covered the front of the legs. These increases in coverage and density were strongly motivated by the requirements for the piece Tactile Topologies envisioned by artist and composer Csenge Kolozsvari, particularly the requirement for continuous patterns of movement to be conveyed across the whole body including the torso. However, the changes to motor placement were also found to be advantageous for other approaches and uses of the system, such as allowing a wider variety of tactons to be developed for the torso.

Fig. 5.

The current motor layout, covering a greater area of the body and with a higher density of motors on the back of the torso than the original version.

The communications protocol was redesigned completely (see Fig. 6) to make it easier for programmers to work with and reduce the number of UDP packets that needed to be sent to achieve a complex pattern of vibration, as well as to improve the modularity of the control boards. Because each control board now had a separate and unique IP address (dynamically assigned by the DHCP server on the local area network), this was co-opted as a convenient identifier for each board. The indexing of the motors controlled by each board was designed to be dynamically assigned so that control boards would be effectively interchangeable, having no hard-coded constants related to their position on the suit. Finally, a simple mapping was implemented in the software library for working with the suit allowing the control boards to be identified by their location on the suit as an alternative to their IP address. This introduced the need for a simple setup procedure any time a control board was moved from one garment to another in order to set its current location; during this procedure, the motors were automatically re-numbered based on the location on the suit identified by the operator performing the setup. The combination of location-based addresses for the control boards and consistent numbering of the motors allowed a programmer working with the system to easily identify any individual motor from memory and greatly improved the readability of the communication protocol.

Fig. 6.

The original OSC message format for the body:suit:score, alongside the current version, with examples of the messages used to create the same amplitude envelope. In the original version, two messages must be sent in separate UDP packets with a time delay in between in order to generate the desired amplitude envelope, whereas in the current version a single message is able to encode the full envelope. Amplitude values are normalized from zero to one, and correspond with the duty cycle of the PWM signal that activates the motors. Time values are in milliseconds.

The linear envelope conceptual model of the original protocol was expanded to a more flexible model based on the line$\sim$object from Max/MSP and Pure Data. In this conceptual model, rather than a fixed number of rising and falling stages, a single pair of numbers represent one stage of the envelope, specifying it’s target amplitude and the duration of the linear ramp toward that target (see Fig. 6). In our implementation up to ten envelope stages may be encoded in a single OSC message by simply sending ten target-time pairs of numbers. An optional preceding number can be used to specify where the envelope should start from; otherwise, the envelope will start from whatever the current amplitude value is when the message is received. The minimum message specifies at least one linear ramp in the form of a single target-time pair, or a single target value can be sent to request that the amplitude is immediately set to a new value. These minimum messages are especially useful when constantly updating the amplitude of a motor, such as when displaying a continuously updating signal; using the old protocol, it was necessary to send messages with very short onset, and long overlapping sustain times to achieve this effect.

This new protocol also mitigates the need for multiple packets to be sent in order to display a single tacton. For tactons that can be achieved with only the six motors attached to a single control board, use of OSC bundles to encapsulate messages targeting different motors into one UDP packet greatly reduces the overall number of packets per tacton. In most cases, even very complex tactons can be transmitted with only one packet per control board involved in the tacton. These improvements to the control message protocol along with changes in the WiFi receiving hardware and reduction in the length of the chain of communication were found to completely eliminate lost or corrupted tactons under typical network conditions, while still allowing UDP to be used as the transport protocol to maintain low latency.

Taken together, these changes have greatly improved the body:suit:score, facilitating all approaches to its use. The improvements in the modularity and adjustability of the garments help to make them more versatile, easier to put on, more comfortable to wear and move in, and much easier to maintain. The addition of visual feedback makes the system easier to troubleshoot and facilitates audience members’ understanding of its functionality. The increased number of motors make a wider variety of approaches to the suit possible. Changes to the communications protocol make the suit easier to program, and improve the reliability with which complex patterns can be sent to the suit. Although the basic structure of the system is very similar to the original version, and the essential capabilities were not drastically changed, the modifications to the design have made the system more maintainable, more reliable, and easier to use for musicians, programmers, and composers alike.

4 Tactile Metronome

In addition to the enhancements based on requirements discovered when using the original version of the body:suit:score, other technological developments have been motivated by the demands of composers working with the system. Early in the development of the second and current version of the body:suit:score, the suits were envisioned as individualized metronomes to be worn by several musicians: The system thus could improve on existing experiments with both individualized click tracks, (Bhagwati, “Iterations” (Berlin 2014)) which disrupt the musicians’ listening, and visual conductors (either several human conductors, blinking lights, or video animations,) which musicians often find confusing and which impose severe spatial constraints on musician and audience placement and movement. In one movement of Bhagwati’s large-scale work Niemandslandhymnen (NLH, Montréal 2017), in which the musicians move within an equally mobile audience, each of the musicians improvises not only in a different tempo, but the constantly changing tempi perform a temporal choreography of pulsations from which a musical architecture emerges that would be impossible to conduct or to feel. The original goal of simply replacing the human conductor thus had been surpassed.

Fig. 7.

A diagram of the tempo of each metronome over time as was employed in Neimandslandhymnen.

A clock synchronization scheme based on network time protocol [7] was implemented to achieve this effect, and a novel algorithm extending the work of [8] was designed that allows the body:suit:score to act as a highly accurate metronome, impervious to network delay, jitter, and dropped packets. The tempi of each body:suit:score acting as a metronome can be synchronized across garments while also allowing arbitrary phase and frequency modulations. Later, the metronome functionality was also found to be useful for composing cyclic or repeating tactons.

The most obvious way to deliver a metronomic pulse across multiple suits would be to simply send a message at the required rate to produce periodic pulses at the appropriate time; in this implementation, the conductor program would keep track of the time elapsed, and emit control messages to produce a pulse every so often according to the required tempo. However, this implementation can be problematic due to the latency and jitter inherent in wireless networked communications: although the pulse message for each suit might be emitted at roughly the same time, there is no way to guarantee that these messages will be received at the same time by each suit. As a consequence, the pulses felt on each suit would not be guaranteed to occur at the same time, thus failing to synchronize the musicians wearing the metronomes. Indeed, even with a single suit the rate of the metronome would be subject to network jitter, and should multiple control boards be used on the same suit, even these might not pulse in sync. All of this is not even considering the possibility that individual pulses could be entirely lost should a UDP packet be dropped. In practice, packets might not be dropped, and jitter might be relatively small, so this implementation is reasonable when only a single metronome is needed. However, when multiple metronomes are required, differences in the latency between the conductor program and the different devices becomes a serious issue.

In order to guarantee that the metronomic pulses felt on each suit would coincide as exactly as possible, it was necessary to use a different approach. We opted to allow the control boards to keep track of time instead of the conductor program, so that each control board would autonomously emit the metronome pulse at the correct time instead of waiting to receive a message from the conductor program. In order for this to work, each control board would have to agree on the present time. Network time protocol (NTP) is a ubiquitous time synchronization protocol widely employed across the world wide web. NTP includes algorithms for how time information should be encoded, how it should be communicated, how the best time reference can be selected, and how to gradually re-synchronize a clock without abruptly changing the current time or allowing time to flow backwards. This latter aspect of NTP, known as the clock discipline algorithm, was implemented based on [9], with the other parts of the protocol being radically simplified, in order to give each control board a shared notion of time. A custom implementation was chosen mainly out of to convenience; to the authors’ knowledge, no NTP implementation was available for the MCU used in the control boards, and it was simpler to implement a subset of NTP rather than to fully comply with the protocol.

In our implementation, the conductor program is considered to be a stable and accurate source of the current time. The motor control modules periodically measure their offset and delay from the conductor according to the NTP method. Each control module individually disciplines its clock in order to gradually reduce the offset and improve the frequency of its clock so that it remains better in sync with the conductor. The UDP packet format defined by NTP was omitted in favor of a simplified OSC-based format, taking advantage of the existing communications implementation. NTP’s extensive algorithms for selecting the best clock source were also omitted, since the conductor can be considered to be the only true source of time. Finally, rather than the 64-bit fixed point time format specified by NTP, a slightly modified format was chosen instead. We use 32- rather than 64-bit integers, since the microcontroller used on the control boards is able to process 32-bit integers with a single instruction, and the least significant bit was chosen to corresponds to one microsecond, since this is the smallest time increment that the motor control boards are able to measure. Using the clock synchronization algorithm, every motor control board is able to keep track of time and stay synchronized within a few milliseconds given typical network conditions.

Given this shared notion of time, the control boards can generate a metronomic pulse using the Global Metronome algorithm described by [8]. Using this algorithm, each control board is given an interval T between pulses (or equivalently, a frequency $f = 1 / T$). The current phase $\phi (t)$ of the metronome is defined according to [8, eq. 4]

$$\begin{aligned} \phi (t) = \left[ 2\pi \left( \frac{t}{T} - \left\lfloor \frac{t}{T}\right\rfloor \right) + \theta \right] \mod 2\pi \end{aligned}$$

(1)

Where $\theta$ is an optional local phase offset. The amplitude signal for the metronome $a(\phi )$ can then be determined as

$$\begin{aligned} a(\phi ) = 1 \text { if } \phi < d, \text { else } 0 \end{aligned}$$

(2)

Where d is a duty cycle parameter between 0 and $2\pi$ that determines how brief the metronome pulse is.

The Global Metronome algorithm guarantees that all metronome pulses across the different suits will coincide, because the phase of the metronome is determined by the shared time. However, for NLH it was also necessary to allow the metronomes to be deliberated brought in and out of sync by gradually modulating the frequency of each metronome to a different tempo, and then to a new shared tempo (see Fig. 7). This presented a challenge, because the frequency parameter of the global metronome cannot be smoothly modulated; any change to this parameter introduces significant immediate phase modulations that disrupt the consistent beat of the metronome. Although the local phase offset could be continuously modulated to achieve a change in frequency, we chose instead to introduce a separate mode for the metronome based on a phasor where $\phi (t)$ is derived from $\phi (t-1)$:

$$\begin{aligned} \phi (t) = (\phi (t-1) + 2\pi f * T_s)\mod 2\pi \end{aligned}$$

(3)

Where $T_s$ is the sampling period. Using this definition of phase, the frequency of a metronome could be smoothly modulated up or down. So to achieve the desired effect, the metronome on each suit would initially be set to the same tempo using the synchronized algorithm. Then, when it was desired to modulate the frequency, the metronome was switched to the modulation-tolerant algorithm. Later, when each metronome was modulated to a new shared tempo, the algorithm was once again switched to the synchronized version.

When switching from the synchronized metronome to the modulated metronome, it was not necessary to change the value of $\phi$. The modulated metronome would simply pick up where the previous algorithm left off. Switching the other way around required the local phase offset $\theta$ to be set so that there would not be a jump in $\phi$. This left one lingering issue, since $\theta$ would usually not be set to the same value for each metronome when the algorithm was switched to the synchronized version after having been modulated to different tempi. This was resolved by allowing $\theta$ to be modulated when using the synchronized algorithm. In this way, the metronomes could be brought back in sync by switching to the synchronized algorithm, and then gradually modulating each metronome’s local phase offset toward zero. Both possible modulations (frequency while using the modulated metronome and phase while using the synchronized metronome) were implemented using the same linear envelope function as the amplitude envelopes, allowing complex frequency or phase modulation patterns to be specified in advance and sent in a single message from the conductor computer.

In later pieces, the metronome functionality was able to be repurposed for other effects. By setting the phase offset appropriately, repeating sweeping or circling sensations could be initiated very concisely compared to using amplitude envelopes to achieve the same effect. This was used to create rising and falling tactons for Bhagwati’s piece Silence Not Absence. The periodic pulses could also be used for amplitude modulation, granting a limited ability to vary the frequency content of the vibrotactile signal generated by the ERM motors⁴ used in the system. This allows ‘rough’ feeling vibratory textures to be rendered on the suit [10]. More in line with its use as a metronome, in Tactile Topologies (described in more detail below) the periodic pulsing was used as an analog to the periodic beating heard between two instruments playing slightly out of tune. If the musician wearing a suit was playing slightly higher than his or her partner, the motors on the back of his or her right leg would be pulsed at a rate equal to the beating frequency between his or her pitch and his or her partner’s pitch. If the musician was playing slightly lower than his or her partner, the beating frequency would instead by displayed on his or her left leg. This mapping permitted the musicians to play precisely in tune with each other, even while unable to hear one another.

The clock synchronization functionality has yet to be utilized other than for the metronome, but at least one other use case has been identified. Specifically, in the future we will implement a forward synchronization policy using the synchronized clocks. By allowing the execution of a message to be deferred to a specific time in the future, concerns about network jitter or latency could be completely eliminated. Combined with using TCP/IP to send messages to the control boards rather than UDP, this would give a mechanism for the transmission of important messages to be guaranteed as long as they can be sent well enough in advance of the time when they need to be acted upon. This could benefit a wide range of applications, especially for symbolic approaches to the system.

5 Sound to Vibrotactile Translation

Another piece that lead to major technical developments is entitled Tactile Topologies. This piece, composed by interdisciplinary artist and composer Csenge Kolozsvari, explores the use of the body:suit:score as a sensory replacement. It considers how the sense of touch might facilitate new kinds of musical interrelationships by inviting two musicians to perform a duet in which their only means of communication is through the body:suit:score. The musicians each wear industrial-grade ear plugs and ear muffs and play in separate rooms. The body:suit:score translates each performer’s sound into choreographies of vibration across the entire suit of the other performer.

In early experiments for Tactile Topologies we explored a variety of methods to attempt to convey as much sonic information through the body:suit:score as possible. We considered a large number of audio features [11], and mapping strategies ranging from machine learning to swarm algorithms. Ultimately, we chose to focus on representing the pitch and loudness of the sound alone. In traditional music notation, these features are given the most detailed attention, and musicians are expected to fill in the finer details of musical expression and interpretation. By focusing on pitch and amplitude, we were able to draw on this pre-existing skill.

The loudness feature was extracted using the pipo$\sim$Max/MSP extension by IRCAM [12], then scaled from zero to one based on the dynamic range observed during sound check before a performance. The mapping to the body:suit:score was designed to overcome the limited dynamic range of the ERM motors used on the suit by recruiting a large number of motors to increase the maximum possible intensity of vibration. 20 motors on the torso were partitioned into five groups, starting at the small of the back and radiating outward and upward to eventually wrap around the whole torso (see Fig. 8). The amplitude for each group $a_g$ was then determined based on the loudness feature l using an ad-hoc exponential scaling that was found to give a good correspondence from the loudness feature to the sensation of vibration intensity:

$$\begin{aligned} a_g(l) = \frac{4 * 2^l}{g} \text { subject to } 0<= a_g(l) <= 1 \end{aligned}$$

(4)

This has the effect that each group turns on one after the other as the loudness feature varies from zero to one, so that at maximum loudness nearly every motor on the torso is activated. The wrap around grouping of the motors was chosen because it was found by the musicians to provide a satisfying and intuitive correspondence with loudness. By gradually increasing the overall area and number of actuators activated, we were able to convey a greater sense of intensity than if fewer motors had been used.

Fig. 8.

The loudness-to-vibration mapping used in Tactile Topologies. As the loudness parameter increases, a greater and greater area of the torso is stimulated. This provides a wide dynamic range of intensity. The arrows indicate the direction in which the vibration seems to spread as the loudness of a sound increases. The numbers indicate the order in which the motors are activated as the loudness increases.

The pitch feature was estimated using a custom fundamental frequency estimator implemented in Max/MSP; a custom implementation was chosen rather than one of the available pitch estimators because we were able to achieve better accuracy by exploiting our prior knowledge about the range of pitches in the piece, and the spectral qualities of the instruments being performed (both musicians played french horn).

It initially seemed intuitive to try mapping pitch as a continuous line from one position on the body (e.g. the ankles) to another (e.g. the wrists), with each discrete pitch assigned to one motor. This approach was ultimately abandoned. The musicians were unable to reliably distinguish between adjacent motors (as was the intention in the design of the suit), and therefore the melodic patterns played in the piece did not produce recognizable patterns on the suit.

A more successful approach was found by separating the pitch mapping into two parts: pitch class, and octave. Two pitches an octave apart are perceived as having a close similarity due to the overlap of harmonic spectra that occurs between two periodic sounds an octave apart; sound perception can therefore be considered to have a spiral-like topology, in which pitches that are an octave apart fall into an equivalence class called the pitch class [13]. In eurological music it is most common to employ twelve pitch classes that make up the familiar chromatic scale. We were able to achieve a more effective pitch mapping for the body:suit:score by drawing on these phenomena. Each pitch class was mapped to a single motor, with the twelve pitch classes forming a circle up one arm and down the other. The octave was then separately mapped to the height of a vibration on the front of the legs. See Fig. 9 for more detail. Although it required slightly more analytic interpretation by the performers than e.g. the amplitude mapping, this pitch mapping was found to permit the musicians to roughly follow what the other was doing. It was also found to produce recognizable patterns across the arms when melodies were played.

Fig. 9.

The pitch-to-vibration mapping used in Tactile Topologies. Pitch class is mapped in a circle around the arms, while octave is represented on the front of the legs. The arrows indicate the direction of motion felt when an ascending melodic pattern is played.

In the end, Tactile Topologies also relied on the musician’s pre-existing knowledge of the piece for its success. Because the musicians knew the overall plan and structure of the piece in advance, they were able to use the signals from the body:suit:score as a close-enough approximation of the other performer’s actions and fill in the details using their expectations and musical interpretation skills. Future experiments will consider the viability of the pitch and amplitude mappings devised for this piece as standalone music representations in absence of prior knowledge. This may include using the mappings to instruct musicians exactly what pitch and loudness to play next, or as an inspirational prompt for improvisation. These applications will allow the pitch and amplitude mappings to be employed as an additional score modality alongside tempo and symbol-based approaches employed in other pieces.

6 Future Work

The body:suit:score allows information and instructions to be transmitted to musicians in real-time. From the beginning of the project, we have also intended to incorporate motion and position sensing into the system in order to allow musicians actions, location, and spatial interrelationship to influence the direction and outcome of the musical system in a closed interactive loop. This remains as future work.

Although our design has improved significantly since its earliest prototypes, we continue to consider further refinements. Distributing the responsibility for power and communications across the suit lead to some of the most striking improvements in the current version of the body:suit:score. In the Vibropixels project [14], which has been developed separately at the same time as the body:suit:score from the same roots in the Ilinx project [3], Hattwick et al. devised an even more decentralized approach to the design of a vibrotactile display that entirely dispenses with motor control boards, instead opting for every individual motor to be equipped with its own battery and wireless-communication-enabled microcontroller. This facilitates even greater freedom over where to position the motors and how many to incorporate in a garment, and also provides some improvements to maintainability and manufacturability. One of the greatest lingering issues with the hardware of the body:suit:score lies in the potential failure of the modular connectors to transmit power from the control boards to the motors. A fully decentralized approach such as that taken with the Vibropixels also avoids this issue. Future iterations of the body:suit:score may adopt a similar approach.

7 Conclusion

By developing the body:suit:score to its current state, we’ve proposed one solution to the problem of making a vibrotactile musical score. We have striven to develop a robust implementation that is easily maintained and manufactured, and that is easy to use for programmers, musicians, and composers alike. Our system is capable of generating a wide range of sensations for its wearers, and we have explored a small portion of the many possibilities offered by the system. Through its use in numerous performances across North America and Europe, the reliability of the body:suit:score has been repeatedly validated.