2003: Sonigraphical Instruments: From FMOL to the reacTable*

Abstract

This paper first introduces two previous software-based music instruments designed by the author, and analyses the crucial importance of the visual feedback introduced by their interfaces. A quick taxonomy and analysis of the visual components in current trends of interactive music software is then proposed, before introducing the reacTable*, a new project that is currently under development. The reacTable* is a collaborative music instrument, aimed both at novices and advanced musicians, which employs computer vision and tangible interfaces technologies, and pushes further the visual feedback interface ideas and techniques aforementioned.

Appendices

Author Commentary: The Genesis of Reactable

Sergi Jordà

2003, the year of the 3rd NIME conference, was also quite an impasse year for me. After graduating in physics in the mid-80s, I spent most of the 90s working as a freelance interactive and media artist/designer/programmer, developing interactive installations, performances and music systems (two of them, Epizoo and FMOL, had already been introduced in my 2001 NIME paper). With the advent of the millennium I had decided to re-enter academia and pursue a PhD in Computer Music at the Music Technology Group (MTG) founded by Xavier Serra in Barcelona. The MTG was still a very young research group in which the lack of professors or post-docs (Serra was still the only doctor!) was happily compensated with the enthusiasm of everyone, and the most mature PhD candidates shouldered the role of team and project leaders. So here I was, trying to figure out my own PhD thesis, while simultaneously coordinating a team of PhD students that included some of the most talented computer music developers I have ever known. It is within these circumstances that this paper provides an analytical revision of some of my past works and discusses prospective research directions.

Through the study of the aforementioned examples (Epizoo and FMOL), the first half of the paper addresses the potential of what I then called “sonigraphical instruments,” a term I coined for referring to instruments in which their GUI acts both as an input for sound control, and as an output that provides a summarized and quickly graspable display of all the sound and music activity. This sonigraphical feature, I argued, is even more pertinent in multithreaded instruments that allow several simultaneous musical processes. These postulates would be furthered in my Phd dissertation published two years later (Jordà 2005).

The second half of the paper introduces a multithreaded and sonigraphical DMI, Reactable (known as reacTable* at that time), discussing its main features and the rationale behind its conception. Considering that this part describes a system that had not yet been tested, not implemented, and in fact, not even fully technically solved, that is, that what it outlines is little more than a personal vision, I now wonder how unlikely it would be that a similar paper would have been accepted today! The presumable different perspectives resultant from this dozen year gap, reflect on one side the maturity currently attained by the NIME field, which was in its infancy in 2003. They could also warn us however about the risks of trying to address too scientifically topics such as the design and evaluation of DMIs, which resist reduction and systematization. For example, although we have published several papers that systemically study the potential benefits of the Reactable in very specific contexts, such as autism (Xambó et al. 2013) or collaboration and peer-learning in public spaces, we have still failed to produce the “ultimate Reactable-music-instrument evaluation” paper.

A few months after the publication of this paper, we solved the most crucial pending technical issue and published reacTIVision, an open-source, cross-platform computer vision framework for the tracking of fiducial markers and combined multi-touch finger tracking (Bencina et al. 2005). Two years later, Reactable was premiered at the ICMC 2005 in Barcelona. It had not been the first musical tabletop; at least the Audiopad (Patten et al. 2002) and the Music Table anticipated it, but it became the more popular one and, arguably, the most popular DMI to emerge from an academic context. In November 2006 we guilelessly published a video in YouTube, which rapidly and unexpectedly reached millions of views. Three months later, Björk contacted us and in April 2007 she started touring with a Reactable. Around this same period, the Reactable team also started touring extensively, giving more than 300 concerts, installations and presentations, in 25 countries during the next 3 years. This helped enormously with fine-tuning and improving the system. In 2008 I published what I consider the “ultimate” Reactable paper (Jordà 2008). In 2009, the company Reactable Systems⁴ was founded, launching the Reactable Live and the Reactable Experience, and later in 2010, also Reactable Mobile for iOS and Android.

With Reactable Systems monopolising all the development of the musical instrument, my research shifted towards other areas of tabletop and tangible interaction. It has just recently shifted back to NIME and digital music creation in the shape of “expert agents for electronic music performance,” combining research and techniques from musicology, music information research (MIR), machine learning and HCI.⁵ My life has not changed so much, but this 2003 paper set the beginning of a nice story that has not yet ended, and I am definitely happy and proud with what we achieved.

Expert Commentary: Pursuing a Sonigraphical Ideal at the Dawn of the NIME Epoch

Charles Martin

A common criticism levelled at the NIME community is that we jump onto the latest available technology, develop one interface, play one concert, send out a paper, and move on. To this criticism, Jordà’s work on sonigraphical instruments sits as a compelling counter-example. Jordà’s paper seems to have appeared at a turning point in his work developing computer-based instruments for collaborative musical interaction, and to be representative of the start of the NIME epoch.

In the first half of his article, Jordà describes Epizoo, a game-like multimedia interface that sits very much in the early-1990s, and FMOL, a collaborative synthesiser running in a late-90s era mouse-driven Windows interface. In the second half, Jordà outlines the in-progress design for Reactable, a table-top tangible interface that went on to be unveiled through YouTube videos⁶ in 2006 to mainstream acclaim in the tech and music press, and was famously used in Björk’s Volta (Andrews 2007) and Biophilia concert tours starting in 2007. The Reactable could be said to be emblematic of a surge of interest in NIME-research around this time—at least that’s when I started to get interested!

So what lessons does this paper hold for the NIME researcher of today? First of all, this paper shows how perspective and experience acquired through long-term effort in developing musical interfaces is often required for the most interesting designs. The paper itself describes around 10 years of work where Jordà and his collaborators produced interfaces and invited beginners as well as experienced performers to use them. While Jordà is satisfied with the utility of FMOL, a successful design by all accounts, years of experience have suggested that there is potential to develop an instrument that is even more intuitive, collaborative, and with fewer compositional assumptions. More than 10 years later, we now know that Reactable achieved these goals, with success as an installation work, as an instrument on the professional concert stage, and with ongoing utility in HCI research (Xambó et al. 2013). It strikes me as unsurprising that this success should follow a long period of experimentation with different designs, and many different kinds of users.

A second important lesson is Jordà’s pursuit of a “sonigraphical” instrument. In Epizoo, FMOL, and in Reactable, Jordà strives for a kind of unification, or at least colocation, of visual and sonic feedback with the user interface elements. In a sonigraphical instrument, the GUI is “both an input for sound control, and an output that intuitively displays all the sound and music activity” (Jordà 2003). In FMOL’s GUI window, audio signals are visualised in oscilloscope-like waveform traces which can be manipulated with the mouse. In Reactable, the unification is even deeper with a tangible interface projected on a round tabletop. Synth elements are denoted by physical markers which are activated as soon as they are placed on the table. When elements are patched together, oscilloscope traces flow between them to show the signal path. In fact, signal patching and visual connections are unified with physical connection in Reactable, as these are all made simply by moving the markers closer together. Mouse-dragging static patch cords around in Pd or Max feels clunky by comparison!

This level of sonigraphical unity was not achieved without effort on the part of Jordà’s team. He writes that their first instinct was to develop a system to control a large visual display with body motions. Such an idea would have had similar technological challenges, but none of the intuitive impact and musical possibilities of their final design. The problem of achieving effective sonigraphical designs still challenges NIME-creators today. While mobile multitouch devices would seem to suggest more expressive, tactile manipulation of sound, conservative software continues to be modelled after physical studio setups and antique DAW designs where the primary visual feedback is a VU meter. In 2003, Jordà defined a benchmark for connections between the sonic, the visual, and the physical in an interface that will amplify the musical intentions of users and minimise frustrations. It is notable in Jordà’s paper that the technical details of his systems are downplayed in favour of explaining the evolution of the sonigraphical design rationale over years of performances and workshops. This, and the success of the Reactable system since NIME 2003, shows us that in NIME research, sustained engagement with performance and users can outweigh short-term technical novelty.