1 Introduction

Atoms and molecules are too small for us to see and consequently their detailed and often colourful depiction in scientific literature originates as much from the imagination of scientists as it does from reality [1]. This ‘molecular aesthetic’ may be intended primarily to convey information and stimulate new understanding, but it also reveals a hidden beauty in the natural world which has inspired new forms of contemporary art [2] and music [3].

Interactive evolutionary computation (IEC) has been used across a range of artistic domains to explore complex design spaces and support artists in the production of novel work [4]. In this paper the potential for IEC is explored for evolving the aesthetics and dynamics of an interactive atomic simulation engine called danceroom Spectroscopy (dS). Following a short description of the dS system and IEC, two interfaces supporting single- and collaborative-user evolution are introduced and studied. Examples of the imagery produced within these studies are provided and discussed, followed by concluding remarks and plans for future work.

2 danceroom Spectroscopy

danceroom Spectroscopy is an internationally acclaimed art/science project fusing interactive technology with rigorous particle physics to create an interactive and immersive simulation of the atomic nanoworld, see Fig. 1. Multiple participants are able to simultaneously traverse and interact with the system through whole body interaction as their image is captured using an array of depth cameras and embedded into an atomic simulation as electrostatic fields. dS has been deployed as the framework underpinning a series of international dance/music performances, art/science/technology installations, and as an educational platform for teaching science and dance to school-age children. For fuller background, mathematical, technical and artistic expositions see [5, 6].

Fig. 1.
figure 1

danceroom Spectroscopy

Full size image

The dS physics and graphical rendering engine is highly flexible, capable of simulating extremely diverse visual and interactive states. It is manually controlled using a dedicated graphical user interface (GUI), shown in Fig. 4. Due to the complexity of the design space, configuring a dS state requires that users possess expert knowledge of atomic physics and computer graphics. Consequently, this interface is rarely exposed to general participants, whose experiences of the system are normally limited to interacting with a number of pre-programmed states. In dance, educational, performance and installation scenarios, the GUI to dS has been observed to present novice users and collaborators with a significant obstacle when attempting to explore the capabilities of the system in an improvisatory manner. Typically, they tend to work closely with an experienced user, verbally describing their intentions - a process that is highly inefficient. However, this barrier to entry can be elegantly addressed with the development of an IEC interface to dS, enabling the domain specific controls of the simulation to be abstracted by a simpler process, guided by the subjective aesthetic preferences of a user.

3 Interactive Evolutionary Computation (IEC)

Interactive evolutionary computation is a long-established and powerful optimisation method for finding solutions to complex design problems. IEC has been utilised across a range of disciplines, most notably in art and design but also within domains such as science and engineering [7]. IEC’s synthesis of computational optimisation and instinctive human enquiry has been shown to produce higher-quality solutions more rapidly than when humans and machines work independently [8–10]. One of the principle advantages of IEC over more traditional optimisation methods is that it enables design solutions to be evaluated by a human user, without the need for a formalised objective function, a requirement of canonical evolutionary algorithms. As the dS design space is large and already requires the subjective evaluation of visual and interactive features, IEC offers the potential for a powerful alternative to the existing GUI interface. The application of IEC to dS also opens up a novel opportunity to explore the interactive evolution of atomic aesthetics.

4 Evolving Atomic Aesthetics

At the heart of dS is a rigorous atomic dynamics simulation based upon Hamilton’s equations of motion, which is a common approach used to study the dynamics of classical and quantum molecular systems. The Cartesian coordinates of up to 30,000 simulated atoms are computed using dedicated GPU accelerated software on a high-performance workstation at 60 frames per second [5]. The system is configured via a software GUI, which is shown in Fig. 4.

To enable the dS visual and interactive design space to be explored using IEC, an interactive evolutionary algorithm was created based upon the (\(1\, \overset{+}{,} \,\lambda \)) Evolution Strategy (ES) [11]. In this approach, a population of individuals is evaluated at each generation from which a single parent is selected and mutated to produce \(\lambda \) offspring. For ease of development and experimentation, this algorithm and its interface were developed as a separate application to the main dS system, communicating through an existing Open Sound Control [12] interface.

4.1 Representation

A subset of 60 dS parameters control the atomic dynamics and aesthetic representation of the system. These include physics parameters (such as atom size, simulation temperature and number of atoms) as well as visual parameters (including atom/electrostatic field colour, time-history visibility and colour inversion). Consequently, a vector of 60 values for each of these parameters represents a single solution within the dS design space and the genome of a single individual within the IEC population. The majority of these parameters take real valued numbers in the range \(0 \rightarrow 1.0\); however, a minority of parameters have binary state values, represented by a boolean in the solution vector.

4.2 Evaluation and Selection

To assess an individual for fitness, the unique combination of parameters comprising its genotype is transferred to dS where it is expressed as a visual and interactive state (or phenotype) whose aesthetic appeal may be evaluated by a human observer [13]. The computational complexity and presentation format of dS limits users to evaluating one individual at a time. Consequently, the IEC interfaces described later were developed to enable users to evaluate each population member in sequence before selecting a single individual to act as a parent of the subsequent generation [7]. The fitness of a dS state is determined by its aesthetic and interactive merit (i.e. the extent to which the user likes and finds it engaging). As part of the selection process the user must provide a fitness score in the range \(0 \rightarrow 1.0\), enabling the user to steer the process towards increasingly optimised regions of the search space.

4.3 Mutation

When the user selects a parent individual from the population, the next generation of progeny is created exclusively by mutation. The mutation process in this work is also taken from the standard ES algorithm: real valued parameters from the parent individual are copied to each offspring with the addition of an independent random number drawn from a standard normal distribution. Binary values are flipped when the outcome of a simulated dice roll is 1. In both cases, the mutation strength is linked with the fitness value the user ascribes to the parent, such that on average larger mutations are observed when parents are awarded lower fitness to promote exploration. Conversely, smaller mutations occur, when parents are of high fitness to promote optimisation. In practice this mutation step size adjustment is achieved for the real valued parameters by setting the standard deviation (\(\sigma \)) of the random normal mutation inversely proportional to an individual’s fitness (f): \(\sigma = (1 - f)^2\). For binary values the number of sides on the ‘die’ is set inversely proportional to the individual’s fitness: \(s = \lceil \frac{1}{f^2} -1\rfloor \), with no flip performed when \(f = 0\). Consequently, parents assigned a fitness value of 0.0 will undergo very strong mutation and parents with fitness values of 1.0 will not be mutated.

5 Experimentation

Two distinct and practical use cases were considered in which users could evolve atomic aesthetics with IEC. First, single-user scenarios where a user creates atomic visual states rapidly in isolation in preparation for dance performances and art installations. Secondly, collaborative-user scenarios in which groups could collectively contribute to the design of a visual state. For example, this might take place during installations or performances where visitors or audience members are able to form a consensus to help evolve designs over longer time periods.

5.1 Single-User IEC Interface

The single-user IEC interface, shown in Fig. 2, is written in C++ using the JUCE application framework [14] and modelled on the seminal IEC example: Dawkins’ ‘Blind Watchmaker’ program for evolving biomorph creatures [15]. The interface presents users with a grid of nine buttons, each representing an individual member of the population. By hovering their mouse over a grid button, users are able to load the associated individual into the running dS engine where it is rendered instantaneously onto a separate display. Users are able to rapidly switch between the nine population members while remaining immersed in the dS experience. When ready, the user can click the button associated with their preferred individual to select it as the parent of the next generation. The underlying evolutionary algorithm implements a (\(1\, + \, 8\)) elitist strategy, meaning that each new generation comprises the selected parent along with eight mutants. Users continue this evolutionary cycle until either a satisfying solution is located or they become fatigued.

Fig. 2.
figure 2

Single-User IEC Interface

Full size image

Following an informal heuristic evaluation with the project contributors [16], subtle modifications to the interface were made to enhance user experience. For example, when candidate solutions are being auditioned, their associated button turns red and a heart image appears at its centre (see Fig. 3a). The size of the heart varies with the x-position of the mouse cursor, enabling the user to express the degree to which they like the associated individual. Upon selection with a single click, the size of the heart is converted to a fitness value, assigned to the chosen individual and factored into the subsequent mutation process as described in Sect. 4.3. At the turn of each generation, the selected individual persists to the next generation: its button turns green (see Fig. 3b) and the remaining individuals are replaced with eight mutated offspring.

Fig. 3.
figure 3

Single-User IEC interface (a) auditioning a candidate solution and (b) candidate selected as parent

Full size image

The immediacy of this IEC interface allows the user to create and review a variety dS states many times more rapidly than would otherwise be possible using the standard GUI. The process is also unbiased in its exploration, jumping to regions of the design space that might not typically be encountered manually. This method of interaction forms an example of ‘mixed initiative co-creativity’ [17], where the user’s creativity is augmented by computational initiative. Computational creativity can be broadly classified in three types: combinatorial, exploratory and transformational [18]. This process makes use of exploratory creativity by presenting users with novel solutions to enable serendipitous discoveries [10]. This is highly useful for human evaluation of aesthetic quality, given that the user’s objective is not a fixed domain (like the objective function of a typical engineering problem) but can change and morph as they interact with the system over time.

Single-User Pilot Study. An initial pilot study was performed to evaluate the efficacy of the single-user IEC interface compared to the standard dS GUI. Seven participants with no prior experience of the dS system were observed interacting with the simulation using both interfaces. The participants were individually sat at a desk in front of a monitor showing the standard dS GUI (see Fig. 4), next to a second monitor showing the IEC grid interface (see Fig. 3). The simulation was running on the dS workstation connected to an 83" wall mounted screen positioned in front of the user.

Fig. 4.
figure 4

dS GUI

Full size image

Each user was given a maximum of four minutes to design an aesthetically engaging visual and interactive state for dS using each interface type. The first interface encountered was alternated for each participant in an attempt to control any effects that ordering might have on the results. Participants were subsequently asked to comment on their experience with each interface and to answer a series of questions. The results are summarised in Table 1 below.

Table 1. Summary of single-user IEC interface pilot study
Full size table

Figure 5 shows screenshots of the dS states created by four participants during the pilot study; the left hand panel shows states designed using the standard dS GUI and the right hand panel shows states designed using the IEC interface:

Fig. 5.
figure 5

Example dS states created by four participants; left images designed using the dS GUI, right images designed using the IEC interface

Full size image

The states designed using the IEC interface are much more vibrant and exhibit greater variation than those made manually using the dS GUI, and this is reaffirmed by the fact that all seven participants stated that they preferred their evolved state. Many participants reported feeling overwhelmed by the sheer number of parameters and their complex names in the dS GUI and few managed to explore beyond the first ten within the time limit. States developed using the standard dS interface exhibit very little aesthetic variance and are all clustered around the experimental starting point within the design space. In four minutes participants were unable to explore the full range of aesthetic potential, and many design aspects including colour remained unchanged. For example, only one person managed to find the parameter controlling the visibility of their electrostatic avatar. However, all of these parameters were simultaneously modified by the IEC interface, allowing participants to traverse a wide range of aesthetic states much more freely. This is reinforced by the differences in time spent using each interface: six out of seven participants ran out of time when using the dS interface. On the other hand, most participants were able to reach a satisfying state within the same time limit using the IEC interface.

Participants also stated that they felt more engaged with the simulation using the IEC interface as they were not focused on the intricacies of the GUI, instead devoting their attention to the on-screen aesthetics of dS. However, it is clear from the results that the majority of participants preferred to use the standard dS interface over the IEC. The main reason given for this was that participants felt they had more direct control over the simulation and were able to more precisely and intentionally change aspects of the visualisation. The IEC interface was often described as more “random” and “chaotic”. However, one participant particularly enjoyed this stochastic behaviour, comparing the differences between the interfaces to selecting music: “the joy of shuffling rather than choosing a song that you like”. Most participants preferring the dS GUI did note that it would take a great deal of time to learn to control the system effectively and that the IEC interface presents a much lower barrier to entry.

One feature that was utilised extensively throughout the study by participants was the undo function. Many participants realised that they had reached a “dead end” and chose to revert to a previous generation. This feature was noted to enhance the evolution of dS states as participants were able to revert back to earlier positions in the search space and explore an alternative design path.

5.2 Collaborative IEC Interface

As the previous study indicates, IEC is an excellent tool for enabling single-users to explore large, subjective fitness landscapes. A second scenario that was considered allowed multiple users to collaboratively control the IEC process, evaluating and selecting individuals by majority consensus. This method of crowdsourced evaluation has been adopted in a number of previous IEC studies to create collaboratively evolved music [19] and imagery [8]. Collaborative IEC methods have the advantage of leveraging the efforts of multiple evaluators to address problems of user fatigue [20], enabling prolonged evolution such that fitter and more universally engaging designs can emerge. Given that dS is frequently deployed as an art installation for extended periods, a collaborative approach is more likely to result in wider exploration of the design space and ensures that the experience is continually changing as participants contribute to the evolutionary process.

To support the collaborative evolution of dS states, the underlying algorithm was modified to a (\(1\,+\,1\)) strategy. That is, at each generation the parent is mutated to produce a new offspring, which is loaded into the dS system for evaluators to observe. If the new offspring receives a positive reaction it replaces its parent. However, if the new individual receives a negative reaction, it is discarded and the original parent persists.

An interface to the collaborative IEC process was developed to accommodate any number of users participating using their smartphones, joining or leaving a voting network as they enter and leave the installation. A simple web application was designed and developed, enabling participants to either ‘like’ or ‘dislike’ the currently loaded dS state. The main program running the evolutionary algorithm was again written in C++ using JUCE and the voting service was written in JavaScript using node.js [21]. Users connect to an open Wi-Fi network and browse to the IP address of the host machine, which serves a simple web page featuring ‘like’, ‘dislike’ buttons and labels displaying the current votes of the group (number of likes/dislikes). A countdown timer is also shown to indicate when the next state will be loaded (see Fig. 6). The duration of the countdown is variable and can be easily adjusted to suit the environment. For example, in the pilot study described in the next section, a countdown of 20 seconds was used to maintain the focus of the audience and make the most of the time available. However, in a more permanent context, such as an art installation, a longer countdown would be more appropriate, providing time for participants to interact and play with states between transitions. Connected users are able to ‘like’ or ‘dislike’ the current state and when the countdown reaches zero, recorded votes are converted to a fitness value, which is used to mutate the selected parent as described in Sect. 4.3. Fitness at the end of each generation was calculated as the normalised ratio of the number of likes (l) to dislikes (d): \(\frac{l/d}{1+ (l/d)}\).

Fig. 6.
figure 6

Collaborative IEC interface web app running on a smartphone device

Full size image

Collaborative IEC Pilot Study. To assess the potential of the collaborative IEC interface, a short pilot study was conducted during an undergraduate Music Technology lecture at the University of the West of England with 48 participants. The dS simulation was projected onto a large screen at the front of the lecture theatre and participants were invited connect to the web server using their smartphones, tablets and/or laptops and to observe or interact with the simulation as shown in Fig. 7.

Fig. 7.
figure 7

Collaborative IEC pilot study

Full size image

The session began with a brief introduction to dS, a description of the experiment and instructions on how to participate. The system was then set running for 25 min, during which time 75 generations elapsed. A range of states were created in this time; a screenshot of one example is shown in Fig. 8. Following the experiment, a short group discussion was conducted to gather feedback.

Fig. 8.
figure 8

Example state produced during the collaborative IEC pilot study

Full size image

The participants commented that they enjoyed the process and liked many of dS visual states that emerged during the experiment. However, there was not always a clear connection between user votes and the subsequent response from the system. A positive consensus would not always produce changes that participants perceived to be positive. In these circumstances a strong positive consensus was frequently followed by a strong negative consensus, if the connecting mutation was judged to be detrimental. From a user perspective there appears to be an expectation that positive votes should produce positive changes; however, an unbiased mutation operator guarantees that offspring will not always be fitter than their parents. This factor could have been improved with a clearer description of the underlying evolutionary process at the beginning of the experiment to help manage user expectations.

Another point raised for discussion was the inability of the web application GUI to capture the nuances motivating participants to vote. For example, participants frequently used the ‘dislike’ button when they became bored with the current state and not necessarily because they disliked it. Perhaps improvements to this interface might incorporate a ‘change’ button to disambiguate fatigue from a genuine dislike. Additionally, more elaborate parallel IEC algorithms might be used in which multiple states are evolved in parallel, and transitions between these different states might help to alleviate fatigue.

6 Conclusion

In this paper, interactive evolutionary computation (IEC) has been used to evolve the aesthetics and dynamics of an interactive and immersive atomic simulation called danceroom Spectroscopy (dS). Two interfaces to the IEC process were presented that enable single- and collaborative-user evolution of parameters to support the artistic process of visual state design. Both interfaces were then used within pilot studies to assess their utility within these contexts.

The single-user IEC interface was studied with seven participants comparing the manual GUI interface with the IEC interface. While the IEC interface was shown to make it simple to rapidly explore the design space, direct and manual control of individual parameters was highlighted as an important feature of the dS GUI. This study suggests that it may be useful in future to consider additional forms of user interaction. For example, in addition to the metaheuristic global search, users could select certain aspects of the design space to evolve, freezing the remaining parameters; a method which has been shown previously to increase the quality and rate of search [22]. In application to dS, this might involve freezing all parameters except those affecting the characteristics of the atoms, or other distinct features of the simulation. In practice, a combination of both interfaces could support the creative process for experts, using the IEC interface to explore the design space before switching to the manual interface for fine-tuning parameters.

The collaborative-user IEC interface was developed to give any number of dS performance and installation attendees the ability to contribute to the design of visual states using their smartphone devices. The study successfully produced a broad range of aesthetic states and subsequent discussion revealed a perceived separation between individual votes and the subsequent response by the system. This was due to the diluting effects of aggregating votes to form a consensus, but also a feature of the mutation operator. The collaborative interface could certainly be developed further if it were deployed for an extended period within the context of an art installation. For example, alternative evolution strategies incorporating recombination could be explored, seeded with existing states that have been developed manually during previous installations. Each state could be evolved in parallel to ensure that users experience a variety of high-quality visual states which may help to reduce user fatigue and increase interaction times. Evaluation might also be gathered from alternative sources. For example, given that the movements of participants are already captured by dS, these movements could be classified to gauge the reactions of participants. Fitness evaluation may then take place without users having to vote explicitly. These avenues of research will certainly form the basis of future work and along with the techniques proposed in this paper, could have the potential for application within a wider range of digital arts platforms with large design spaces.