Keywords

2.1 Introduction

As discussed in Chap. 1, the exploration of digital touch communication provided in this book is situated within a broad social revaluing of people’s sensorial experience and re-evaluation of the roles of the senses, a part of which is a technological awakening to the sensory and changing social configurations to notions of connection, and possibilities for touch enabled through technological innovation. This is driving a new wave of digital sensory communication devices and environments.

We approach digitally mediated touch as a (emergent) communicative mode, a sensorial experience entangled in the materiality and sociality of the body, the environment and technologies. In this book, the sociality of digital touch is our starting point, rather than the physiology of touch to move away from a concern with sensory receptors, tactile perception and neurological processes. We understand the physical, material, and sensory aspects of touch as a part of when and how touch-based resources are taken up (or excluded) and how they can shape – or are shaped by – people to become semiotic resources. While, we draw insights on the dimensions of touch from a psycho-physical and neuropsychological account of the physical experience and perception of touch, these accounts are limited through their focus on the individual, understanding of the senses as universal (and somewhat ‘fixed’), and not recognising “the role that culture plays in the modulation of perception senses function” (Howes 2011: 161). From our perspective, touch ‘sensation’ is much more than a biological and physiological process, insights on the physical dimensions of touch and the physiological processes through which ‘signals’ or tactile sensations (e.g. pain, temperature, pressure) are perceived are the ‘stuff’ of semiotics.

That digital touch communication is both a rapidly evolving area, and at the same time ‘state-of-the-art’ touch technologies are necessarily at an early stage of development and readiness, poses a number of methodological challenges for those who research it. In this chapter, we explore two such challenges.

First, the challenge of researching digital touch technologies that are unstable, lab-bound, and not yet domesticated. We have responded to this challenge of studying how people interact with such technologies, by using a range of methods and selecting a mix of technologies to enable ‘naturalistic’ interaction ‘in the wild’ to be observed; some early stable prototypes that can be demoed; alongside the lab-bound observation of unstable experiments and speculative early designs.

Second, the significant challenge of researching digital touch with under-developed methods and theories. We address this challenge through methodological innovation and an interdisciplinary approach using appropriate tools from multimodality, ethnographic tools attuned to the senses, art and design-based methods, and Human Computer Interaction (HCI) approaches. While multimodality and sensory ethnography provide the primary theoretical frame for this book, and InTouch more generally, our case studies all, albeit in different ways and to different extents, involve interdisciplinary collaboration and dialogue.

In the next section, we introduce our main framework, and make a case for our combination of a multimodal and sensory approach to the sociality of digital touch. We then turn to discuss our use of prototyping as a way to gain access to and generate digital touch experiences and imaginations and to support interdisciplinary dialogues on touch.

2.2 A Multimodal and Sensory Lens on Digital Touch Communication

Understanding bodily knowing through research on and with the body is a founding feature of the authors’ work within multimodality (Jewitt et al. 2016; Jewitt 2014; Kress et al. 2005, 2014) and sensory ethnography (Leder Mackley and Pink 2014) especially as articulated in relation to digital communication and interaction (Jewitt and Price 2019; Price et al. 2016). We bring multimodality and sensory methods together to explore touch in response to the methodological challenge of how to understand the changing social landscape of touch, and the need for embodied methods to help gain insight on socially orientated understandings of digital touch. In this section, we offer a brief overview of multimodality and sensory ethnography (for a discussion of the challenges of doing so, see Jewitt and Leder Mackley 2018).

Within multimodality, objects and sequences of interaction are understood as meaningful signs – the outcome of a person’s or people’s actions, imbued with the maker’s interests mediated through the environment in which the sign was produced or encountered (Kress 2010). Meaning is understood as socially situated choice from a (dynamic) set of available resources; the affordances of which are shaped through their historical, cultural and social usage and their materiality – all of which relate to and are shaped by technologies. Here we use affordance to refer to the idea that different modes offer different potentials for making meaning – a form of action. Modal affordances affect the kinds of semiotic work a mode can be used for, the ease with which it can be done, and the different ways in which modes can be used to achieve broadly similar semiotic work. Modal affordances are connected both to a mode’s material and social histories, that is, the social purposes that it has been used for in a specific context.

Multimodality enables us to describe, categorize and understand the material and social resources and affordances of touch, the principles that underpin them, how they are shaped and used. For instance, how signifier materials (e.g. temperature, pressure, texture) are, through their social usage (the work of people, communities and societies), made into signs shared by a community and used to communicate, and to shape, establish or maintain social norms and conventions (e.g. notions of gendered touch, see Chap. 4). As an approach to communication it stresses the relationship between meaning systems and the social needs they are used to serve (Bezemer and Kress 2016; Kress 2010). InTouch uses a multimodal approach to ask what is counted as touch by participants in a given context, what semiotic meanings they associate with the dimensions of touch (e.g. location, temporality), and how these are used and interpreted by people to make meaning. For instance, placing one’s hand on the shoulder of another person and holding it there for a long time, with pressure, to communicate intimacy and reassurance, or power and control (depending on the context). In this way, multimodality raises issues of power and agency, for instance in relation to who can touch. Using multimodality, we are starting to map the emergent dimensions of digital touch and the social conditions and contexts that shape it as well as to characterise people’s use of touch for communication with attention to the cultural and social norms and power relations that shape their use (Jewitt 2017).

If multimodality asks how meaning is made and communicated, what meanings are made, and by whom, sensory ethnography sets out to account for the experiential, how meaning is perceived, the sensorial and often unspoken dimensions of everyday life and human activity (Pink 2015). It presents a set of phenomenological approaches that are attuned to people’s sensory worlds and exist in theoretical-methodological dialogue with wider theories and concepts around human perception, place, knowing, memory, imagination, affect, and movement (Leder Mackley and Pink 2013). A key methodological feature of sensory ethnography is shorter-focused encounters with participants and the notion that much of what is important about our feelings and activities is not easily observed or put into language – tacit, embodied, and unspoken. This approach enables us to find routes through which to share or imaginatively empathize with the actions of people, collaboratively exploring with participants their ways of knowing, being and doing, whilst drawing on their own embodied and emplaced understandings (Dicks 2014). Video is used to generate ethnographic encounters in ways that account for their multi-sensoriality and functions, through a ‘form of acquaintance rather than description’ (Pink 2015: 2), as a way for the researcher to feel their way back into the research context. The video re-enactments used in our In Touch with Baby case study illustrate this ‘empathetic encounter’ with the participants’ sensory worlds with an emphasis on understanding the place of touch.

The analysis of the case study data uses a multimodal and multisensorial approach to explore how participants know and tell through touch and bodily interaction. Our engagement with the case study videos as data begins through re-viewing the recordings as a team, making notes of interactional details and tensions, reflecting on our own embodied experiences of the research activities and materials, and revisiting and handling prototypes with attention to their sensorial and social properties. We examine when and how touch, the sensorial and materiality are brought into the scope/discursive space of making meaning, as a way to generate understanding of participants’ conceptualizations and realizations of digitally mediated touch. This approach focuses in on the modal and sensory choices that people make to represent and communicate, how these choices are shaped by people’s interests, social position and context, and seeks to understand the social implications of their choices for meaning, communication and knowledge. These conceptualisations and organisations of touch are understood as framed by social-cultural historical contexts as dynamic and changing over time under the influence of new social factors. In this way, analysis of digital touch communication is grounded in the broad social framing provided by both approaches to emphasize the social-cultural embedded-ness of digital touch and to capture the nuances of the lived sensory accounts that shape the digital design and use of touch.

2.3 Interdisciplinary Dialogues of Digital Touch Communication

InTouch brings multimodality and sensory ethnography into conversation with a range of other disciplines. The emergent state of digital touch communication often means it is not possible to observe these technologies in use in a naturalistic setting (e.g. the home). This is especially problematic for multimodal and sensory ethnography, which usually observe technologies in naturalistic or ‘in-the-wild’ contexts, in order to explore issues of agency and power through uptake and the domestication of technologies (Rogers et al. 2013). However, HCI, design and the arts can create new digital touch communication environments through making, speculative prototyping, staged scenario work, and generating artistic experiences, enable us as social scientists to investigate new forms or potentials of social touch.

Our case studies (see Chap. 1) involve an element of interdisciplinary working through a mix of research collaborations, workshops and events, the research literature, the use of mixed methods and on-going dialogue, notably with:

  • Artists (e.g. The Art of Remote Contact, Virtual Touch case studies);

  • Neuroscience (e.g. Tactile Emoticon);

  • Designers (e.g. Designing Digital Touch case study);

  • HCI, computer science and engineering (e.g. In Touch with Baby, Tactile Emoticon, and Virtual Touch case studies);

This interdisciplinary mix is also reflected in the InTouch team, we have backgrounds in Art, Design, Fashion, HCI, Media and Communication Studies, Psychology, and Sociology. We seek to achieve a rich and nuanced account of digital touch communication, working across our methodological and conceptual differences to understand points of difference, and, where useful, how they can be productively brought into dialogue by contrasting and layering disciplinary understandings.

Disciplinary differences in where and how touch research takes place (Berker et al. 2006) offers potential for productive interdisciplinary collaboration. At a most basic level, we have had to explore how our different conceptualisations of touch and the digital: do we mean the same thing when we talk of touch? Each discipline has distinct, and sometimes incompatible conceptualisations of touch. These are embedded in disciplinary histories and methodological approaches. Within Tactile Emoticon, for example, whilst as social scientists, HCI interaction designers, computer scientists and neuroscientists we all understand touch as a complex phenomenon involving the body, brain, and the social environment, we conceptualise and research touch in ways that emphasise and attend to these elements differently. From a multimodal and multisensorial ethnographic approach, we account for a broad context of participants’ engagement with emoticons and digital communication, and their semiotic and sensory interactions with the device and one another, always reading this in relation to social and cultural norms. The HCI and computer science researchers are focused on designing and understanding interaction between the users and the device. The neuroscience researcher approaches touch in relation to the individual, physical realizations (the brain and the body systems), mechanisms and processes of touch perception, and the relationship between stimuli and the sensations and perceptions they affect.

It has been useful for us to understand the different views of technology across our interdisciplinary conversations and collaborations: the extent to which the digital is foregrounded or valued, the expectations placed on it, and the sense to which technology is mutually shaped through its use. For example, Invisible Flock, collaborators in the Art of Remote Contact case study, described themselves as having an ‘agnostic approach to technologies’, sometimes working with stable consumer technology and sometimes using physical computing systems to prototype their own, and the importance of not having a ‘default go-to technology…So we try to stay open to technology…the idea comes first”. The technology is introduced at a later stage of their research process than, for example, was the case in the Tactile Emoticon study – which had specific communicative ideas underpinning the design and therefore suggested the technologies to be used at an early stage.

Where and how the social comes into researching touch, has been a central aspect of our interdisciplinary work on digital touch communication. In the case of the Tactile Emoticon case study, for instance, while we all engage with the social, we did so by focusing in on different levels of the social. The social is at the heart of a multimodal and multisensory approach and its conceptualisation of touch a form of communication realising and realised through the social functions of a society or group, which are always present at the level of the individual. While HCI is a broad field, it is inherently social in the recognition of the user and their relationship with a device or technology, and places them at the heart of its research, ‘always looking at people and scenarios of use, not just how the person works’ (Field-note). Our neuroscience colleagues engage with the social in relation to understanding what prior experiences or pre-existing beliefs participants may bring to an experimental context.

Across the other case studies, both the artists, and differently so, the designers critiqued the social expectations of touch. A member of Invisible Flock spoke of trying to create ‘a digital layer of friction between these normal interactions to then perhaps begin a conversation around it. So, you make holding hands a little bit more complicated so that maybe you stop and think about it a little bit more and we can begin a new conversation.’ In other words, they work to actively extend and problematize a felt social experience for a visitor, they seek to create a new social touch moment, while we as social scientists attempt to capture, interrogate and understand it. Our interdisciplinary collaborations with art, design and HCI to create such environments or experiences, enable us as social scientists to investigate new forms or potentials of social touch have proven invaluable: opening up discourses of touch, and prompting speculation and imaginaries on digital touch desires and futures.

2.4 Prototyping

The difficulties many people experience in articulating bodily experiences, imaginations, and tacit knowledge raise challenges for research (Tarr et al. 2017: 1). We bring prototyping into the frame of social science as a way to engage research participants in exploring touch and digital touch communication. As social researchers exploring the multimodal and multisensorial qualities of touch, the ways prototyping enable the body to play a central role in generating qualitative data are significant (Jewitt et al. 2019).

With its origins in product development within Engineering, Design, Computer Science, and Human Computer Interaction, prototyping has typically been concerned with developing ‘an idea about a product, system, service or policy to meet human needs and devising a plan for executing that idea’ (Binder et al. 2011). It is also associated with Design Thinking, which advocates for ‘thinking with your hands’ as a way of quickly and practically exploring an idea and the feasibility and development solutions, to pre-empt wasting time and money on something that might not work or might not be ‘user’ centred, in a quick, cost-effective and contextually aware way (Dunne and Raby 2013). The re-orientation of prototyping to high level concepts and ideas, rather than design products and skills, has enabled it to travel across the boundaries of engineering and design into the humanities and social sciences, including anthropology (Salazar et al. 2017), and more recently, sociology (Lupton 2018).

The methodological migration of prototyping to social research has partly been fuelled by a desire to research ‘emerging and uncertain worlds’ (Myers and Dumit 2011; Salazar et al. 2017), notably imagined digital futures, ‘configuring future imaginaries that may not be expected to come to pass’ (Lupton 2018: 5). We have found, this method aligns particularly well with researching the unstable, uncertain, future-facing technological devices and environments associated with digital touch. By bringing prototyping into the frame of multimodal and multisensorial work, we situate it within a wider move towards innovative and creative social science methods (Jewitt et al. 2017).

Our use of prototyping across the InTouch case studies suggests prototyping can serve as a point of connection, a bridge, across disciplinary differences to support interdisciplinary research in the emergent and provisional area of digital touch. We have used prototyping, in which digital touch technologies feature as a research resource, in four different ways:

  • Observing participants from within HCI, engineering and industry demo their prototypes as part of an interview process

  • Deploying existing prototypes as research probes

  • Facilitating prototyping workshops with research participants

  • Collaborating with artists and HCI designers to inform the design and build of digital touch prototypes and observing interaction with these designs

These approaches to prototyping varied in relation to the function of the prototype, who had access to it, who was involved in making, the material and/or technological resources involved, and the degree of conceptual or technological ‘finish’. They provided an opportunity for case study participants, and us as researchers, to externalize unrefined concepts in material ways and, in the process, to identify and clarify key aspects of ideas, to make present new scenarios for digital touch communication, provoked questions and surfaced differences (e.g. in conceptualisations of digital touch and/or communication). The prototypes were reflected on, assessed, and refined, and provided a prop for participants (and researchers) to enact the experience of using a proposed artefact.

We found prototyping an effective research tool for exploring digital touch communication with participants from design, art, and HCI who are familiar with it, as well as with others with no prior design experience. The use of prototyping, in these different forms, helped to facilitate interdisciplinary dialogues and collaboration on digital touch communication across the social sciences, the arts, HCI and neuroscience.

The Imagining Remote Personal Touch case study, for example, used prototyping to explore the participants’ experiences, memories and imaginations of remote personal digital (touch) communication. Participants were asked to prototype a remote digital touch device, environment or system for use in a personal relationship. The prototyping focused on the process of making, using a diverse collection of materials (silicon, leather, feathers etc.), objects and prompt words (a wall of touch words on post-it notes), aiming to foster creative explorations around different sensory touch interactions. Of key interest was how participants used their bodies in the generation of ideas, the making process and demo process, when asked to ‘perform’ how their prototypes might be used. In addition to focusing on the process, we approached the prototypes that participants produced as meaningful multimodal and multisensorial signs, material traces of thinking, decision-making: signs of digital touch. We reflected on our own embodied experiences of the workshops and materials, revisiting and handling participants’ prototypes with attention to their sensorial and social properties. We focused in on the prototyping process, paying specific attention to how materials were brought into the making, in relation to which parts of the body, and to what consequences for the social implications of digital touch. In a separate workshop activity, participants also engaged with Kissenger, an existing prototype kissing machine (Zhang and Cheok 2016) as a technological probe. Participants’ interactions with Kissenger sparked conversations on the appropriateness of digital touch communication in remote personal relationships, intimacy and sex. Overall, prototyping enabled us to engage with participants’ sociotechnical imaginations of the materiality of digital touch, map imaginations of touch and technology to the body and digital touch technology interfaces (Fig. 2.1).

Fig. 2.1
figure 1

Prototyping to explore participants’ experiences, memories and imaginations of remote personal digital (touch) communication

Full size image

Prototyping featured in the Designing Digital Touch case study, a collaboration with User Experience (UX) lecturers at Loughborough Design School, in two ways: first, we co-facilitated a rapid prototyping session with 70 participating design students; second, we developed a prototype toolkit. Here, prototyping was part of a longer design process – a one-term module where students used a design brief to develop an innovative, future-facing digital product or service that enhances communication through touch for personal relationships, leisure, or health and well-being. Observations on the prototyping session provided ethnographic background and insights for the analysis of the students’ final concept boards and videos. The case study research led us to develop a prototype – the Designing Digital Touch Toolkit (see Chap. 1). We prototyped the toolkit, a card-based resource using the Double Diamond Design model (Design Council 2007), using a mix of brainstorming, referencing research papers and experiences on touch and digital touch, re-enacting pedagogic design scenarios from the module with our UX collaborators, drawing on the analytical themes that we saw across the work of the cohort, and referring to specific student concepts. Both these uses of prototyping provided a powerful point of connection between our different approaches to digital touch, opening up and articulating the sociality and sensorality of touch into the UX design space for digital touch communication (Fig. 2.2).

Fig. 2.2
figure 2

Prototyping session as ethnographic background and insights for the analysis of Design students’ final concept boards and videos. From top left to right: Amare by Betsy Cousins @betsyc_design; Puls by Joe Slatter https://www.behance.net/joeslatts82ca. Loughborough University design students exploring touch during rapid prototyping workshop

Full size image

The Art of Remote Contact case study used prototypes as research probes in the context of artistic practice-based research. Through ethnographic visits, demos and interviews we observed the artists’ development of the prototypes, the decisions, ideas and processes involved in their making as well as sharing links, papers, photographs and ideas in the development process. Observing the development of the prototypes gave us useful insight on the artists’ conceptualisation of touch, technology and digital touch. For instance, the prototypes, one of the artists explained, involved, “…taking existing tech and looking at them differently through a very human or poetic lens”, rather than creating technology to “solve a specific problem or for commercialization”. This stance created a very particular type of prototype, in contrast to the UX prototypes, designed to be unfinished without the visitor interaction. The prototypes were conceived as provocations, aiming to unsettle, generate friction with the seemingly familiar, to surprise, and to produce unpredictable tactile responses, rather than not ‘art-works’. Their ‘unfinished-ness’ was signalled in its design through the use of plain wood frames and pedestals, off-the-shelf TV/computer screens, and the artistic decision to leave wires and plugs exposed making the technology visible to the gallery visitors. This use of low-fi materiality, rejecting the visual ‘gallery-aesthetic’ invited the audience to contribute to the prototype’s completion through their interaction: invited them to touch. In this case study, prototyping generated a new digital touch environment, a public exhibition, for touchy exploration of digital touch communication, memories, experiences, and desires, as well contributing to exploration of the methodological potentials of collaborative working between social research and interactive art as research (Fig. 2.3).

Fig. 2.3
figure 3

Remote Contact exhibition, artistic provocations by Invisible Flock generate new digital touch experiences that can be observed and researched

Full size image

The Tactile Emoticon case study was a collaboration with colleagues in neuroscience and HCI on the design and build of a prototype device to send and receive affective or socially supportive touch. The prototype device was designed to provoke interaction and imagined uses of digital touch interaction between two people. It sends touch feedback to the hands of two remote users using heat, pressure, and vibration, features of which were controlled by the sender. The design and development of the prototype device itself, and study participant responses and uses of it, informed the research. Prototyping, as a shared design and making activity, helped to elicit discussion on touch. The act of experiencing touch together was a generative process, helping to foster creative, unexpected and unpredictable conversations and ideas on digital touch. Research by doing and making, the iterations, failures and imperfect outcomes that we experienced, worked to expose our different conceptualisations of digital touch, our orientations to the social, and our expectations and requirements of digital technology. The iterative process of developing the prototype, its perceived utility and glitches, along with desires for further functionality and control, provided a space of interaction for us as researchers, as well as participants, and the final prototype functioned as a research tool for studies within our different disciplines. The social science and HCI researchers undertook qualitative studies with the device, exploring participants free-play with it, their responses to a series of scenarios, and the ‘languages’ of touch communication that participants developed. Alongside this, a controlled, quantitative neuroscience study validated the tactile emoticon device to identify the affective responses to digital touch communication provided via the device. The prototype informed the development of design considerations for digital touch devices and future work (Fig. 2.4).

Fig. 2.4
figure 4

Iterative prototype development by Frederik Brudy as part of the Tactile Emoticon case study

Full size image

Within the Virtual Touch case study, prototypes provided a useful point of connection between expert developers within HCI, engineering and industry and us as social researchers. Several experts demonstrated their prototypes as part of the interview, for example, in the context of a VR rehabilitation environment, the designer invited the researcher to be in the role of patient and she, the designer, took the role of physio, talking the researcher through the tactile basis of her experience – the type of grip required, the level of pressure to use and so on, while the researcher challenged and experimented with the affordances of touch available to her: together they re-enacted the user situation through touch which provided a felt basis for the interview. Another expert used various prototypes (e.g. an exoskeleton glove and a robotic arm) in the interview to demonstrate their touch affordances, functions and limitations, and to highlight the “gaps of human perception” that design and engineering capitalize on to create a “realistic sense of touch”. While in another interview, we used our experience of the technology to prompt questions to the expert, to clarify and elaborate the touch potentials of the virtual touch afforded, and to specify and concretise the abstract concepts that were raised in interview. By attending to the different ways in which the experts utilised their prototypes in the interviews we conducted, what they brought to the fore, enabled us to move beyond the virtual technology to gain insight on the narratives, social questions, and contexts of use that informed their work on virtual touch (Fig. 2.5).

Fig. 2.5
figure 5

Dr. David Swapp, Manager of the Immersive VR Lab at UCL, London, demonstrating touch affordances of an exoskeleton glove

Full size image

Sharing and exploring concepts with collaborators from other disciplines through processes of making, touching, and manipulating materials and objects promoted collaborative interdisciplinary dialogues and thinking towards gaining new knowledge about relevant phenomena (Camburn et al. 2017): in this case, digital touch.

2.5 Conclusion

This chapter has made the case for making the sociality of digital touch our starting point and the focus of this book. We have discussed the methodological challenges of researching digital touch communication at a time when technologies are evolving rapidly and are not yet ‘domesticated’, and methods and theories remain under-developed. Throughout the chapter, and the book more generally, we have argued for the need to attend to the multimodal and multi-sensorial aspects of touch and the significance of interdisciplinary dialogues with, for instance, art, neuroscience, HCI, and design. We have made the case for an approach to digitally mediated touch as a communicative mode, a sensorial experience entangled in the materiality and sociality of the body, the environment and technologies. We have discussed the potential of four different uses of prototyping to bridge interdisciplinary differences in order to gain access to and generate digital touch experiences and imaginations for research purposes. In the next chapter, we map the complex terrain of digital touch technologies for communication.