Keywords

1 Introduction

IoT - the Internet of Things - is increasingly described as the next step forward for digitalization in our society. IoT has recently been described as a driver for sustainability - including the development of smart homes and smart cities, for e-health solutions, and for learning. Further on, it has been proposed as a solution for more energy efficient transportation solutions, including logistics. In short, IoT has been proposed as a solution to a number of societal challenges.

In reviewing the current initiatives taken in this area it is not an understatement to say that the current expectations are high on these IoT systems to address a wide range of societal challenges. For instance, and in terms of how IoT systems have been proposed as a driver for sustainability we can notice this orientation under the emergence of IoT enabled areas such as smart homes, smart cities, and smart transportation solutions) (Firner et al. 2011; Pan et al. 2013). Further, we notice initiatives taken on using IoT to address health problems (Savola et al. 2012; Garcia, et al. 2017; Chishiro et al. 2017) and as an emergent opportunity in the areas of Financial technologies and Cyber Security to utilize IoT technologies in the design of smart systems – ranging from NFC and RFID solutions for mobile payments to finger print based encryption for mobile devices.

Here it should be noted that all of these IoT solutions build on “material interactions” as the central interaction model, and that these solution are all dependent on the following three factors: (1) tight integration of computing and networking with physical materials and objects (Want 2015; Romano 2017), (2) alignment to people’s needs (Pignotti et al. 2014), and (3) the development of methods and approaches (Wiberg 2013; Karana et al. 2016; Garbajosa et al. 2017) for working across digital and physical materials in interaction design projects (Sulistyo 2013).

However, and despite the current efforts made in this area to address societal needs, and despite these identified dependencies we still lack a stable knowledge base concerning interaction design built on material interactions – in particular in terms of (1) understanding user needs and (2) validated design methods. The consequence is that resource consuming trial-and-error approaches are currently applied. Furthermore, the overwhelming risk is that the current expectations on IoT to solve societal needs might not be fulfilled.

Motivated by these identified needs the aim of this paper is to propose a design method for doing material-centered interaction design that acknowledge and correspond to user needs while being relevant for the design of IoT systems that heavily depend on tight integration of computing and networking with physical materials and objects.

2 Proposing a Method for Material-Centered Design

With a point of departure taken in this background this paper proposes a method focused on material interactions as a unique approach for working across physical and digital materials in design.

Material interactions is a new approach to interaction design that stretches across physical and digital materials and approaches to the design of interactive systems and networked products (See e.g.: Wiberg 2018; Wiberg et al. (2013a); Wiberg et al. (2012; 2013)).

In the construction of the method proposed in this paper we take stock in the increasing interest in our community to address interaction design through a material lens. This includes for instance Jenkins (2015) approach to prototyping material interactions for IoT - the internet of things, material programming (Vallgårda et al. 2016), and material probes (Jung and Stolterman 2011). These approaches belong to a growing strand of research in interaction design where new design approaches to the materiality of IoT systems are currently being explored - including the work by Berzowska (2012) on approaches to programming materiality and the work by Karana et al. (2016) on craft-based approaches to the tuning of materials.

In this paper we describe (1) how our proposed method adds to this current body of research in HCI, and (2) we present a model for doing material-centered interaction design, and (3) we outline the implications from this proposed method for the design of IoT systems. Finally, we conclude the paper with a practical case that illustrate the need for such methods in practice followed by a draft of this method and suggestions about three cornerstones of importance for the further development of this method to address and scaffold the design process of developing new IoT systems.

3 Theoretical Ground for a Material-Centered Approach

In proposing a Material-Centered Approach to Interaction Design in general, and in particular in relation to the development of IoT systems we ground this proposed method in the growing body of research on material interactions in the area of HCI/interaction design research (See e.g.: Wiberg 2018; Wiberg et al. (2013a); Wiberg et al. (2012)) and current efforts made in interaction design research on addressing interaction design from the viewpoint of its materiality.

This notion of ‘materiality’ is a growing theoretical perspective in interaction design research. As formulated by Wiberg et al. (2013b) in their paper “Materiality matters – Experience Materials” it denotes a new perspective that enable design across physical and digital design, and it has its roots in Ishii's pioneering research on tangible interaction design (see e.g. Ishii and Ulmer 1997 and the most recent Ishii et al. 2012). As further suggested by Wiberg (2016; 2018) this approach is now expanding into new hybrid forms of digital products, including smart watches, smart cars, and the Internet of things – and it signals a trend toward combining digital and analog materials in design. As interaction with these new hybrid forms is increasingly mediated through physical materials interaction design is increasingly a material concern. One could even argue that the “material turn” in human-computer interaction has moved beyond a representation-driven paradigm (Robles and Wiberg 2010; Wiberg and Robles 2010), and in relation to this Wiberg (2018) has recently suggested that this idea of “material-centered interaction design” might work as a new approach to interaction design and its materials. This approach embraces a view of interaction design as a practice of imagining and designing interaction through material manifestations. Further, a material-centered approach to interaction design enables a fundamental design method for working across digital, physical, and even immaterial materials in interaction design projects.

This proposed method also takes an explicit point of departure in a set of related and recent research approaches to material-centered interaction design. This includes e.g. Giaccardi and Karana’s (2015) approach to understand material experiences, Jenkins and Bogost’s (2014) approach to prototyping material interactions for the internet of things, material programming (Vallgårda et al. 2016), and material probes (Jung and Stolterman 2011). These approaches belong to a growing strand of research in interaction design where new design approaches to the materiality of interactive systems are currently being explored – see e.g. Wiberg 2018; Wiberg et al. 2012 including aspects of the form (Jung and Stolterman 2011), and agency (Tholander et al. 2012), of interactive materials and systems and how they are experienced (Pignotti, et al. 2014). In terms of additional methods developed for working with this materiality this perspective is for instance reflected in the work by Berzowska (2012) on approaches to programming materiality and the work by Karana et al. (2016) on craft-based approaches to the tuning of materials.

While there is this growing body of research on material-centered approaches to interaction design there is also an even more stable research area on user-centered interaction design that we can turn to here as to address IoT in relation to both user needs and computing in the form of material configurations. In fact, it should be highlighted here that over the last 30 years the methods for studying user behaviors and to work with users in interaction design projects has been constantly refined and developed. Today there are specific methods for doing user studies (ranging from ethnographic observational studies), to more controlled experiments (c.f. Nielsen et al. 1993, Karat 1997, Dix et al. 1998, Virzi 1997, Wiberg 2003). Further, if reviewing the basic knowledge base for working across physical and digital materials in design of interactive systems we need to acknowledge that the area of industrial design serves as a stable knowledge base related to the design of things (including product design) and, in similar terms, we have a stable knowledge base that can inform the design of digital services.

To summarize, there is at the current moment (1) a new knowledge base forming around the tight integration of physical and digital materials in design, and (2) there exist a well-established body of research and methods for doing user-centered design. However, less is know about how to integrate these two strands. Accordingly, and in line with the research question formulated in this paper we need to develop new design methods for doing material-centered interaction design that acknowledge and correspond to user needs. In this paper we suggest that this is key for the further development of solutions that is built around material interactions as a central design principle in interactive systems design in general, and in particular when it comes to the design of IoT systems.

4 Exemplifying a Material-Centered Approach to IxD

I have now motivated the proposed material-centered approach from a theoretical viewpoint, but before moving forward to a draft where we present the cornerstones of one such method or approach I will now also illustrate how the experimentation with the integration of physical and digital material play out in practice.

So, as to illustrate this focus on material interactions and what it can look like when doing material-centered interaction design I will now in the following section present an example from our ongoing collaboration with the IT-company KnowIT in Sweden.

Here, this particular example illustrates material-centered interaction design in the context of an ongoing IoT project at KnowIT in Umeå. In this particular project the design challenge was to design an interactive system for a smart parking service. The problem addressed concerned how to design an interactive system that automatically can detect if cars occupy the parking lots, and to present this data to the driver who wants to park his/her car. Some commercial solutions are already in place for some parking garages that builds on counting cars and displaying the number of available slots on a screen, but more lightweight solutions that builds on data from the cars actually parked in each parking lot are still not available. In relation to this design challenge the IT-company KnowIT is currently exploring different solutions to this problem trough a trail-and-error approach.

Their first approach, Fig. 1(A, B and C) is to use ultrasonic sensors for object detection as to determine if a car is parked in a particular parking lot (1A). If that is the case the sensor detects that and the data is transmitted via a LoRa-network (1B), to a server and then a script sends a signal to an arduino board to turn a lamp on in the window (1C) so that a driver can see if the parking lot in the basement is occupied or not. Here, a combination of sensors, networks, server tech, and a lamp is used for the system architecture, and the position of the car serves as input to this system. Accordingly, this solution demonstrates design across digital materials and physical objects (in this case the position of the car), to offer a new service to the user, i.e. to meet a particular user need. However, this first approach has its limitations. It needs sensors installed for every parking lot, and it demand one representation/parking lot (in this case a lamp2).

Fig. 1.
figure 1

Two examples of design explorations in the area of material interactions at KnowIT.

Full size image

To overcome these problems KnowIT is now exploring an alternative solution (2A and 2B). For this second solution KnowIT is experimenting with computer vision. By using a camera and the YOLO v2 framework for object detection the camera can read the whole parking lot and look for parked cars in the parking spaces. To test this solution they first did a small-scale test with a number of toy cars (2A), before going for a full-scale implementation (2B). This small-scale design allowed to test the system by moving the cars around, whereas the full-scale test allow for testing the solution in different weather conditions (e.g. during a full snow fall as in Fig. 2B).

Again, this second solution also builds on the tight integration of physical and digital materials, but a completely different material configuration (in this case a combination of camera tech, computer vision for object detection, and the positions of the cars parked as input to the interactive system). As such this short example illustrates how different material configurations can enable a particular digital service. Further, it illustrate how different material configurations might solve a particular problem, in different ways, but also with different associated pros and cons – an aspect that again is a call for more systematic approaches and methods for design of interactive systems that depend on the interplay between physical and digital materials.

5 Discussion – Towards a Material-Centered IxD Method

So what can be learnt from this particular case in relation to the development of a method for doing material-centered interaction design in the context of IoT systems?

Well, first of all this practical case illustrate that it is possible to come up with different designs in relation to the same design challenge, second that each solution builds upon different material configurations, and finally, that each solution integrate the physical and the digital in different ways. For instance, in the first case the integration between the physical and the digital is made by using a proximity sensor to measure if a (physical) car is near the (digital) sensor, whereas in the second solution it is a (digital) camera that uses the YOLO v2 framework for object detection as to detect if a (physical) car occupies a parking lot or not.

On a more general level, and if now starting to draft the skeleton of a method for doing material-centered interaction design we can notice how a material-centered approach for sure has materials as a central concern. From a design viewpoint it is about understanding how different materials can be combined in the design of new interactive solutions, and that in return demands an understanding of what materials are available in the first place, second how different material properties can be used in the design, and further to have an understanding of how different material properties can be set in relation to each other in the design of interactive systems. In relation to IoT systems design I would say that this calls for a skill that I would like to label as a “material sensitivity”, i.e. an ability to carefully consider how different materials could be brought together in the design of an interactive system.

Further, and if now moving towards an understanding of how this might be done, that is to turn this into a method and approach, there are again a couple of things to consider. If truly subscribing to a material-centered approach, then such design becomes an activity of carefully shifting between imagining what different materials might add to the design, and really hands-on explorations of how such combinations of materials actually work in the design. In short, a dialectic process that is constantly shifting between reflecting upon, and trying out, different material solutions. As such, the material-centered approach becomes a craft-based practice that is both about intellectually exploring and imaging computing in material form, as much as it is about crafting, experimenting and building new solutions with the materials at hand.

In moving forward I would now like to take three quotes from the newly released book “The Materiality of Interaction” (Wiberg 2018) to discuss three cornerstones of importance for the establishment of a material-centered approach to interaction design.

In doing so I first start with the following quote that is about how a material-centered approach recognizes the multitude of materials available for the design of interactive systems:

“If computing is no longer limited to one single substrate (digital materials), and if the set of available materials (digital, physical, and smart) is growing at a rapid pace, then the biggest challenge is not finding ways to manifest interaction in material form; instead, the challenge is navigating this landscape of available materials and devising a method and an approach for doing this job”. (Wiberg 2018)

Further, the following quote illustrates that among these different materials we should make no categorical distinction between different matters. In short, there is no point in making a separation between physical and digital materials!

“It is important to remember is that a material-centered approach to interaction design does push for looking at interaction design through a material lens. In doing so it makes no ontological or metaphysical distinction between digital and physical materials” (Wiberg 2018).

Finally, the third quote that I want to introduce here illustrates “the interaction first principle” which is a central cornerstone for a material-centered approach to interaction design. It is central as it illustrate how a material-centered approach to interaction design at the same time can be established with user needs, i.e. the interaction being supported, as its main and foremost concern:

“The interaction-first principle is about conceptually defining the mode and form of the interaction being supported. It is about defining who the user is and how he or she will interact with the interactive system”. (Wiberg 2018)

With these three cornerstones as a point of departure, in the theoretical grounding in material interactions, in the current literature on “materiality” and how it plays out in the area of interaction design, and with this practical case as introduced in the paper as an empirical illustration I suggest that a draft of a method for doing material-centered interaction design can include the following three components and main activities:

  1. (1)

    Exploring and defining the form of interaction being designed

  2. (2)

    Exploring and evaluating the range of possible materials that can be used for designing the interaction

  3. (3)

    Working iteratively between the integration of different materials in the design of the interaction, while recurrently revisiting and if necessary revising the initial idea of the interaction.

Further, I should here also underscore that these three parts of one such method works well together with the “interaction-first principle” as to do material-centered design in close relation to user needs.

6 Conclusion

In this paper I have taken a point of departure in how IoT (the Internet of Things) is increasingly described as the next step forward for digitalization.

As a background I have described how a great number of applied research projects and development efforts has been conducted to address various specific needs, and I have argued that there is still a lack of a stable knowledge base – including developed theories and methods - for working across physical and digital materials in the design of IoT solutions.

Motivated by this identified lack of methods I have in this paper presented a theoretical and empirical ground for the development of a material-centered approach to the design of IoT systems. The proposed method is focused on material interactions as an approach for working across physical and digital materials in design.

In more particular terms I have in this paper (1) described how this proposed method adds to this current body of research in HCI, (2) I have presented a model for doing material-centered interaction design, and (3) I have outlined some methodological implications for the development of a method for the design of IoT systems.

Finally, I have introduced an empirical case to serve as a demonstration of the need for such methods in practice as to address IoT, and as to push the design of IoT systems forward.