Keywords

1 Introduction

Researchers are designing automated exercise coaches for health [6, 7] and sport experts or beginners [10]. A growing body of evidence from Psychology and Sport Sciences shows that physical activity can be a cost-effective and safe intervention for the prevention and treatment of a wide range of mental and physical health problems [1]. Yet, the design of virtual health agents faces multiple challenges such as [9]: (1) interpreting the situation and people’s intentions, (2) intervention reasoning, (3) generating informative, educative, persuasive computer behavior, and (4) engineering generic solutions. Research in domains such as the Internet of Things (IoT), wearables and persuasive technologies [2] does suggest that a coach intended to promote physical activities needs to provide tailored interaction [8, 17], personalized messages and bring into play persuasive strategies that depend on gender and personality [3], and stage of behavior change [4]. The use of persuasive technologies and virtual coaches for promoting physical activity might help stress management [5], but adhesion might also depend on aspects of the self [15, 16].

Thus tailoring information is significantly more effective when it provides specific details on how to improve the performance, rather than when it just indicates whether the performance is correct or not [30]. When tailoring information involves emotional dimension, interaction becomes more attractive, increasing student engagement and motivation [29, 31]. Finally, when users manage the tailoring process, Kim et al. [32] observe in a positive attitude toward the conveyed information.

Although lots of people still use a phone for collecting data during a physical activity, smartwatches display several advantages over mobile phones: they are closer to the body, safer to use than the mobile phone when moving, and they are designed to have a longer battery life. Using smartwatches to support affective interaction nevertheless rises challenges in terms of human-computer interaction. Multiple commercial wearables for sport activities are available. We claim that in order to become smart, an automated coach requires to provide via a smartwatch: (1) interactions that are personalized to the user, (2) dynamic support occurring not only before and after a physical activity but also during a physical activity, and (3) multimodal interactions (including subtle and complementary use of embedded graphics, audio and touch).

In this paper, we introduce WE-nner, a framework for the design of a coach supporting personalized interaction for physical activities. We illustrate how we implemented this coach using a commercial platform for embedded programming. We discuss how this framework impacts the relation between users and computing devices (smartwatch, mobile phone, web site) and may improve the relationships between users and computing devices.

2 WE-nner Framework

Brinkman [9] proposes a research framework for behavior change support systems that brings into play situation interpretation, intervention reasoning (using personal and population data; possibly calling for remote assistance from human health professionals), and the generation of informative, educative and persuasive behaviors. In our framework we focus on this last step which requires that the system handles relevant features of user’s profile and is able to use it dynamically during a physical activity to generate personalized and multimodal interactions.

2.1 Multimodal Interaction During Physical Activity

The WWHT framework [19] describes how to present multimodal information to users along several questions: What is the information to present? Which modalities should we use to present this information? How to present the information using these modalities? How to handle the evolution of the resulting presentation?

Few studies about multimodal interaction were conducted with smartwatches during physical activity. Lee et al. designed multimodal and mobile interaction between a user and a sport-watch while the user is walking on a treadmill [14].

Interaction with a sport-watch can be personalized along several dimensions:

  • Time: interaction can take place with the user at different timescales around a physical activity, for example several days before an activity, just before the activity starts, during the activity, just after the end of the activity, and several hours or days after the activity. One might expect that interactions at these different times would have different goals and results

  • Modality: information can be presented to the user on several channels and modalities: on the visual channel (graphics, text messages, text menus: displayed on the screen), touch (vibrations), and audio signals

  • Content: technical messages, motivating message, warning messages,

  • Goal: the task of coaching a user involves several components and use cases (e.g. weekly burnt calories)

Currently, user’s profile include the user’s name, gender, history of activities and weekly goal in terms of burnt calories.

2.2 Software Implementation

Few smartwatches provide a software Development Kit and a programming language that enables embedded computing on the smartwatch. This is nevertheless required to dynamically customize the interaction with users. This allows to design customized interactions that fit between minimal interactions available on other commercial wearables and full symbiosis that are investigated in research.

We implemented the WE-nner framework using the GARMIN Connect IQ environment and its MonkeyC object oriented programming language. GARMIN is one of the few smartwatch manufacturer that provides a Software Development Kit that enables to develop a program on a PC and to upload and embed the software on the watch itself. GARMIN offers a dedicated programming environment called Connect IQFootnote 1 which enables to program and upload several kinds of applications on a smartwatch (face, widget, data field and apps). Different types of applications have different degrees of access to the sensors and actuators of the watch and allow for different types of interactive capabilities. GARMIN wearables support four types of interactions components:

  • WatchFaces provide personalized “passive” displays of the main screen. Users are able to choose and download the watch face they prefer and add any information they like.

  • Widgets provide at-a-glance information to the user that meets the individual customization. They are usually small practical tools like a compass or a weather report and are limited in terms of interaction capabilities.

  • DataFields are fields displaying data which is computed or available at runtime (e.g. speed, time). The expert user can select the fields that she wants to be displayed and their order of presentation. DataFiels do not support any interaction.

  • Apps are the most interactive components that can be uploaded on the watch. They can contain menus, data, textual and graphical messages that can be selected and combined at runtime. An app is explicitly started from the main menu of the watch.

Interaction with advanced sport smartwatches such as a GARMIN watch can be quite tricky for the ordinary user. Furthermore, even apps available on the GARMIN store do not provide flexible and personalized multimodal interactions.

We implemented our WE-nner framework as a Connect IQ App in order to benefit from the maximum access to sensors and actuators. Figure 1 illustrates how the WE-nner software uploaded on the GARMIN Fenix 5 smartwatch provides interaction that is personalized at runtime before an activity starts. The smartwatch collects information from an .xml file detailing user’s profile (e.g. user name, user birth date, activity history). The menus as well as the responses of the different physical buttons that surround the watch can be completely changed at runtime to cope with a given user.

Fig. 1.
figure 1

Screendumps showing personalized interaction before an activity starts: (a) WE-nner displays a personalized message including user’s name, (b) it suggests several activities (which can consider user’s history of previous activities), (c) the user is able to select the activity she wants to do, and (d) the activity is ready to start.

Full size image

Interaction can also be personalized during a physical activity. For example, a warning text, audio and tactile message (vibrations) can be generated dynamically by WE-nner when the heart rate frequency goes beyond a given percentage of the current user’s maximum heart rate frequency (Fig. 2).

Fig. 2.
figure 2

Screendumps showing personalized interaction during an activity: (a) at some point during an activity, a multimodal (graphics, audio, vibrations) personalized message is dynamically computed and displayed including user’s name and considering a threshold of heart rate frequency that is specific to this user.

Full size image

Finally, WE-nner also enables to personalize interaction after the end of the activity, and display for example a congratulation message that embeds user’s name and information about the current achievements in terms of burnt calories (Fig. 3).

Fig. 3.
figure 3

Screendumps showing personalized interaction after an activity: (a) a personalized message is computed at runtime on the watch and includes user’s name and the number of burnt calories during the activity, and (b) the percentage of this user’s weekly calories goals can be graphically displayed.

Full size image

Figure 4 illustrates the displays on the watch during an outdoor test.

Fig. 4.
figure 4

Personalized messages displayed on the smartwatch during an outdoor test.

Full size image

2.3 Software Architecture

The smartwatch provides sensors, screen and loudspeaker that enables a user to track her daily physical activities. The activity information that includes time, duration, speed, distance, calories burned and heart rate is mostly saved in .FIT binary files. The GARMIN connect web site enables a user to login and visualize the data related to his/her recorded activities and health status. The GARMIN Connect Mobile Application is required to synchronize the data between the watch and the web site.

We have developed an Android application called WEnnerMobile. It enables a user to view his activity history, visualize his heart rate records by curve chart, and test his/her pressure level by a answering a questionnaire. It will also provide a virtual coach to give the user personal advice on exercises. The user can choose a FIT or CSV fitness file stored in the phone memory. When a FIT file is chosen, it is be decoded into a CSV file, and the data can be processed and displayed in tables and charts.

Finally a website is used (WEnnerWeb) to store data collected with the watch and the mobile phone. The administrator of this web site can define the sport types, user’s initial preferences of the sport and the target training time. Each time when an activity is finished, the WEnnerWatch app can collect Ecological Momentary Assessment and ask the user to self report on her mood (e.g. “Positive”, “Negative” or “Neutral”). Such information can be exploited to tailor and adapt follow-up sessions (Fig. 5).

Fig. 5.
figure 5

Devices and software architecture of the WEnner framework.

Full size image

3 Conclusions and Future Directions

We introduced the WE-nner technological framework for designing a coach that supports physical activity and enables personalized, multimodal and dynamic interactions between a user and a wearable. We explained and illustrated how this framework was implemented using a commercially available software development kit for smartwatches.

The next step is to define the specifications of the personalized multimodal coach. We will go through a requirement analysis involving users with different levels of physical activities practice. This requirement analysis will identify the desired functions of the coach as well as its multimodal interaction capacities. This includes the selection of opportune motivational interactions delivery timing [28].

Next steps include the modeling and implementation of relevant personality and inter-individual differences features in the WE-nner user’s profile. We are considering two theories from Psychology: the OCEAN personality traits and the regulatory focus theory. These two theories because they have an impact either in terms of physical activity itself or in terms of persuasiveness. The OCEAN/Big Five personality traits, also known as the five factor model (FFM), is a model based on common language descriptors of personality [26]. The five factors have been defined as openness to experience, conscientiousness, extraversion, agreeableness, and neuroticism, often represented by the acronym OCEAN. Relations have been found between the OCEAN personality traits and motivation to learn [27], but also with physical exercise (see [21] for a review). For example, Tolea et al. found [20] found some associations between personality traits and physical activity level. Saklofske et al. observed that self-report emotional intelligence mediated the relationship between personality and exercise behavior [22].

The second theory that we are considering is the regulatory focus [24]. Regulatory focus has been shown to influence how individuals make judgments and decisions [23]. We have already regulatory focus in our MARC virtual agent platform [12, 13]. Regulatory focus is also being used for the generation of persuasive messages [11]. Individuals are either gain-oriented (“promotion-focused”) or loss-oriented (“prevention-focus”). Framing messages influence individuals’ cognitive processing of messages [25].

In terms of interaction, we will extend our framework for supporting the dynamic selection of output modalities and their combinations (e.g. complementarity, redundancy) to achieve an appropriate integration of the senses by users. This requires considering contextual information (e.g. if the user is on the move, a vibration can be used to inform the user that she should stop and look at an important message on the watch). Frameworks for multimodal output generation will be considered [19]. We will also consider the design of consistent interactions between the smartwatch, the mobile phone and a web site in order to support the relation between the user and her personalized coach which is in fact dispatched over several devices (possibly including other sensors and wearables).

We are also considering on how an animated and expressive agent displayed on the smartphone (and simple representations of it on the watch) can motivate the user based on affective reasoning and data [18].

These dimensions of personality and multimodal interaction will be considered as input for the modelling and implementation of the coach using the WEnner technological framework introduced in this paper.

Long term user studies need to be conducted to test if this wearable and its personalized and multimodal interactions do induce engagement and behavior change, and to assess how much they are impacted by aspect of the self [15, 16]. Ethics and privacy are also considered since personal data is collected and can induce behavior change.

Finally, sport coaching will be articulated with another coach in charge of stress management so that both coaches can exchange information in order to fine-ture and tailor consistent interactions.