Keywords

1 Introduction

The popularity of mobile devices has made the application market even more competitive for these platforms. The commercial success and the loyalty of users to an application are consequences of the pleasurable experience of its use [1]. Since the nature of user interactions with applications is opportunistic and depends on the willingness to perform these activities, the evaluation of an application should take into account aspects such as personality, emotions, mood, goals, preferences, previous experience and knowledge of the user. Moreover, the physical, social and virtual contexts where the interactions occur are other important aspects to be taken into account [2].

When usability involves the user experience, the evaluation metrics should be extended. For example, Jordan [3] adds the need to measure the pleasure and the pride awakened in the user during their interactions with an application. Norman [4] emphasizes that the goal to be achieved by the new applications is to extend the capacity of the user engagement. Valdes and Gootzit [5] evaluate the UX under the aspect of benefits and financial returns, concentrating its studies on increased revenue, cost reduction, and shorter time to market insertion of new products.

In a context where applications are not focused only on the user performance and execution of tasks, but also on the experience offered, it is important for professionals to understand how problems evolve and how they impact the user experience. The main idea is to observe how the product is experienced in real world scenarios and how to obtain information, which will improve the application interface. Having seen exponential growth of new software for mobile devices, it becomes important to find methods at low cost suitable for iterative development cycle and release of new versions [6]. Therefore, this work seeks to contribute to a better understanding of issues related to the user experience (UX), trying to answer the following research question: How to assess the user experience with applications used in daily life?

Accordingly, the paper describes an approach to collect and analyze interaction data. The proposed approach is supported by a platform called Sherlock, which is comprised of two main units. The first, called Data Collection Unit, is responsible for capturing data from the technique Experience Sampling Method (ESM) [7]. The second, called Data Correlation Unit is responsible for storing and enabling data analysis. The dataset is divided into seven dimensions: (i) the user profile; (ii) the device characteristics; (iii) the social context; (iv) the emotional context; (v) the spatial context; (vi) usability; and (vii) location.

The proposed approach was evaluated through an experiment conducted with two main objectives: (i) to verify the efficiency of the components responsible for the collection and analysis of data; and (ii) to identify the user experience with mobile devices, based on the dimensions provided for the collection and analysis of data.

The remainder of this article is divided into six sections. Section 2 describes the Sherlock platform. Section 3 details the methodology used to conduct the experiment to assess the proposed approach. Section 4 shows the results found. Section 5 describes some related work. Finally, Sect. 6 presents some conclusions and perspectives.

2 The Sherlock Platform

The ESM technique is the base concept of Sherlock platform for performing data acquisition. Several authors use the ESM technique in experiments to evaluate the user experience with mobile devices [8–12]. In this work, the technique is used to collect subjective data and targeted responses, allowing participants in an experiment to answer the questions contextualized in different dimensions. The technique is then applied to obtain information about the contexts, such as emotional, social and spatial aspects of interaction. In addition, some usability attributes, such as ease of use, satisfaction, learning, operability and flexibility are also investigated.

ESM allows researchers to collect qualitative data using two kinds of questions: (i) sending direct questions to users with predetermined response; and (ii) measuring the type (positive or negative) and the intensity of the emotion of the user when using an application [7].

Figure 1 illustrates examples of forms used to capture qualitative data. The left side of the Figure illustrates the use of the technique with specific questions about the social dimension in which the user is at that moment. On the right side, the caricatures are associated with the emotional state of the user, ranging from very unsatisfied to very satisfied.

Fig. 1.
figure 1

Examples of use of the ESM technique in the context of this work

Full size image

The main functional requirements of the Sherlock platform are:

  • Provide data collection in the natural environment of the user interaction (in real-life settings).

  • Allow investigation regardless the number of users.

  • Allow performing experiments for extended periods of time.

  • Support the event configuration so that the questions can be generated according to the following: (i) application (specific or independent); (ii) time (predetermined or random); (iii) hours (before, around or after of a programed event); (iv) in accordance with events of Operating System (touch in button, scroll bar, hotkey, etc.); and (v) when the device is connected in specific address IP.

  • Provide a quick and simple way to perform analyze of the captured data.

  • Allow reviews involving the crossing of the user profile with contextual and subjective data.

  • Capture the location of users to identify their daily routines.

The Sherlock platform was divided conceptually in two units, Data Collection and Data Correlation, which will be discussed in the following subsections.

2.1 The Data Collection Unit

The Data Collection Unit consists of an application for the Android platform called SherlockApp. It must be installed on the user’s mobile device. For this purpose, those who are interested in participating in an experiment should access the website Footnote 1 where the SherlockApp is available and accept its terms of use.

The features of SherlockApp are structured into four distinct services:

Detect running applications – This service connects the questions directed to users running applications. The questions are proposed only if the user is interacting with a specific application or all running applications, according to the target of the experiment.

Collect data with the ESM technique – When an application is running, random questions about its use, covering four dimensions (usability, social, emotional and spatial contexts) are generated. The objective is, later, to relate this information to evaluate the UX with a specific application or all applications installed on their mobile devices. Moreover, the SherlockApp can be used to assess an event in a real-life scenario (for example, to identify the student’s expectative before a class and his/her satisfaction after the class).

User location – A location service, based on the mobile devices’ GPS, collects data about the position of the user. This service can be configured to get position from time to time or when the user moves for some meters. The objective of this service is to enable the discovery of patterns of user movement, allowing, for example, to create services which assist the user in their urban mobility.

Transfer of information captured – This service is responsible for detecting new information stored locally on the user’s device and sending them to a repository, available on a server in the cloud. To transfer the data, the service identifies the availability of Internet access and creates a JSON object filled with the information captured by SherlockApp. The object created is directed to a Web service responsible for storing the data in a Database Management System (DBMS).

2.2 The Data Correlation Unit

The Data Correlation Unit was built on Amazon EC2Footnote 2, a micro instance of a cloud server, which allowed the configuration of three essential services to the proposal of Sherlock platform:

Web Server – it is responsible for providing the SherlockApp downloadable application and programs to populate the database. The Web Server used is the Apache Software FoundationFootnote 3. The programs were developed in PHP language. The choice of this language was due to the support to receive the JSON objects and the availability of APIs to interact with the MySQL database. The execution of the Web services followed these steps:

  • Connection to the database (DB).

  • Processing of incoming objects in JSON format to the data collected, followed by the organization of the data processed in variables.

  • Inclusion of variables in the DB using the functions provided by PHP language.

  • Sending of a message to SherlockApp application notifying the process result.

Database Management System (DBMS) – This service was implemented on the MySQL ServerFootnote 4 and aims to store the data sent by SherlockApp. The MySQL Server was chosen because it is one of the most popular DBMS today, has extensive documentation and is free.

Online Analytical Processing Tool (OLAP Tool) – The software used to enable the achievement of the data correlations was Pentaho Analysis ServicesFootnote 5. The choice of this tool is due to the fact that it is free for academic purposes, has extensive documentation and also, because it is in constant evolution.

3 Methodology Adopted

The methodology adopted for the experiment focused on applying four distinct phases: planning, implementation, data preparation and analysis of results.

3.1 Planning the Experiment

The first action taken at this stage was to define the focus of the experiment: analyzing the user experience with applications considering the seven dimensions previously described. This information will be obtained from the correlation of data automatically collected from user devices.

The second step was configuration of the module responsible for sending the questions to participants of the experiment. We chose to evaluate all applications running on the devices of the volunteers. The questions were defined to be exhibited to the participants at random time with maximum interval of four minutes in each application.

The third action was to build a website to provide information regarding the project and the SherlockApp application for the capturing and transmission of data interaction. Social networks were used to promote the project and recruit participants for the experiment.

3.2 Implementation of the Experiment

After agreeing with the terms of use, downloading and installing the SherlockApp application, each user shall be identified by the International Mobile Equipment Identity (IMEI) of the device, automatically detected by the application. Information about the size and resolution of the device’s display is also captured at this moment. Then the users must enter some data relating to their profile, such as: education, social class, age, education, occupation, city and state, using the application window. After finishing this step, participants will receive, at random intervals, questions related to the dimensions to be analyzed during the experiment.

3.3 Data Preparation

The preparation of data for the analysis was conducted according to the set of questions to be answered with the data captured during the experiment. Based on the objectives to be reached, we designed a set of questions that guide the analysis of the data obtained. The questions to be answered are:

  • What is the satisfaction level of users with the most popular applications?

  • What is the relation between the social context and the use of mobile applications?

  • What are the possible interferences of the spatial context in user interactions?

  • The momentary mental status interferes on the satisfaction with the applications?

  • What is the satisfaction of users with the usability of the applications?

  • What is the satisfaction level of user with their devices according to the configuration?

Next section is dedicated to the last phase of the methodology and presents the results of the evaluations performed during the experiment.

4 Analysis Result

The data acquisition occurred from 05/25/2014 to 06/25/2014 and involved 68 participants. In an attempt to answer the questions proposed in Sect. 3.3, the next subsections present the results of experimental data analysis regarding factors like satisfaction level, social and spatial context where interactions take place.

4.1 Satisfaction Level of Users with the Most Popular Applications

Participants ran 254 different applications during the experimentation period. The most frequently used apps are shown in the Table 1. It can be observed that WhatsApp was the most popular among the participants with almost half of the executions (44.3 %).

Table 1. The most used applications
Full size table

Another important point, in this assessment, refers to the type of application used. It was found that the most frequently used types were: an application to send instant messages, one to browse the Web, two in a social network category and one to arrange the contacts.

Table 2 shows the levels of satisfaction with the five most popular apps on the experiment. It must be emphasized that 100 % of participants declared themselves “very satisfied” with Google Chrome. As Google is the company which owns Android operating system and the Chrome browser, it is likely to have set the application to the needs of its platform, thus, enabling good use experiences to their customers. It can also be observed that all apps have more than 50 % satisfied or very satisfied users. This can be a good indication that these applications are among the most popular for the participants of the experiment.

Table 2. Satisfaction with the most frequently used apps
Full size table

4.2 Relationship Between the Social Context and the Use of Mobile Apps

The second investigation dealt with the social context in which the participants performed their interactions with mobile apps. According to Ickin et al. [8], this factor is a major contributor to the change in the user experience with mobile devices. Table 3 presents some findings obtained from this context dimension analysis.

Table 3. Analysis of the social context dimension
Full size table

The data related to Social Interaction indicates that in 40.8 % of cases the use of applications occur in the presence of other people. Furthermore, the data related to location indicates that 35 % of interactions occur in public places, where the presence of other people is usually inevitable.

Another fact that needs to be emphasized is that 50 % of interactions occur when users are studying or working, according to data related to the time the activity takes place. This information must be considered in the design of applications for them to be as simple as possible, minimizing possible attention deficits.

4.3 Interferences of the Spatial Context in User Interactions

The third study refers to the analysis of the influence of spatial characteristics of the interaction scenario. Table 4 summarizes the main findings obtained from the analysis of the data representing the spatial context of the interactions.

Table 4. Analysis of the spatial context dimension
Full size table

Initially, it was noted that most applications are accessed when users are sitting or lying down and that it does not interfere with the satisfaction of people. One fact that stands out is the percentage of 29.7 % of interactions when the participant is lying down; it shows that many people take their smartphones to interact before falling asleep or as soon as they wake up. Other data that contribute to this observation is that 24.6 % of interactions occur in environments with low luminosity and 29.6 % of interactions are in quiet places. This information can be useful for the launching of new applications for these specific situations.

Another fact that stands out refers to the interactions performed using own transportation (61.4 %). This percentage indicates that many people interact with applications when they are driving. A solution to this problem should be to use other modalities of interaction instead of touch-screen inputs.

4.4 Emotional State of the Participants During the Interactions

In Table 5, it is possible to visualize the emotional state of the participants during the interactions with applications. About 56.5 % of interactions occur with users when calm, happy or hopeful, while 43.5 % of the actions take place when the participants feel some kind of discomfort. However, no changes in user satisfaction were observed with respect to the applications according to their emotional state during interactions. In most cases, even when users have declared themselves tired, sick, sad, angry or furious, the level of satisfaction with the applications remained similar as when they are happy, calm or hopeful.

Table 5. Emotional state of the participants vs. the satisfaction level with the apps
Full size table

4.5 Satisfaction of Users with the Usability of the Applications

To evaluate the usability of the most popular applications, there was an investigation of nine attributes (efficiency, effectiveness, satisfaction, learning, operability, accessibility, flexibility, utility and ease of use). According to Kronbauer and Santos [13], these attributes are the most appropriate to measure the usability of mobile applications.

The data presented in Table 6 correspond to the arithmetic average of the participants’ satisfaction level in relation to the attributes of usability investigated, for the five most popular applications. It is possible to identify that most users are either satisfied or very satisfied with the usability of the applications. The only application that features the highest level of dissatisfaction is Contacts with 42.9 %. This is an indication that the application usability should be improved in new versions.

Table 6. Satisfaction level with the usability of applications
Full size table

4.6 Level of User Satisfaction with Their Devices Depending on Their Configuration

To perform this investigation, the screen resolution of the devices was related to the question that measures the level of user satisfaction with their devices. Table 7 presents the percentage of satisfaction based on screen resolution. The main finding is that about 30 % of the users that interact with low-resolution devices show themselves either dissatisfied or very dissatisfied with their devices. On the other hand, users who have devices with medium or high resolution tend to be satisfied or very satisfied, with no instances of dissatisfaction.

Table 7. Screen resolution x user satisfaction
Full size table

Kronbauer and Santos [12], when evaluating the usability of smartphones, identified that screens with high resolution and large size facilitate users’ interactions when compared to smartphones with smaller screens and lower resolution. Therefore, it is possible to understand why users feel more satisfied with larger devices and high resolution.

5 Related Works

Currently, the ESM technique is one of the most used to obtain subjective information about the feeling of users in relation to a particular subject. In order to collect data concerning mobile devices usage, some relevant works can be highlighted, as described below.

Ickin et al. [8] use the ESM technique for identifying users’ satisfaction regarding certain events and apply the technique for obtaining information on the context of user interactions, such as location, social context and level of mobility.

Hicks et al. [9] combine the display of messages with events captured by the smartphone sensors, for example, to go through some specific location (using GPS), when approaching a person from your circle of friends (using Bluetooth), or also according to a particular movement (using the accelerometer).

Lai et al. [10] developed an application for smartphones called Life360 in order to investigate the attitude and behavior of people on the interaction environment. The research objective is to propose a new approach for identifying the different lifestyles and personalities that characterize a certain population. The application displays questionnaires to users with a frequency from 8 to 12 daily interlocutions, collecting information related to current location of respondents, the activity in which they are involved, the people that are in the environment, the emotional state of the participants and the feeling regarding usability.

Another work that uses the ESM technique was proposed by Meschtscherjakov [11], called MAESTRO. The application was designed according to the client-server architecture and has as main objective to classify users, allowing to identify a pattern of behavior. This action allows you to send personalized questions after a certain period of the application utilization.

The infrastructure UXEProject [12] allows to analyze the usability level of application tasks for smartphones. The proposal is to map in an application the tasks that will be investigated and collect performance statistics data from the user when executing them. Moreover, the infrastructure supports the collection of contextual data through sensors of smartphones and subjective information with the ESM technique. In the specific case of the subjective data collection, the UXEProject only captures the feeling of the users in relation to the specific application being evaluated.

When conducting a comparative analysis of Sherlock platform in relation to the previously mentioned studies, it is possible to stand out as contributions: (i) the possibility to relate questions to all or single applications installed on a mobile device; (ii) the extent of approach, as it works with seven dimensions that may be correlated to evaluate the UX; (iii) the availability of a specific location for storage and evaluation of data; and (iv) the ease of use of the platform to perform the evaluations, since it is only necessary to install the SherlockApp on devices which will participate in an experiment.

6 Conclusions and Perspectives

This paper presents the components and associated services for a new platform with the capability to collect and evaluate data regarding the user experience with mobile applications. The main contribution of Sherlock platform is the use of the ESM technique for data collection, covering various dimensions. These data, when correlated, enable more detailed interpretations of the user experience about the different aspects associated with the dimensions.

From the data collected during the execution of an experiment to verify the potential of the platform the following conclusions were reached: (i) the users of mobile devices are likely to have attention deficit caused by social interactions; (ii) It was observed a tendency for people to interact with their smartphones before sleeping or after waking up; (iii) due to the large number of interactions in own transportation vehicles, it may be alerted that many people interact with their applications while they are driving; (iv) most of participants declared themselves either satisfied or very satisfied with the five most common applications; (v) the usability evaluation of the most popular applications indicates the need to rework on the Contacts application available from Android platform; (vi) it was noted that the low resolution of devices creates dissatisfaction among the users.

In general, it was possible to verify that Sherlock platform is appropriate for its goals. The Data Collection Unit, represented by SherlockApp application, enables to collect data in the seven dimensions proposed by the project, is easy to install and use. Moreover, it allows to relate the collected data with other applications running on the devices of experiment participants. The Data Correlation Unit, comprising a Web server, a DBMS and an OLAP tool, allows the storage and analysis of data in a fast and easy way.

The SherlockApp can be configured to generate questions in accordance with different events, such as applications, time, hours, events of Operating System and specific IP address. This flexibility allows the platform to be used to evaluate different types of products or services.

As prospects, it is intended to continue with the implementation of the experiment for a longer period and engage new participants. The objective is to build a database that encompasses a large number of users, making it possible to implement the second stage of the project that aimed to improve the experience of people in relation to urban mobility. Furthermore, we also intend to use the platform to evaluate products and services not specifically embedded in smartphones.