Keywords

1 Introduction

Design Science (DS) is a research paradigm which aims at creating and evaluating innovative artifacts that address important and relevant organizational problems. Hevner et al. strongly promoted the design science approach in the field of information systems in their book from 2004 [17]. They pointed out that design science along with behavioral science were two key paradigms used in information systems research. According to their words, design science aims “to extend the boundaries of human and organizational capabilities by creating new and innovative artifacts.” A framework of seven guidelines was created by the same authors [17], focusing a design as an artifact, problem relevance, design evaluation, research contributions, research rigor, design as a search process and communication of research.

This framework was adopted and adapted by different authors but Peffers et al. [35] from 2007 made the biggest impact with their definition of Design Science Research Methodology (DSRM) which incorporates principles, practices and procedures required to perform DS research. This methodology included six steps (see Fig. 1) including problem identification and motivation, definition of objectives for a solution, design and development of an artifact, demonstration, evaluation and communication.

Fig. 1.
figure 1

adopted from Peffers [35] and adapted by Dalen and Kraemer [9]

Design science process model

Full size image

Although the process is structured in a nominally sequential order – in a problem-centered approach, researchers could take different entry points and move outwards, depending on their approach: an objective-centered approach, a design- and development-centered approach or context-initiated approach [35]. For this research we are focusing mainly on a design- and development-centered approach which starts from an artifact which is later proved to be solution for the domain where it will be used. This approach puts special focus on demonstration and evaluation activities which become of special interest for our research as well.

On the other hand, in recent years, design science has gained interest in HCI research community as well, as an applicable research framework for design and evaluation of user experience constructs. User experience (UX), the term coined by Don Norman, is a broad concept explained by Norman itself, as a way a person experiences reality around him/her, e.g. the way he/she experiences the world, a life, a service, an application or a product [28]. More precisely, but still on the high level of abstraction, it refers to every aspects of the user’s interaction with the company, its services, and its products [29, 56]. Formalization of the UX term is given by the ISO standard 9241-110:2010 as “a person’s perceptions and responses resulting from the use and/or anticipated use of an interactive system, and from the user’s interaction with the organization that supplies or delivers the interactive system; from discovering the system, adopting and using it, through to final use” [19], which emphasizes experiences created before, during or after interaction with the system, service or a product and also user’s relationship to organization which provide the product.

While there is no consensus on the definition of user experience [16], researchers agree that UX encompasses aspects that go beyond usability and user interface design. Usability is related to the user’s accomplishment of the task while interacting with the products, systems or services, while user experience is related to hedonic aspects of that interaction and possession, such as beauty, challenge, stimulation, or self-expression [13]. User experience is subjective category that can change over time and according to the user’ internal state. User’s state is also underpinned by the definition of Hassenzahl and Tractinsky [14] who define UX as “a consequence of a user’s internal state (predispositions, expectations, needs, motivation, mood, etc.), the characteristics of the designed system (e.g. complexity, purpose, usability, functionality, etc.) and the context (or the environment) within which the interaction occurs (e.g. organizational/social setting, meaningfulness of the activity, voluntariness of use, etc.).” Those dimensions, a user, the system and the context, are considered the building blocks of UX.

Djamasbi et al. argue that the user experience and design science research paradigms have much in common; they both contribute to information systems research by providing guidelines for designing successful information technology (IT) systems [10]. The same authors also state that each has weaknesses that could benefit from the strengths of the other, arguing that UX research could benefit from the formal structure of DSR (e.g., the mentioned framework) to better communicate its findings and contribution to theory and practice, while DS could benefit from UX principles that provide specific guidelines, practices, and metrics for measuring the development progress of IT systems designed for a variety of users [10].

To further define our research playground, we must define the key aspect of design science paradigm – an artifact. An artifact may be defined as “an object that has been intentionally made or produced for a certain purpose” [18] or it may refer to “one of many kinds of tangible byproduct produced during the development of software” [34]. Also, in the context of software development artifact could be defined as “any piece of software (i.e. models/descriptions) developed and used during software development and maintenance” [8]. Finally, in the context of design science and human computer interaction, we could adopt and adapt the definition from [55] which points out that the term artifact denotes any outcome of the activities in a design or an UX design process.

Thus, taking all into consideration, two important questions within the UX field arise: how to design the system that would evoke positive experience in the user within the given context and how to evaluate the created artifacts to make sure that user’s positive experience is achieved. In the context of UX design, various UX design processes could be used, and in the context of UX evaluation, there are many evaluation techniques that could be used.

The goal of our research is to investigate and systematically map available evidence on the use of design science in creation and evaluation of UX artifacts and to determine what type of UX artifact evaluation techniques are used by researchers and practitioners.

The rest of this paper is structured as follows: in the second chapter we describe the research approach and methodology of systematic mapping study that we performed in our research. In the third section we present the results of the study along with the systematic map of found scientific evidence. In the last chapter we bring the most important conclusions and wrap the results presented in this paper.

2 Research Approach

2.1 Systematic Mapping Study

A scientific method of analysing, identifying and structuring all available research evidence and results on a topic of research interest or on a whole research area is called a systematic mapping study (SMS) [3, 37]. As well as a systematic literature review (SLR) [48], SMS also requires the activities of research planning and questions definition, objective setting, rigour search strategies definition etc., but SMS in general has a goal of providing a more coarse-grained overview of the topic, and thus requires less effort to be performed [37]. Petersen et al. [36] state that SMS provides a valuable baseline for subsequent research of a topic. Additionally, mentioned authors state that SMS could save time in subsequent studies, if performed well it presents a solid overview of the researched area, gives very good visualization of research trends, gaps and related work trends etc.

In the context of user experience, a lot of reviews and several SMS’s have been performed mostly with the focus on UX evaluation methods. Systematic mapping study performed by Rivero and Conte [39] aimed to identify technologies (methods/techniques/tools/others) that have been proposed for the evaluation of user experience in the development of applications and how have these methods been used during period of 2010–2015. Their study revealed the need for new UX evaluation methods, e.g. the one that takes into account both qualitative and quantitative data, the one that suggest improvements of the software once a problem is found etc. Nakamura et al. performed SMS to found out which usability and UX evaluation techniques were applied on Learning Management Systems and how have they been used? [25]. Their study showed that inquiry type of evaluation techniques (questionnaires, focus groups and interviews) were the most common ones, while indicating several gaps, e.g. the need for techniques that cover both usability and UX aspects, the need for techniques that suggest improvements of the LMS etc. Subsequently, they have proposed new technique to evaluate UX in e-learning by applying Design Science Research methodology [24].

Typically, SMS is performed through three main phases: planning, conducting and reporting on the mapping.

2.2 Planning the Study

Planning of the study puts the basis for the rest of the scientific activities. During this phase, the objectives and the research questions are defined. Our main research question was: “How is design science (DS) being used in creating and evaluating UX artefacts?”. Although the research questions in a systematic mapping could be less specific than in a systematic literature review [36], the search strategy and inclusion and exclusion criteria should be well defined. Choosing a search strategy is required in order to determine the way to find information and studies of an area. In our case, a search string was composed of three groups joined by Boolean AND: keywords related to design science, keywords related to UX and usability as UX subset, and keywords related to evaluation and design including alternative spellings and synonyms of those terms. Manual search was planned to be performed on major databases by executing the previously defined and tested search string. Inclusion and exclusion criteria must be defined in order to simplify the process of filtering the search results. Final step in this phase was to define extracting data strategy and to establish a classification scheme. Extracting data strategy defines what data is needed in a study to enable researchers to classify it. Depending on the field being observed, one can use an existing classification found in field literature or apply a new one derived from the search. We planned to use the existing well-known classification scheme of UX evaluation techniques enhanced by our own keyword extraction. The results of each of these steps are as presented in the Table 1.

Table 1. Blueprint for the systematic mapping study
Full size table

2.3 Conducting Systematic Mapping Study

By following the previously defined plan, we implemented mentioned strategies and criteria to find, select, classify and map the findings. Petersen et al. [36] note that this process can be iterative and require revisions, and also encourage researchers to document every step in the process since it could be very helpful in subsequent iterations. Thus, as presented in Fig. 2, our process contained several iterations that are presented below.

Fig. 2.
figure 2

Systematic mapping process

Full size image

A search criterion identified a total of 253 sources from the above-mentioned databases. The metadata of obtained papers was exported from the original databases and imported to reference management software which was used by the researchers to maintain the papers during the whole systematic mapping process.

The first iteration was to apply inclusion and exclusion criteria just by reading a title and the abstract of the studies. The reference management tool helped researchers to identify duplicated entries and those entries that were not referring to research results (e.g. the names of events) which were excluded in this phase leaving 142 papers to be included in the analysis. In this process we applied and open-world-principle (OWP), meaning that all papers were included and only those sources that were undoubtedly not passing inclusion criteria where eliminated. This iteration ended up with a total of 77 candidate papers.

The most important phase of the systematic mapping research was to apply inclusion criteria on 77 candidate papers and to identify those that are related to our research questions. In this process we used closed-world-principle approach, meaning that none of the papers were included by default, but only after they passed inclusion and exclusion criteria, and at this point, the content of the paper was taken into consideration as well. Additionally, during this process, keywords relevant to the research question were identified and extracted.

Finally, a total of 43 papers was included and used for systematization and map creation in the last phase of the process.

2.4 Reporting

Mapping report summarizes what was done in previous steps by visualizing the mapping results and using all information collected in the mapping conduction to highlight the findings. The 43 papers are classified into 5 categories related to method and techniques of evaluation of UX artefacts during the design science process. The classification is presented in results section.

3 Results of the Study

A total of 43 papers passed our inclusion and exclusion criteria. These papers are identified as related to our research question and are describing the use of design science research to develop and evaluate UX related artefacts. Due to inconsistent structure of the papers, along with the use of different naming approaches, it turn out that classification and keyword extraction were not a trivial tasks. The analysis of keywords and focused topics in the papers also proved that the research covered rather different and disjunct topics. The studies were classified into five categories related to UX evaluation techniques, and 32 different keywords related to the topic were identified (see Fig. 3).

Fig. 3.
figure 3

Studies classification map

Full size image

Most of the papers (21) reported the use of different types of questionnaires to evaluate created artefacts. These approaches included the use of SUS, USE, TAM, UTAUT and different surveys. The keyword analysis showed that the keyword questionnaire was not in the focus of all of these papers as they used different specific titles as presented above.

The second most covered topic (14 papers) was related to performance of usability/user testing including other related techniques to evaluate the artefacts. This group included the techniques such as thinking aloud, eyetracking and walkthrough.

The use of interviews and related techniques was reported in 11 studies. This group also included the papers mentioning the use of feedbacks, focus groups feedback, commenting, description etc. Finally, different inspection methods were reported in 7 studies: heuristic evaluation, design review and checklist analysis. Last, but not least, field or laboratory observation were reported in the 6 studies.

Some studies reported the use of more than one technique. The detailed list of analysed papers and their relations to the identified topics are presented in the Table 2.

Table 2. Systematization of available research
Full size table

The keyword analysis performed during the last phase of systematic mapping process resulted in more than 40 different keywords. The list of most common used keywords to explain the UX evaluation techniques and approaches is presented in Fig. 3. Majority of the keyword were used very rarely while just a few keywords were mentioned more than three times: usability testing, interview, focus group, usability commenting and questionnaire.

4 Conclusions

In this paper we reported the results of the systematic mapping study performed on research evidence related to the use of design science research methodology in creation and evaluation of artifacts related to user experience design. The study analyses more than 250 papers and upon applying inclusion and exclusion criteria 43 papers were included in the results. Those papers were classified into five main classes related to the use of UX evaluation techniques and it turned out that most of the researchers (21 papers) are using different types of questionnaires to evaluate created artefacts. These approaches included the use of SUS, USE, TAM, UTAUT and different surveys. The second most covered topic (14 papers) was related to performance of usability/user testing including other related techniques to evaluate the artefacts. This group included the techniques such as thinking aloud, eyetracking and walkthrough. Other UX techniques were also included but in much lower intensity.

Apart from performing predefined classification, we also performed a keyword analysis and found out that researchers are using more than 30 different keywords which are related to UX design and evaluation techniques.

This research represents a solid ground for additional exploration of the field in terms of fine-grained analysis of the performed processes of DS in general and of the UX evaluation activities.