Keywords

1 Introduction

Many organizations have integrated User Experience (UX) processes into their formal software development model because UX processes help improving the user experience with their products and increasing organizational efficiencies [1]. User-Centred Design (UCD) is a methodological approach for product development which places users and their needs at the core of the development process [11]. It is by now generally accepted that UCD should integrate UX processes [15]. Moreover, organizations in sectors such as automotive industry, education, entertainment, health or retail could benefit from the integration of UX processes into their software development model in order to develop products that better fit user needs and expectations [24, 25].

This growing interest for UX has resulted into numerous ad-hoc approach variations for integrating UX into software development models [19], making it difficult or even impossible to assess the UX capability/maturity of an organization. The UX capability/maturity of an organization is the extent to which this organization consistently achieves UX processes. UX Capability/Maturity Models (UXCMMs) have generally been associated with the assessment of UX capability/maturity, as these models also provide support for increasing UX organizational efficiencies [12, 14]. However, the assessment of UX capability/maturity remains a challenge because of the lack of UXCMMs validated in practice, lack of empirical data demonstrating their benefit for organizations and lack of documentation or support for their use in practice [14].

Capability/Maturity Models (CMMs) include a Process Reference Model (PRM) and a Process Assessment Model (PAM). A PRM describes a set of processes and their interrelations within a process lifecycle, whereas a PAM is a measurement structure for the assessment of the capability or performance of organizations to implement processes [9]. Following Paulk’s pioneer model [20], CMMs typically include five maturity levels: initial (level 1), repeatable (level 2), defined (level 3), managed (level 4) and optimized (level 5). The purpose of CMMs is to support organizations moving from lower to higher maturity levels. In a CMM, both base practices and work products serve as indicators of the capability/maturity of processes.

Our previous work [13] focuses on the specification of a UXPRM describing primary UX lifecycle processes and a set of supporting UX methods and artifacts. In this paper, we present a UX Process Assessment Model (UXPAM) for measuring the UX capability/maturity of organizations. The proposed UXPAM consists of a capability scale, a maturity scale, a rating scale and a set of process attributes as measurable indicators of the achievement of UX processes. In particular, we use UX artifacts, methods and resources as indicators as we argue that UX artifacts, methods and resources contribute to the execution of UX processes as follows: the production of UX artifacts demonstrates the execution of UX processes, whereas the use of UX methods demonstrates the implementation of specific UX processes with specific UX resources. The contribution of this paper is threefold:

  1. 1.

    The description of a complete measurement structure for assessing UX capability/maturity in organizations;

  2. 2.

    A questionnaire that serves as tool for measuring UX capability/maturity in organizations by means of survey research or interview;

  3. 3.

    The documentation of a case study demonstrating the efficiency of the proposed UXPAM in a project conducted with four industrial partners.

2 Background

2.1 Capability/Maturity Model

CMMs address both generic and specific contexts such as Usability Engineering (UE) [5], healthcare [24] and software development [20, 23]. They were introduced to evaluate an organization’s maturity and capability. The maturity refers to the ability to consistently implement processes, while the capability of processes refers to the ability to achieve the required goals of a process [3, 14]. CMMs describe the organization’s evolution from a less ordered to a more structured state. The maturity of an organization can be assessed by measuring the capability of processes [14]. In addition, CMMs give recommendations for improving processes by adopting base practices and help organizations to achieve their business goals. Only a few evaluations of CMMs have been published because they are resource-intensive. Goldenson and Gibson [7] document 12 case studies that show how the implementation of the Capability Maturity Model Integration (CMMI) framework [23] increases the performance of organizational processes (e.g. reduction of costs and delays) and brings benefits such as increased customer satisfaction. Typically, CMMs include a PRM and a PAM.

2.2 Process Reference Model

PRMs describe a set of processes and their interrelations in a process lifecycle [9, 10, 14]. Additionally, they describe the objectives, expected outcomes and related work products that demonstrate the execution of a specific process. Usually, PRMs are refined into base practices that contribute to the production of a work product. In turn, PRMs define a set of process-related indicators to evaluate for assessing the capability/maturity of an organization. Indicators allow the comparison between current and desired capability of processes [14].

2.3 Process Assessment Model

PAMs are measurement structures intended for measuring the capability of processes, more specifically for assessing how well organizations comply with a prescribed PRM [8]. Major CMMs such as Paulk’s CMM [20] or CMMI-DEV [23] comply with the principle of defining a measurement structure in CMMs, which is also recommended by ISO 15504 [9]. The measurement structure contains three elements: process attributes, a capability/maturity scale and a rating scale.

Process attributes are measurable characteristics of a process which are directly related to its maturity and/or capability [5, 14, 23]. A maturity scale is a scale that describes Maturity Levels (ML) where each level builds on top of the level below and includes a set of characteristics that reflect the level of UX maturity [24]. A capability scale contains Capability Levels (CL) and represents an evolutionary order of stages to measure the achievement of processes. A rating scale is an ordinal scale used to rate the capability of processes. Typically, MLs are expressed as a set of Process Areas (PA) that have achieved a certain CL, whereas CLs relate to the capability of individual PAs or even processes [14, 23]. A PA is a group of related activities that together contribute to the achievement of a common goal [14, 23]. The structure of a PAM is shown in Fig. 1.

Fig. 1.
figure 1

A structure of a Process Assessment Model

Full size image

3 Related Work

3.1 UX Capability/Maturity Model (UXCMM)

A Systematic Literature Review (SLR) reported in [14] identifies 15 CMMs which focus on UX and usability. The authors raise the question of the validity of generic CMMs when applied to specific domains such as UX and usability by making the following statements about the identified CMMs:

  • They do not describe in sufficient details the methodology that was adopted to develop them;

  • Only a third of them (5 out of 15) includes a UXPRM;

  • Although they all include a UXPAM, they do not provide any method or tool to perform the process assessment, which in turn decreases the possibility of their adoption in practice;

  • Their validation remains questionable due to a lack of scientific rigor of their validation.

The relevant literature often refers to the following CMMs: Schaffer’s usability CMM [22] including six MLs but which has not been validated [24]; Nielsen’s usability/UX CMM [17, 18] including 8 MLs, which makes it difficult to distinguish between levels [14]; Earthy’s usability CMM [4] including 6 MLs together with detailed documentation about its application.

The Healthcare Information and Management Systems Society (HIMSS) proposes a Healthcare Usability Maturity Model (UMM) in order to increase the awareness about usability in healthcare organizations. The paper describes wake-up calls such as failed products, competitive pressure or important needs for increasing patient’s safety that serve as triggers for the adoption of UCD. The model includes 5 MLs described across five dimensions: focus on users, management, process & infrastructure, resources and education. Additionally, it gives recommendations for transitions from lower to higher level of maturity.

Table 1. Supporting UX methods
Full size table

3.2 UX Process Reference Model (UXPRM)

Previous work [13] specifies a UXPRM including a complete description of primary UX lifecycle processes and a set of supporting UX methods and artifacts. The UX lifecycle involves four processes (analysis, design, formative and summative evaluation). Supporting UX methods (Table 1) include knowledge elicitation methods without users and knowledge elicitation methods either focused on users’ attitude or on users’ behavior. UX artifacts (Table 2), also referred to as artifact-mediated communication methods or work products, are means to increase communication and facilitate collaboration between distinct development teams [6]. For example, coded-prototypes, personas and user stories facilitate the communication between agile and UCD teams [6], while paper prototypes are most frequently used artifacts when performing usability inspection [2].

Table 2. Supporting UX artifacts
Full size table

3.3 UX Process Assessment Model (UXPAM)

The UXCMM presented in [21] includes an assessment tool based on a survey of UX professionals about their use of existing CMMs. The paper identifies the integration of UX within the organization, the UX budget, the researcher-designer-developer ratio, the UX buy-in throughout organization and the frequency of UX evaluations as key indicators of UX capability/maturity. In a similar way, Jokela and Abrahamsson [12] suggest three dimensions of usability capability, which are also applicable to UX capability: infrastructure to implement UCD in projects (e.g. prototyping tools or usability lab), efficiency, effectiveness and quality of the planning and implementation of UCD, and the commitment of the management to support UCD.

4 Proposed UX Process Assessment Model (UXPAM)

4.1 Model Development Method

Maturity models have to be developed systematically, mention the sources used and have to be developed to address a specific domain or have a more general scope. We decided to adopt the maturity model development methodology by de Bruin [3]. This methodology has six phases, whose order is important to follow, and decisions made in each phase have an impact on the later phases. Namely, the phases are Scope, Design, Populate, Test, Deploy and Maintain. Also, the development methodology supports iterative design, especially in phases such as Populate, Design and Test. In this paper, we present the work we did in the first four phases.

4.2 Scope Phase

In the Scope phase, we determined the focus of our CMM and its domain. This allowed us to set its target definition and characterize the difference between existing models. As a result, we aimed at developing a domain-specific CMM, whose stakeholders are academics and practitioners [3], to support the integration of UX processes in software development models. Our model aims to help organizations assess the capabilities of their UX processes through a set of process attributes. Moreover, the model provides a basis for the UX process improvement. First, we used the available literature to identify the relevant domain issues and the shortcomings of the existing models. Our model documents its development methodology, provides an assessment tool and covers non-process elements such as resources, culture and management attitude [14].

4.3 Design Phase

In the Design phase, we made decisions related to the needs of our model’s target audience as well as decisions related to the model’s architecture. According to the SLR [14], most reviewed models use the 6-level capability scale adopted from ISO 15504 [9], whereas the maturity scales range between three and six levels. Capability levels are shown in Table 3. Regarding maturity scales, we opted for the most common one, a 5-level scale as it is detailed enough to describe the evolutionary stages of organization’s maturity, and simple enough to distinguish the differences between levels. Our maturity levels are shown in Table 4. Also, having in mind the audience that will typically perform the self-assessments, we chose to develop a questionnaire as an effective assessment tool. Respondents to this questionnaire will mostly consist of management and employees of the assessed organization, including UX staff, developers, researchers and managers.

Table 3. Capability levels description from ISO 15504 [9]
Full size table
Table 4. Maturity levels
Full size table

4.4 Populate Phase

In the Populate phase, we extracted the relevant domain components of our UX PAM from the literature, that are mutually exclusive and collectively exhaustive. This allowed us to identify which process attributes are good candidates for the measurement of capability/maturity. A core part of the PAM are the PAs. We clustered process attributes into PAs, depending on their relatedness. Based on relevant literature [12, 16, 24, 25], we have realized the importance of UX resources, culture and literacy as fundamental for the UX process improvement. Besides, according to an in-depth analysis reported in [11], these non-process elements were regularly excluded from CMMs focused on UCD. However, according to [14], recent CMMs seem to put more emphasis on the management issues, management of UX/usability resources, processes and their integration. They serve as indicators of the effectiveness of UX processes [11, 12] and enable the measurement of UX capability and maturity. Table 5 shows PAs and process attributes we use in our PAM. The structure of our PAM is depicted in Fig. 1.

4.5 Test Phase

In the Test phase, we conducted a survey among our partners in the industry. The survey consisted of an online questionnaire and remote interviews. We interviewed some participants to cross-check their survey answers and to gather information related to their UX literacy and the ways they think UX can facilitate the development of their products. Specifically, we checked their understanding of the UX Return On Investment (ROI) and their organization’s attitude towards UX discipline. ROI is demonstrated through increased organizational efficiencies, reduced development time and costs or reduced need for technical support [1].

Table 5. Process areas and process attributes
Full size table

4.6 The UXCM Questionnaire

The proposed questionnaire is divided into five blocks. The first three blocks focus on the frequency of use of UX artifact and UX methods and the availability of UX resources. The two remaining blocks focus on UX literacy and UX culture. These blocks correspond directly to the indicators of UX process capability described previously in the paper. The blocks 1–3 are shown in Table 6 and blocks 4–5 are presented in Table 7.

Table 6. Questionnaire for the UXCM assessment - part 1
Full size table
Table 7. Questionnaire for the UXCM assessment - part 2
Full size table

5 Preliminary Questionnaire Validation Results

We performed the initial validation study of our questionnaire. We used the questionnaire presented in Tables 6 and 7 to perform a two-round assessment. In total, four companies were involved in the survey. We distributed it to our industrial partners working on a project in automotive sector. Our role in the project is to provide support for the integration of UX activities and lead the UX-related tasks.

5.1 First Round

In the first round, we conducted an online survey containing questions from Table 6, that had been opened to participants to respond for two weeks. Additionally, the online survey contained a section related to participants’ personal information, asking them to provide their job title or their function, their highest educational degree level, and the title of their degree (i.e. field of education). The goal of a survey was to assess the UX capabilities of UX processes. We used the project’s mailing list to reach the participants. Altogether, 20 respondents took the survey and fully completed it, 4 participants only partially submitted their answers, and 10 did not enter any data but only opened the questionnaire’s homepage. Thus, we had a response rate of 57%, based on 35 invitations sent. Participants mostly work in business, management, engineering and telecommunications fields, but their roles are broad, ranging from managers and directors to researchers and engineers. None of them had an educational background in HCI. Figure 2 shows capability scores for each partner company in terms of the two UX capability indicators, UX artifacts and methods, and their overall capability score.

Fig. 2.
figure 2

Capability scores per company

Full size image

5.2 Second Round

In the second round, we have conducted phone interviews with 6 participants, in order to find out more about their organizational practices and debrief their survey answers. We used the set of questions shown in Table 7. The findings allowed us on one hand to assess the UX literacy and UX culture of our partners, and on the other to double-check their answers with the goal to validate the measurement tool, i.e., the questionnaire. During interviews, we described each artifact and method to interviewees and allowed them to change their answers. This is shown in Table 8. We noticed a small number of changed answers, mostly at the degree of change equal to one, and twice at the degree of two. This suggests that our questionnaire is robust enough to support the assessment of UX capabilities and that participants’ answers are reliable. Most commonly, answer changes resulted from participants’ misunderstanding of what a method/artifacts is.

Table 8. Round 1 vs Round 2 differences (blanks mean “I don’t know”)
Full size table

6 Data Analysis

We analyzed the data collected from the “UX Capabilities Survey”. We performed this assessment at the beginning of the project to better understand how the UX is perceived and understood by our partners and to obtain the overview of their capabilities which will, in turn, enable us to better plan UX activities.

6.1 Resources

UX resources represent the necessary infrastructure for the organization to effectively plan and implement UX processes in their development projects. It is essential to have the UX staff with the right amount of UX skills who will perform the UX activities. The Designer:Developer (D:D) ratio is a good indicator of the potential capability to perform UX activities. We present D:D ratio as a range between best and worst-case scenario given that the provided number of UX staff and developers varied among respondents. We could only calculate it for P1, and it ranges between 1:4 and 1:40. It is clear that the resources are scarce, UX staff exists in only one company (P1) and there are no dedicated labs or tools to perform UX tests. Therefore the resources can hardly support the effective UX work. Figure 3 shows who is performing UX work in each company.

Fig. 3.
figure 3

Distribution of UX activities by role

Full size image

6.2 Methods and Artifacts

Tables 9 and 10 present results from the first round of the survey, blocks 1 and 2, respectively. The heat map clearly shows the most used methods and artifacts. Blanks represent the answer “I don’t know this method/artifact”. Zeros indicate the answer “I don’t know how often”. In the last column an average capability score for each method and artifact is given. Numbers in the table correspond to the answers on the rating scale: I don’t know how often (0), never (1), rarely (2), sometimes (3), often (4) and always (5).

Table 9. UX artifacts (blanks mean “I don’t know”)
Full size table
Table 10. UX methods (blanks mean “I don’t know”)
Full size table

6.3 Literacy

Knowledge of the UX artifacts and UX methods is considered essential in order to properly plan and perform UX activities. UX literacy refers to the knowledge and skills of an organization regarding UX. We decided to put the “I don’t know this method/artifact” answer in our questionnaire so that we can properly distinguish the knowledge of a particular method or artifact and its unknown frequency of use from being unfamiliar with it. The latter feeds into the UX literacy dimension and therefore allows us to calculate the literacy scores, based on participants’ answers. Last two rows in Table 11 present the literacy percentage scores for UX methods and UX artifacts per company. They are calculated as an average of percentage shares between two categories: a sum of all answers on the frequency rating scale and all “I don’t know this method/artifacts” answers for each method/artifact, where higher means better. Furthermore, our model addresses the understanding of UX ROI, attitude towards users and UX training as fundamental to UX literacy. In the second round, we asked participants the questions from Block 4. Results are presented in Table 11.

Table 11. UX literacy assessment
Full size table

6.4 UX Culture

This is a summary of questions from Block 5. Generally, all partners see the value UX could bring to their projects, but fail to integrate it properly into their software development models. They generally think UX comes as an additional work which might generate delays in development. The positive side is that they understand that UX processes should be employed from the beginning of the development process. From the management’s perspective, UX is only slightly supported and they don’t seem to be much aware of it. For employees, on the other hand, UX is either not a concern, or they see it as beneficial to the development of high-quality products that meet user needs. P2 agrees that UX could be a way to achieve business goals, whereas P1 is not sure how can UX be offered from the business perspective.

6.5 Summary

Table 12 shows the summary of UX process capability by PA. Overall, the highest capability was measured in Product Development, User Research and Contextual Design in descending order. First one does not come as a surprise because high-fidelity prototypes closely resemble the finished product and summative evaluation can be easily performed on them. The second and the third PAs are closely related because artifacts describing user needs and the context of use often stem from the user research activities (interviews, surveys, personas creation). On the other hand, partners overall score is low in Stakeholders and Experts Involvement, and Monitoring of UX PAs. Again, for these to score better, UX should be more deeply integrated into their organizations and UX culture established. Finally, the Score column indicates the overall capability score for each partner across PAs and it could be considered an initial overall maturity score.

Table 12. Process areas assessment per partner company (red = 1 SD below average; orange = 1 SD above average), M = 1.71, SD = 0.5498
Full size table

6.6 Discussion

Conducting the case study allowed us to learn the following:

Use of UX Methods. Most often used UX methods are those that require less effort and resources to be implemented such as literature review or user interviews. Most used artifacts are user stories, user scenarios and personas. We noticed a lack of formative UX evaluations, but none of the companies have dedicated UX teams that could work in parallel with development teams. This is indicated with the high tendency to use high-fidelity prototypes for evaluation. Furthermore, during interviews in second round, we noticed that participants did not always understand the correct meaning of UX artifacts or methods, which led to change of answers in the second round. This suggests that further refinement is needed to remove any eventual ambiguity.

Identify Barriers. The findings allowed us to improve our UX activity plan and to anticipate bottlenecks in product development, such as lack of motivation for user requirements analysis (product-oriented versus user-centered mindset), which demonstrates a low level of UX capability. This does not come as a surprise as they have expressed the need for UX expertise in their project, but have claimed that they already do some UX work.

Insights. P2 consistently performed best among four companies. Surprisingly, it is a technologically-oriented and research-driven company, but they showed knowledge of a majority of UX methods. As a matter of fact, their results demonstrate the highest capability scores in all dimensions. Their perceived value of UX was made clear during interviews where all interviewees had a similar vision of the benefits of UX as well as confirmed the regular implementation of UX activities when the project demands it. P1, as a main project stakeholder, scored less than expected, despite their high interest in the implementation of UX processes. Other two companies P3 and P4 scored relatively low, due to their focus on technological development and delivering software to other providers. Generally, they don’t have the connection to end-users.

7 Conclusion

The assessment of UXCM is a prerequisite to the UX process improvement. To bridge this gap, this paper presented a UXPAM accompanied with an efficient method to assess UXCM, using a questionnaire as a data collection method supporting the use of online surveys and remote interviews. Building on top of previous studies, we assess the UXCM across five UX dimensions: artifacts, methods, resources, literacy and culture. We performed an initial case study demonstrating the use of our UXPAM. The results seem to accurately capture the current UX capability of an organization. However, further work is required to provide the assessment feedback in a more systematic way.

8 Future Work

Future research will further evaluate and validate our UXPAM. Concretely, we aim to conduct a longitudinal study to test its usefulness and ability to improve the UX processes in an organization. We will implement the phases five and six of de Bruin’s [3] methodology, Deploy and Maintain. Additionally, we intend to merge our UX PAM with the UXPRM into one UX Capability Maturity Model (UXCMM). To achieve this, we need to establish a mapping between the PAM and the PRM, as required per ISO 15504. In addition, more empirical analysis is needed to establish the relation between CLs that each PA must meet to achieve a certain ML. We also aim to provide a structured approach toward the increase of organizational maturity and deliver empirical results demonstrating the model’s applicability in practice. Finally, we intend to allow practitioners to make justified choices when selecting the UX methods and provide support to successfully integrate UX activities in the software development model.