Keywords

1 Introduction

Open source software (OSS) has spread so swiftly that it now rivals commercial software systems [1]. OSS communities do not as yet enact standard processes capable of ensuring, bearing in mind the characteristics of the OSS community as a whole, that the products that they develop have the attributes of good software [2]. An inadequate definition of processes, activities, tasks and techniques within OSS development has led researchers from several areas to gravitate towards this field of research with the aim of correcting this situation [3,4,5]. The growth in the number of non-developer open source software (OSS) application users and the escalating use of these applications have created a need for and interest in developing usable OSS [6,7,8,9,10].

Usability is one of the key quality attributes in software development [11]. In recent years, OSS has come to be an important part of computing [12,13,14,15]. However, several authors have acknowledged that the usability of OSS is poor [6, 16, 17]. In this respect, the empirical study conducted by Raza et al. [7] reports that 60 per cent of respondents (non-developer users) stated that poor usability is the main obstacle to be overcome by OSS applications if users are to migrate away from commercial software. On this ground, OSS projects must tackle their level of usability and usability-related problems more conscientiously [17].

On one hand, the human-computer interaction (HCI) field offers usability techniques whose key aim is to build usable software. However, they are applied as part of HCI methods and not within the OSS development process. On the other hand, the OSS development process focuses on source code and thus on feature development. The OSS development process has a number of characteristics (for example, developers and users are usually the same person). This prevents many of the HCI usability techniques from being adopted directly [18].

Even so, the OSS community has now started to adopt some usability techniques. Most of the techniques that the OSS community has taken on board are for evaluating usability [18]. Some usability techniques have been adapted ad hoc for adoption in OSS development projects [18]. This paper addresses the research problem of how to adopt the focus groups usability technique for requirements engineering activities as part of the development process of a real OSS project known as ERMasterFootnote 1. To do this, we first identified and analysed which obstacles had to be overcome in order to apply focus groups in OSS projects.

Ferré [19] compiled a list of usability techniques recognized by HCI. He determined the most representative HCI process activities: use context specification, usability specifications, product concept development, prototyping, interaction design and usability evaluation. He then mapped these activities (and each of their associated techniques) to software engineering (SE) development stages: requirements engineering, design and evaluation.

Our research spans two areas: SE and HCI. We use usability techniques as a bridge to communicate these two areas, where our aim is to deploy HCI knowledge in the SE field and especially in the OSS development process. If adapted, usability techniques can be adopted in the OSS development process [18]. Therefore, this paper has two goals. Firstly, we intend to adapt the focus groups usability technique [20] for adoption in the OSS development process. Secondly, we aim to determine the feasibility of adopting this usability technique in a real OSS project.

Requirements engineering activities play a very important role in the success or failure of an OSS project. However, they are sometimes extremely hard to perform because there is no definition of OSS user segments before the software is developed. Also, it is far from straightforward to address all the requirements analysis activities due to the particular characteristics of OSS development groups (for example, global geographic distribution of user sites or code-focused world view). On this ground, this paper considers just the product concept development activity. Additionally, OSS projects have not adopted many usability techniques related to the requirements engineering and product concept development activities [18]. The next step after selecting the activity is to pick a related usability technique for adoption in the OSS development process.

The main reasons for the generally poor usability of OSS developments are: OSS developers have tended to develop software for themselves [4, 10] and the developer community is very much in the dark about who its users are [9, 16]. The aim of the focus groups technique is to gather information related to user opinions, problems and concerns at meetings scheduled for this purpose [18,19,20]. This technique helps to focus product concept design on its hypothetical functionality [20]. The focus groups technique requires a small research sample for the purposes of product evaluation. Consequently, the participation of just a few users is sufficient to represent the product concept model, that is, developers use this technique to discover a user’s mental model of the product. On this ground, we selected the focus groups technique for adoption in an OSS project.

This paper makes a significant contribution to the field of SE and particularly to OSS development projects because we have not been able to identify papers reporting the use of the focus groups technique and detailing how it has been applied in OSS development projects [21, 22].

There are several OSS project repositories. One of the most popular is SourceForge.net [23]. This repository classifies OSS projects by categories. Since this technique is related to requirements engineering for product concept development, we looked at projects with a low level of coding (that is, projects where key features were still being added) that were not overly ambitious and were at the very early development stages (alpha version) in order to select a suitable OSS project in which to adopt the selected usability technique. Considering the above, we selected the ERMaster OSS project. Thanks to the characteristics of this project, we can adopt a usability technique related to a requirements activity (product concept development). Therefore, the benefits of applying the technique will have a bigger impact on the development process and software system usability. We have adapted the technique based on the integration framework proposed by Castro [18].

We used a case study as the research method to test the feasibility of our proposal for adopting usability techniques in OSS projects [24]. Consequently, we had to volunteer for the selected OSS project and join the community.

This paper is organized as follows. Section 2 describes the characteristics of the OSS projects. Section 3 illustrates the research method followed to apply the usability technique in an OSS project. Section 4 outlines the state of the art. Section 5 reports the proposed solution. Section 6 discusses the results. Finally, Sect. 7 outlines the conclusions and future research.

2 OSS Project Characteristics

OSS applications are typically built by a group of independent developers and volunteers distributed all over the world [18]. The OSS community uses different web artefacts to communicate and synchronize its development practices (for example, email lists, Internet Relay Chat (IRC), source code repositories and bug reporting systems). Text is the OSS community’s primary means of communication, as well as the main object of interest (specifically, source code). Developers mainly contribute to OSS communities by developing new features and fixing bugs reported by users [25].

At the beginning of the OSS movement, application developers and users were one and the same people. As the years have gone by and the popularity of OSS has grown, the user profile has changed. Today, there are basically two groups of OSS users. On one hand, we have users that are computer literate, experienced software users or are very interested in anything technology related. This is the group of users that are more often in contact with the principal developers and provide the best feedback. On the other hand, we have users in the strict sense. This group of users have switched over to and use these applications at their workplace [18].

Even though OSS community developers and users are geographically distributed, there are meeting points, like conferences or workshops, which are generally sponsored by companies. These events are organized with the aim of officially releasing new versions of the applications and offering tutorials and workshops where developers and users can exchange opinions. For example, developers and users participated in a meeting held in Madrid (Spain) in June 2013 to present BonitaSoft 6.0 (a business process management application).

A few OSS projects are sponsored by companies, and thus have the resources that they need to apply usability techniques as prescribed by HCI (for example, usability experts and usability laboratories). For example, the usability specifications technique [16, 26] has been adopted in some OSS projects that could bank on the participation of usability experts. These are, however, exceptions because most OSS projects are run by volunteers working to small budgets and cannot afford external experts (like graphic designers or usability experts) [4, 9, 16].

3 Research Method

We used a case study as the research method to validate our research [27]. From a case study, we learn about the experiences of applying usability techniques adapted to OSS projects. This research method is used when the phenomenon under investigation (in this case, the adoption of an adapted usability technique) is studied within its real setting (in this case, an OSS project). OSS projects are the perfect setting for the case study reported here because OSS communities are generally uninformed about usability techniques, do not have the resources to test usability and cannot usually count on usability expert involvement [4, 9, 16].

The case study addresses the following research question (RQ): Is it possible to determine whether, if adapted, the focus groups usability technique can be used in requirements engineering activities within an OSS project?

ERMaster, a graphical editing tool for entity-relation diagrams (ERD), was selected as the OSS project in which to adopt the focus groups technique [28].

In this research, we first identified the obstacles to applying the focus groups technique in the ERMaster project. We then decided how to deal with the obstacles. Finally, we proposed the adaptations necessary to adopt the focus groups technique in this project.

We created web artefacts to improve communication with OSS community members and efficiently synchronize the necessary activities to apply the focus groups usability technique. The web artefact used to test the feasibility of the proposed technique was a forum. Forums are used in the focus groups technique to gather information and compile sketches related to the application user interface. Thanks to this web artefact, we were able to set up a virtual meeting point with OSS users who are geographically distributed all over the world. Using such web artefacts, we aimed to record user opinions about the selected OSS project user interface.

4 State of the Art

In recent years, the worldwide OSS community has adopted just over 50% of the HCI techniques related to evaluation. However, only about 20% of the usability techniques related to requirements engineering and design activities have been adopted [18]. Therefore, more research is required to support the adoption of techniques related to requirements engineering in OSS developments.

In view of the importance of HCI and SE, it is only logical to study the user-centred software development activities in OSS projects. This is especially true of the requirements engineering stage, because the discovery of user requirements during the early development activities is useful for putting right any defects in software detected later on [26].

The HCI vision of software development is somewhat different to the usual SE approach. Despite being based on the same development processes and activities, HCI focuses primarily on the user as the hinge of the system. Coutaze claims that HCI and SE intersect [27], whereas Ferré defines a scheme categorizing HCI activities and their relationship to SE [19].

There are a wide variety of techniques in HCI. The same technique may be referred to differently by different authors, and there may be different variants of the same technique. Fortunately, SE authors have already put together a catalogue of HCI techniques [19]. We referred to the catalogue of techniques compiled by Ferré [19]. The activities reported in this catalogue tie in with the early software development activities corresponding to requirements engineering, whose aim is to model users and determine their mental model [19]. This is a tricky issue owing to the types of development environments and user participation in the field of OSS.

In this paper, we adapt the focus groups technique used in the product concept development activity. According to Preece et al. [27], product concept development relies on the creation of a mental model based on psychological theories related to HCI. Ferré explains that this activity covers issues regarding how users envisage the system [19]. Therefore, this activity aims to provide a picture of the product before defining the features that the system should offer.

Usability technique definition and integration into OSS projects is a complicated process, about which there are few papers [6, 26, 29, 30]. These papers suggest that usability techniques should be reconceptualized, but they do not explain how the OSS community should go about adaptation. Nichols and Twidale [4] are the only authors to put forward some general ideas for improving usability. However, the issues to be taken into account to adopt such techniques in OSS developments are unclear.

On the other hand, Castro [18] proposes a framework for integrating usability techniques into OSS developments. This framework is composed of a number of adaptations in response to the adverse conditions for adopting usability techniques in OSS development projects. In order to adopt usability techniques in OSS development projects, it is, according to Castro [18], necessary to: (i) study the adverse conditions preventing the use of HCI techniques, and (ii) analyse what types of and which adaptations are necessary if these techniques are to be used in OSS projects.

Although research examining usability in OSS has been published [9, 26, 31, 32], there is no standardized procedure for determining how to adopt usability in OSS development. The first step in our research is to study how the OSS community uses usability techniques in their development projects. Castro’s is the only published research to study usability problems and techniques occasionally adopted in OSS projects in an integrated manner and to report the current state of usability in the OSS community [18].

It appears to be less straightforward to integrate usability into the OSS development process than into commercial development projects due to some of the characteristics of the OSS community, like: (i) feature-centred development, (ii) worldwide geographical distribution, (iii) limited resources, and (iv) a culture that may be alien to interaction designers. Consequently, usability technique adoption is a demanding task because most HCI techniques are not designed for the type of environment in which OSS is developed [18].

In the wake of the literature review, we can say that only one of the research papers reports a general and systematic proposal for integrating usability techniques into the OSS development process [18]. To do this, it considers the particular characteristics, philosophy and idiosyncrasy of the OSS development process, without forfeiting the essence of usability techniques. Two systematic mapping studies (SMS) related to usability in OSS were conducted in advance of our research. A SMS reviews the literature on a particular field of interest [32]. The first SMS was conducted by Castro [18] reviewing papers published up until 30 July 2013. The second SMS was conducted with a search range from 1 August 2013 to 30 April 2015 [33].

Castro’s research [18] was validated on only two OSS projects (OpenOffice Writer and FreeMind) and for three usability techniques (user profiles, direct observation and post-test observation). Therefore, Castro’s proposal [18] requires further validation by adapting new usability techniques and participating in more OSS projects.

5 Proposed Solution

In this section, we describe the focus groups usability technique applied in an OSS project. Firstly, we describe the case study design. Secondly, we specify the characteristics of the selected OSS project (ERMaster). Thirdly, we describe the selected usability technique (focus groups) as prescribed by HCI. We then introduce the adaptations made to the focus groups technique for application in an OSS project. Finally, we report the results of applying the focus groups usability technique.

5.1 Case Study Design

Case studies are one of the most popular forms of qualitative empirical research [34]. A case study investigates the phenomenon of interest in its real-world context. The phenomenon of interest for this research is the adoption of usability techniques with adaptations, whereas the real-world context is OSS projects. It is not easy to run controlled experiments in the field of OSS because the characteristics of OSS communities (for example, age, availability, expertise, experience, etc.) are unmanageable. Since not all OSS project team members have the same characteristics, it is impossible to minimize the effects of external factors (for example, geographic distribution and time differences). This rules out evaluation by means of an experiment. On this ground, we selected the case study methodology to validate the feasibility of our proposal for adopting a usability technique in an OSS project.

We describe the case study following the guidelines set out by Runeson and Host [24]. According to these guidelines, we divide our research into two parts: an exploratory part and a descriptive part. We start by looking at what happens in a real-world scenario and then we describe what happens when we apply the adapted techniques to improve application usability [24].

5.2 ERMaster OSS Project Characteristics

We selected ERMaster [28] as the OSS project in which to adopt the focus groups technique. ERMaster is an Eclipse plug-in and is very useful for novice or expert database (DB) designers. The reported number of downloads from its website is 1200 per week [28].

5.3 Focus Groups Usability Technique Description

The focus groups technique is a useful tool for evaluating user needs and feelings about a product expressed at group sessions [35]. More formalized definitions in the field of HCI describe the focus groups technique as a qualitative technique whose aim is to gather information about user opinions, problems and concerns at meetings planned for the purpose [18,19,20].

According to the literature, several authors [18, 19, 36] neither consider the planning required before and after applying the focus groups technique nor propose definite steps for applying the technique either. By contrast, Mayhew proposes a number of specific steps for applying this technique [20]. According to Mayhew [20], the focus groups technique is composed of the five steps described below.

Step 1 (Design the focus groups format) involves designing a script for the purpose of implementing a planned sequence of activities to be performed before, during and after conducting the focus group in order to achieve the goals set out in this study. Step 2 (Design data collection forms) involves designing a data input form (for example, to note down the opinions, problems and comments raised by focus group participants). Additionally, a list of specific questions (related, for example, to the user interface and the work environment) has to be compiled and addressed and discussed by the focus group. Step 3 (Conduct the focus group) should take about one to two hours. According to Mayhew, a good-sized focus group would have between six and eight members. Additionally, she believes that the moderator and note-taker play a very important role with respect to the key information stated by participants [20]. Steps 4 and 5 of the focus groups technique (Analyse data and Draw/document conclusions) address the transcription, analysis and summary of the results to draw and document the focus group conclusions.

5.4 Adaptations of the Focus Groups Usability Technique

The focus groups usability technique cannot be applied directly in the OSS development process because this community has features that do not conform to the HCI world, like, for example: the worldwide geographical distribution of its members, a code-centred world view, a shortage of resources and a culture that can be somewhat alien to interaction developers. Even though usability techniques demand conditions that, as a rule, OSS projects cannot meet, the techniques can be adapted to bring them into line with the idiosyncrasy of the OSS world. In the following, we describe the adaptations of the focus groups usability technique for application in OSS projects.

A usability expert is indispensable for applying Step 1 (Design focus group format) of the technique [20]. This expert is needed to structure the scripted objectives, topics and questions to be analysed when the focus group is conducted. We propose to substitute this expert with the principal developer, an experienced OSS project user or a HCI student under the supervision of a mentor to guide the focus groups format development. With regard to Step 2 (Design a data collection form), we found that the topics to be dealt with in the focus group cannot be physically handed out to participants because they are distributed all over the world. On this ground, we suggest that remote participation in the OSS community should be logged (online forum). Additionally, the outline of the topics should be posted on the same online forum so that users can recall their experiences with the software system interface under study.

In Step 3 (Conduct focus groups), we found that users are required to meet face to face to participate in technique application. Additionally, we found that a moderator and a note-taker had to be there in person to guide discussions and take notes during focus group application, respectively. This condition cannot be met due to the characteristics of OSS projects. On this ground, we suggest the following adaptations: (i) users will participate remotely in virtual meetings via the online forum; (ii) the moderator will be replaced by the principal developer, an expert OSS project user or a HCI student under the supervision of a mentor, (iii) there will be no note-taker during the conduct of the focus group because the online forum will be logged automatically.

In Step 4 (Analyse data), the information should be organized and then grouped by characteristics (such as age range, gender, occupation, etc.) before analysis. This simplifies the process of data analysis for the purpose of comparing and correctly interpreting the information gathered in the focus group [37]. Finally, Step 5 (Draw/document conclusions) draws the conclusions with respect to the opinions expressed by users. We did not identify any adverse conditions for the last two steps, and therefore no adaptations had to be made. Table 1 summarizes the steps, identified adverse conditions and suggested adaptations for the focus groups technique [20]. There are mainly two adaptations. First, users participate online via a forum. Secondly, the usability expert is replaced by a developer, expert user or a HCI student under the supervision of a mentor. In this particular case, a HCI student under the supervision of a mentor substituted the expert.

Table 1. Steps, adverse conditions and proposed adaptations for the focus groups technique
Full size table

According to HCI prescriptions, design tips for a new application feature output by the focus groups technique are appraised by a usability expert [19]. In the adapted focus groups technique, the end users submitted their designs and opinions via web artefacts (like forums and emails) and not at face-to-face meetings due to the characteristics of the OSS communities.

In sum, two main adaptations were taken into account for the adoption of the focus groups technique in OSS projects. Firstly, the focus groups usability technique requires a usability expert for application, as a result of which we recommend that the expert be replaced by a HCI student (under the supervision of a mentor). Secondly, as this technique is reliant on user participation, we propose that OSS users participate remotely via chats, forums, blogs and wikis.

5.5 Case Study Results

We applied the focus groups technique in the ERMaster project. This is a DB data management application that works with several DB engines. We had trouble recruiting real users because it took a long time to get permission from the principal developer. We had to contact the principal developer by means of several media (email and personal wiki) before he gave us his consent. Six ERMaster users participated in the focus groups technique application.

After designing the focus group format taking into account the stated topics and objectives, we proceeded to phrase the questions in order to apply the focus groups technique. Table 2 shows the (unstructured) format design. The questions should be aligned with the objectives addressed in the focus groups and are related to the ERMaster application work environment. The focus groups questions are designed to evaluate usability issues such as ease of learning, efficiency of use, memorability, errors and satisfaction [36]. By studying these factors, we focus on user-centred design, an issue neglected in OSS development projects. We sent the questions and an invitation to participate in the focus groups technique to a mailing list of active ERMaster community users. This list was supplied by the ERMaster principal developer. The questions that were sent to the users by electronic mail are listed on the web siteFootnote 2. We expected a higher rate of participation from the ERMaster community. Since it took a long time to get the principal developer’s permission and users did not show much interest in participating in our research, we only managed to recruit six participants.

Table 2. Focus groups technique format
Full size table

The focus group was moderated by the principal developer. However, he did not comment on the opinions of the users posted on the online forum. We believe that the principal developer did not get involved in the open online forum because he was not unduly concerned about improving the usability of the OSS application. The presence of a note-taker was unnecessary in order to conduct the focus group, as, on one hand, the comments posted on the open online forum were logged automatically and, on the other, the researchers kept all the emails that they received.

The principal developer sent an introduction and formal invitation to the ERMaster project community to participate in the open online forum. For many application users, however, their forum registration is the only record available as they did not post any opinions. Some users answered the questions and submitted their responses by email instead of publishing them on the forum. Other users eventually responded to one or two questions related to their major field of interest but failed complete the entire questionnaire. A few other users stated that they were happy with the tool and did not answer any of the questions. We then screened all the feedback, and selected the contributors who answered all or most of the questions. As a result of this screening, we got a sample of six users for our research.

The noteworthy results of the application of the focus groups technique considering the data gathered include: (i) novice users had problems with installation (because it is an Eclipse plug-in), (ii) expert users regard ERMaster is being a tool that is easy to learn, easy to use and easy to understand and had no trouble remembering how to use it, (iii) ERMaster is designed ergonomically using menus, action bars and easy access icons, but some users requested the addition of options of interest (for example, export DBs to Excel), and (iv) users consider the ERMaster work environment to be adequate, as there are Help and Query tools.

6 Discussion of Results

After applying the focus groups technique to the ERMaster project, we were able to confirm that it is very hard to get a representative set of users. We believe that the main reason for this is that users are unmotivated. We had to be persistent and use different communication mechanisms (for example, personal wikis and electronic mails) to get the consent of the principal developers (only one out of five principal developers responded). The biggest problem with applying the focus groups technique was user availability: most users are volunteers and had very little spare time. In fact, the participants did not have the time to enter their comments in the online forum and ended up emailing their opinions to one of the researchers. Since the focus groups participants had a medium level of experience with respect to both the ERMaster tool and the field of computing, they did not pinpoint any major problems which novice users may have had.

Table 3 summarizes the results of the analysis of the characteristics of the case study. Note that the bottom row is shaded grey, denoting, for the purposes of this research, that the adoption of the adapted focus groups technique was acceptable as this technique requires a small number of participants to get reliable results. With regard to our proposal of substituting a developer, expert user or HCI student under the supervision of a mentor for the usability expert, the expert was replaced in this case by a HCI student supervised by a mentor. Note that this student was in his final year of the Master of Information and Communication Technologies Research and Innovation at the Autonomous University of Madrid and was taking two HCI courses. Additionally, the student was supervised by two usability experts. On this ground, there is no risk of the proposed adaptation for the selected technique having a negative impact on the quality of the software.

Table 3. Summary of the case study for the adapted focus groups technique
Full size table

Note that the problem stated and the solution proposed in this research are of interest and importance not only to OSS development communities but also to the field of global software development (GSD), since GSD business and industrial settings share a number of the characteristics of OSS communities. GSD is now a major issue in SE research and practice [38]. Some noteworthy negative factors that GSD has to tackle are neglect by project managers, member participation and allocated resources influencing organizational success [39]. Against this backdrop, the results of our research can be extrapolated in order to deal with the above negative factors within the GSD field.

We can conclude that the results of the adoption of the focus groups usability technique were not what we expected. Firstly, we banked on the participation of a large number of users based on the statistics provided on the application web site. Secondly, it was hard to contact and recruit users to participate in the research. Note that OSS community members are all volunteers, and they participate in their spare time. Despite all these problems, however, the adaptation of the focus groups technique was reliable for adoption in the ERMaster project, as it does not take many users to get a reliable result.

7 Conclusions and Future Research

The goal of this research was to evaluate the feasibility of adopting HCI usability techniques in OSS projects. We adapted the focus groups technique for adoption. Through adaptation, we were able to account for some OSS development characteristics that pose an obstacle to the application of the technique as per HCI recommendations (for example, OSS developers and users are geographically distributed). In particular, we adapted the focus groups usability technique for application in the ERMaster OSS project.

It is not easy to recruit volunteer users to participate in OSS usability projects. As already mentioned, users often do not have much time, and it is hard to get them to take part without an incentive. With the focus groups technique, although we did not get much collaboration from users or even the principal developers, we did manage to apply the technique because it requires only a small number of participants to get a reliable result [19, 20, 35,36,37]. Author opinions differ as to the number of users to be taken into account for the focus groups technique to be successful [19, 20, 35,36,37]. Most of these authors agree that a focus group should include from six to nine users if it is to work. Fewer than six participants would not generate enough ideas for discussion. In this research, however, any users that are willing to collaborate are allowed to, that is, there is no limit on the number of users because this would go against the working philosophy of the OSS community where anyone who wants to is welcome to participate. In sum, our proposed adaptation does not place any constraints on the number of technique participants. This adaptation is a response to the OSS community working philosophy rather than to an adverse condition posing an obstacle to technique application.

The focus groups technique is useful for gathering opinions and suggestions from participant users for the product concept development activity and its results are descriptive. After analysing and applying the focus groups usability technique in requirements engineering activities in OSS developments, we found that there are adverse conditions that are an obstacle to its application like, for example, the shortage of OSS users interested in applying the technique, community geographical and temporal distribution and OSS community motivation.

We believe that, in order to improve the integration of usability techniques in OSS projects, the OSS community has to start attaching importance to and raising awareness about the repercussions that the issues addressed by the HCI field have on software development. Additionally, as HCI techniques need to be adapted to overcome the adverse conditions for adoption in OSS development projects, the OSS community also has to broaden its view of software development in order to consider usability and not focus exclusively on feature development. In the future, we aim to conduct further case studies to adapt and apply other usability techniques in OSS projects. We will analyse other web artefacts that can be adapted to improve communication in OSS communities (for example, social networks) and gradually raise the awareness of OSS developers about the benefits of applying HCI usability techniques.