Keywords

1 Introduction

Requirements enable system developers to “design the right thing”, beyond “designing the thing right [1] ”. In order to “design the right thing”, it is important to initiate system development process through well-defined requirements which are described as achievable, verifiable, unambiguous, complete, correct and consistent [2]. On the other hand, non-functional requirements related to the property and quality attribute that systems must have are usually described as ambiguous and difficult to measure contrary to definite and straight-forward functional requirements about what systems must do [3]. Even though non-functional requirements play crucial role in system development process [4], and neglecting them is counted as one of the top risks of requirements engineering [5], they are still poorly understood and difficult to elicit.

One type of non-functional requirements which derives from the needs, preferences and physical/cognitive capabilities of the users is design requirements. Even the importance of design requirements referring to the specifications that systems provide to its users is discussed through different sources [6, 7]; the elicitation of formal design requirements presents a challenge [8]. Therefore, practitioners often describe design specifications with ambiguous statements like “easy to use” or “user friendly” which have different meanings in different contexts [9]. Additionally, the early phases of system development process are already characterized by the greatest degree of uncertainty and crucial user information is usually lost or not gathered [10]. When critical user information is not realized in the early phases of the development process, modification costs increase drastically as the development process proceeds towards the end [11].

Within the framework of this problem, this research was conducted in order to determine design requirements that may contribute to the development of an electronic warfare system’s hand terminal through which “…the user shall easily operate the functions of the system…” as stated in the technical specification document of the related project. By this means, the aim was to explain the implicit user demands on system functions and the specifications that the hand terminal should provide to its users in a way guiding the designers to define design requirements and to realize the final design.

2 Method

2.1 Participants

As a critical first step element to define requirements [12], the participants of the study were selected among stakeholders representing the actual actors of the system. However, when compared to the civilian, military population is too small. Especially regarding the end-users of a special military product, it is very difficult to find enough number of test participants because of time considerations and also bureaucratic reasons. Therefore, only six of 16 participants were potential end-users who had used the previous version of the related system (M = 31.3). Other 10 participants were recruited to represent a sample of development team personnel including systems, hardware and software engineers (M = 30.4).

2.2 Test Materials

Prototyping is assessed as one of the most effective ways for identification of stakeholder requirements at the early phases of development processes [13]. Low fidelity paper prototypes of two hand-held terminals were used in the study. These prototypes were designed to fulfill the functional specifications of the related system and to help user accomplish the same tasks through different user interfaces.

Even the outer dimensions of the prototypes (length: 125 mm, breadth: 80 mm, depth: 20 mm) and the graphical symbols used in the interfaces were the same; design specifications including menu structure, LCD screen, layout and positioning of interaction elements were differed in order to make design specifications comparable and gather more prosperous data (see Fig. 1).

Fig. 1.
figure 1

(a) Prototype 1 and (b) Prototype 2

Full size image

The main difference between two designs was the type of LCD screens used. While a 16 characters by four lines LCD which could only display ASCII characters and some symbols was used at Prototype 1, Prototype 2 had a graphic LCD using a dot matrix pattern to display both text and image. This difference also provided a base to diversify other interface elements. By this means, half of the panel keys of Prototype 1 were assigned to fixed functions separately; while functions of the all keys of Prototype 2 varied in accordance with the display interface. The summary of the differences between prototypes is given in Table 1.

Table 1. Main differences of the test materials related to interface elements.
Full size table

2.3 Data Collection Techniques

There are many methods and techniques to discover users’ cognitive demands and needs such as questionnaire, survey, focus group, observation and etc. Two main approaches referring these methods and techniques are called formative and summative approaches. While summative approaches evaluate the final design at the end of the development process; formative approaches intent to form the required information in the early phases and describe what and how the design should do. Since the study aimed to investigate design requirements at the early stages of development process, formative approach was used.

The qualitative responses of the participants were gathered through semi-structured in-depth interviews and it was attempted to understand the implicit demands through probing. The participants were also asked to rate the perceived ease of use of the test materials. Therefore, in order to make the prototypes comparable quantitatively, subjective evaluation ratings were gathered as an experimental measure. Compromising four items of TAM3 [14], perceived ease of use was measured. The behavioral intention to use was also intended to be predicted through the evaluation questionnaire, and it was measured through one item. All items were examined through 5-point scales, labeled “strongly disagree” (1) and “strongly agree” (5) at the end points. Thus, formative data about the perceived ease of use and the intention to use were gathered.

2.4 Procedure

In their workspaces, the participants recruited were randomly assigned to one of the two prototypes first. Then, the participants were informed that they would evaluate two low fidelity prototypes having the same functional specifications with a view that their feedback would be used to define the requirements of the hand terminal design. Later; a demonstration of the prototypes including the functions of interface elements was given by the researcher. Afterwards, the same procedure was repeated for the other prototype.

The technical specifications of the related system demand both lower and higher user skills. Therefore, nine task scenarios were selected to represent a sample corresponding both basic and complex capabilities. The scenarios were demonstrated on cardboards as separate sequence diagrams. A sample task (the activations of predefined operational modes) represented in a sequence diagram was given in Fig. 2.

Fig. 2.
figure 2

Sample sequence diagrams of a task

Full size image

After having completed the demonstrations, the participants were asked to fill in the questionnaire and the behavioral intension to use scale for each prototype respectively. Later, any issues regarding the design specifications of the prototypes and problems referring to the task scenarios were probed through interviews.

3 Results

3.1 Results of Subjective Evaluation Ratings

Two paired-samples t-tests were conducted to evaluate the results of subjective ratings through main factors of the questionnaire; perceived ease of use (PEOU) and behavioral intention to use (BIU). As shown in Table 2, statistically significant differences were not found in the PEOU scores for Prototype 1 (M = 4.20, SD = 0.68) and Prototype 2 (M = 3.95, SD = 0.53), t (15) = 1.49, p > .05; and in the BIU scores for Prototype 1 (M = 4.06, SD = 0.93) and Prototype 2 (M = 4.19, SD = 0.75), t (15) = −.46, p > .05.

Table 2. Paired-samples t-test results.
Full size table

Even any significant differences between the level of prototype did not emerge, it is remarkable that (see Table 3), Prototype 1 was rated with higher scores in the PEOU factor (MP1 = 4.20, MP2 = 3.95); while getting lower scores in the BIU factor (MP1 = 4.06, MP2 = 4.19).

Table 3. Descriptive stats related to test materials.
Full size table

3.2 Results of In-Depth Interviews

In order to achieve a holistic perspective; the issues mentioned by the participants during interviews were analyzed and assigned to a category system including two main themes, six sub-themes and seven design items. While one of the main themes was concerned with the extent to which the specifications of user interface designs and tasks performed were quickly and clearly interpreted (comprehensibility); the other was concerned with the extent to which the functions of the system were efficiently and effectively used (effectiveness). As well as seven design items regarding the design specifications of interfaces; the sub-themes were identified with regard to the variety of mentioned issues including intuitiveness (e.g. setting signal output violates common stereotypes), familiarity (e.g. interface layout is similar to ATM’s), visibility of the system status (e.g. tracking warnings through distinctive indicators), task completion performance (e.g. accessing task window with one action), error prevention (e.g. display characters are big enough to read) and compatibility with hand use (e.g. controls are not easily accessible for one hand use).

Later, in order to elaborate the interrelation within category system; the main themes were associated with the related sub-themes and design items. As an example, when a participant said “I could lose time since my hand obscures the display while using keys”, two interactions were set between effectiveness and task completion performance, and effectiveness and layout of interface elements. By this way, not only the interrelations between categories, but also the strength of these interrelations through the frequencies of mentioning were clarified, and the data were visualized into two maps (see Fig. 3).

Fig. 3.
figure 3

Interrelations among categories

Full size image

The results showed that, out of 87 mentioned issues about PEOU, 54 of them were about effectiveness of the system functions. On the other hand, the most frequently mentioned sub-theme was familiarity (26), followed by task performance (22) and error prevention (16). Other categories were referred less frequently. There were also a number of design items which were emphasized more. However, there was a homogeneous distribution when compared to sub-themes. Fixed/variable key functions (19), function labeling (17) and menu depth (15) were the most frequently mentioned design items related to PEOU. However, while function labeling (13) was associated with comprehensibility; fixed/variable key functions (16) and menu depth (12) were mostly associated with effectiveness.

A similar analysis was conducted to evaluate the issues mentioned within the topic of BIU. It was seen that, of out 25 issues, almost half of them were about PEOU of the hand terminals (12). There were two sub-themes which were not mentioned within PEOU issues. One of them is to ability to have a configurable architecture which can allow developers to integrate new functions to the hand terminals (4). The variable key functions and menu depth were associated with the issue. The other one was technological appearances that hand terminals had (8). The bigger sized, graphic LCD screen of Prototype 2 was mostly referred when mentioning the perception of technological appearance.

Finally, apart from the interrelations among categories, there were a number of issues which were related to the functional aspects of the hand terminal. These included restoring to the default settings (5), blackout mode when operational security is required (3) and storage of custom operational modes generated by system operators (3).

4 Discussion

The first aim of the study was to clarify the implicit user demands on the functions of the system and the specifications of the hand terminal related to ease of use. Rather than focusing on the functional capabilities of the related system, it was attempted to identify as many issues as possible through the use of two test materials offering same functional attributes with different interface designs. Using a semi-structured interview with open-ended questions and also recruiting participants that could represent the stakeholders were crucial, since these enabled to get prosperous information from different perspectives.

When taking the overall pattern of the results into account, it was seen that the issues related to the efficient and effective use of the hand terminals were mentioned noticeably more than the issues related to the clearness and comprehensibility of the interface elements. The main reason of this situation was assessed as the necessity of performance expectancy in military systems. Another reason was associated with the test procedure through which the functional capabilities of test materials and the completion of task scenarios were explained by the researcher. The demonstrations might be appraised as training sessions decreasing the comprehensibility issues.

In the PEOU case, effectiveness surfaced in two subtle forms. Firstly, it was strongly associated with the issues of task completion performance and error prevention. It was not surprising, since military systems should aim to minimize factors degrading human performance and increasing errors. Additionally, it was observed that nearly all study participants focused on the worst cases by referring to the potential novice users and the task scenarios having more detailed levels. Secondly; the keys of interfaces, whether assigned to fixed or variable functions, were associated with effectiveness. When compared to variable function keys, fixed function keys serving direct access to the related menus were evaluated as determinants of task completion and error prevention. However, they were also criticized, since they could cause complexity and degrade human performance when there were an excessive number of functions. Apart from key functions; number of keys and menu depth were also mentioned and evaluated in a similar manner. Therefore, the trade-off among fixed/variable key functions, number of keys and menu depth presented an important parameter influencing perceived effectiveness. Another emphasized design item was the position of warning indicators. For the sake of the visibility of system status, warning indicators were noticeably demanded as hard, visual indicators positioned separately in the interface; instead of soft indicators placed on LCD displays. The theme comprehensibility, on the other hand, was mostly mentioned through familiarity. As well as referring to products which had similar interface layouts like previously used hand terminals and ATM’s, the appropriateness of function labeling was associated with the perception of comprehensibility. While the use of descriptive wording, standard icons on keys and displays were appreciated, a notable amount of participants pointed out the ambiguity within control-display integration. The lack of key status presentation on the display face and the lack of direct marking between display and associated control, especially for variable function keys, caused the impenetrability of proper identification and utilization.

The other factor of the study, BIU, was also associated with PEOU through formerly mentioned performance expectancy requirements and also with two issues. One of them was to ability to have a configurable architecture perceived through variable key functions and menu depth; the other was technological appearance perceived through the size of LCD display. Prototype 2 was mostly referred when mentioning these issues, and also evaluated with higher scores than Prototype 1, even statistically significant differences were not found. It should be also noted that fixed function keys and menu depth of Prototype 1 serving direct access to the related menus were identified with efficient and effective use; thus Prototype 1 was rated with higher scores in the PEOU factor. Finally, there were a number of issues which are concerned with the functional aspects of the hand terminal should also have. These functional aspects were defined based on the experiences of the participants regarding the context of use. Therefore, the results of the study revealed that; even design requirements are considered non-functional requirements establishing constraints on the means to meet functional aspects of a system, in order to develop a system conforming users, design requirements should also shape functional requirements.

5 Conclusion

Even design requirements have an important role in conformance of a system, collecting user information and deriving them into unambiguous and verifiable requirements presents a challenge. This study supports the value of conducting user research at the early stages of development process with an approach using low fidelity prototypes, questionnaire and in-depth interviews to collect data, followed by a focused analysis to present user demands in a generalizable manner to derive research data into actionable design requirements. The information gathered has positive contribution to system development process; since not only the specifications that system should have are clarified and the design requirements are elicited, but also the user acceptance is aimed to be fostered in a mission critical military system.