1 Introduction

Car manufacturers have been pioneers in the use of 3D environments for prototyping and evaluating a product’s design [1], and today automakers are exploring, with growing interest, the use of virtual training solutions for training operators by investing in international research projects. Car service maintenance is considered a key topic for automotive industries, because service operators are the main interface between brand and costumers with car issues [2, 3]. Therefore, the objective of automotive industries is to invest in effective tools to enhance service operators’ procedural skills. Several studies in the literature [4] outlined that virtual training has a positive impact on operators’ acquisition of new skills – see for instance: techniques of welding, prototyping, object assembly etc. Nevertheless, currently only a few studies are available in the literature about virtual training of car service operators [5, 6]. Moreover, practitioners in 3D training of automotive operators are used to focus their attention on the enhancement of the interactive aspects of virtual environments only in terms of functionalities, with minimal effort in the assessment of the end-users’ experience of the training system.

As Mantovani [7] claimed the interaction experience of a 3D environment could significantly affect the trainees’ interaction, thus compromising the trainees’ acquisition of the content. In light of this, the usability – intended as the “extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” [ISO 9241-11, 8] – is a key aspect for the success of a training application. Despite the importance of the usability of 3D environments being a well-known concept, as underlined by several authors [7, 9, 10] practitioners in the virtual reality field are used to applying a qualitative approach to usability assessment, and a large part of the studies in the field discuss the outcomes of summative evaluations – i.e., when the product is in an advanced stage of development – by presenting data obtained through qualitative and unstandardized questionnaires administrated to the end-users. Differently from the human computer interaction field, rarely do practitioners in the virtual training field discuss the approaches and methods they applied during the formative evaluation of the prototype – i.e., the phases of prototype evaluation and redesign which aim to integrate designers and end-users’ mental models [11].

In this paper we will discuss a full formative evaluation paradigm applied to assess a new 3D training system for car service operators consisting of three phases. The first phase was an experts’ analysis carried out by a heuristic list [12] and a four-question cognitive walkthrough [13]. The second phase was a redesign phase in which designers, in line with AGILE principles [14], solved only the most relevant interaction issues revealed by experts. The third phase consisted of a user test carried out through a scenario based analysis. In this last step, we gathered the users’ errors during the interaction with the system and their satisfaction in the use through a standardized questionnaire – i.e., the System Usability Scale [SUS, see: 15]. Both the experts and the end-users interaction issues will be also modelled through estimation models, in line with the Grounded Procedure approach [i.e., GP, see: 16–18], to identify the percentage of problems identified during the evaluation test.

2 Initial Prototype

The prototype application applied in this study was developed by HoloVis international, as part of the Live Augmented Reality Training Experience project (LARTE project, TSB – 101509). LARTE software is based on the HoloVis game engine (InMo™), which enables end-users to visualise and interact with CAD data by using a number of supported devices (see Fig. 1): multi-sided CAVE, powerwall, Oculus Rift, and interactive table-top displays. For this study, LARTE system is adapted to the specific requirements and desiderata of JLR, and is used as a prototype to train car service operators on maintenance procedures. Designers developed the prototype to train in a 3D environment people who could receive an automatic step-by-step explanation of the procedure. Moreover, people could manipulate and make trials of the procedure against time limits to test their level of performance proficiency. LARTE application functions are variables to grant a degree of freedom to trainers of different industries to set up a virtual experiential activity.

Fig. 1.
figure 1

LARTE prototype experienced through Zspace (Section a.) and CAVE (Section b.)

Full size image

The main five main functions of this training application can be described as follows:

  • Trainers can create a training by importing a CAD object, and setting a step-by-step training video of the procedure;

  • Trainers can define a set of rules and relationships among the components of the 3D objects;

  • Trainees can autonomously visualize each single step of the procedure by receiving in the virtual environment verbal and textual instructions of a service procedure, through a video recorded virtual explanation of the procedure;

  • Trainees can manipulate the 3D car – e.g., rotate the whole CAD, assembly, disassembly, rotate and move any single components of the car, zoom in and out etc.

  • Each trainee’s interaction can be fully recorded and observed in the 3D environment.

Finally, LARTE prototype is designed to be used through different devices, this characteristics allow operators to use during the training not only the CAVE but also portable tools, such as interactive 3D table.

2.1 Procedure

The evaluation of the LARTE prototype was carried out by using two different set of devices:

  • Not portable: a 4-sided CAVE (VR1) with rear-projected floor (Sony VPL-FE40) with a mirror rig and 2 projectors per wall – 8 in total. The resolution of each projector was 1440 × 1050. The controller of the CAVE was a ART DTrack optical head and wand tracking. The tracking was integrated in a pair of lightweight polarized passive glasses. The system was run by 9 workstations.

  • Portable: zSpace holographic 3D table (VR2) composed by a 24 in. (1920 × 1080) LCD monitor (tablet display) running at 120 Hz. The controller was a laser-based wired six-degree-of-freedom stylus device. The tracking was integrated in a pair of lightweight polarized passive glasses [19].

A virtual version of a Range Rover Evoque car was created to perform the usability analysis. A trainer explained to both experts and end users, for five minutes the functions of the prototype, and the use of the physical controller of the device. After the explanation experts and end-users were required to achieve the following five tasks (presented by scenarios): (i) use the controller to visualize the virtual video training; (ii) skip and rewind sections of the video training; (iii) use the control to zoom in the virtual car and remove a specific bolt; (iv) bring the removed bolt out of the car, and zoom out; (v) rotate the car and reassembly a bolt in the correct position. Five experts (2 Male; Age M.: 28.5; SD: 3.4) with at least 3 years of experience in human factors, interaction and virtual reality were asked to perform the five tasks by using both the CAVE and the zSpace. The order by which experts experienced the portable or not portable device was randomly defined. After each interaction session with a device, each expert filled the evaluation form composed of the ten classic heuristics items [12] and a four questions cognitive walkthrough [13]. Forty end users (Male: 14; Age M.: 35.89, SD: 13.11) were randomly assigned to experience the prototype by using or the CAVE or the zSPACE (i.e., 20 subjects per each device). After the interaction participants were asked to fill the SUS, composed of 10 items.

2.2 Experts’ Evaluation

Experts identified 10 usability problems of interaction with LARTE tool in the CAVE, and 7 during the experience with the zSpace. In line with the GP approach, the Return of Investment model [20, 21] and the Good-Turing model probabilistic model [22, 23] were used to estimate the upper and the lower percentage of the total usability problems identified by the experts. Our analysis showed that experts identified from 92 % to 97 % of usability problems in the CAVE, and from 95 % to 98 % of problems in the zSpace. The relevance of the problems was assessed in terms of experts’ agreement on the severity by using a scale from 0 to 100 (see Table 1).

Table 1. Interactive problems identified by experts in the use of LARTE application experienced in the CAVE and through the zSpace, and percentage of relevance of the problems.
Full size table

Redesign After the Experts Analysis.

The outcomes of the experts analysis was used to develop a list of priorities. In tune with AGILE principles, designers decided to solve, before the user test, only 10 out of 17 problems, i.e., only the issues estimated to have over 50 % relevance (see Table 1). The remained issues will be deferred and aggregated to the end-users’ feedback for the further step of redesign. The redesign of the LARTE was performed in three weeks. The new version was reviewed by an external expert to assess the congruity between the outcomes of the experts’ assessment and the redesign of the application. The new version of the tool, in line with indications of the experts, included revised features, such as for instance, a new and improved snap function, and new functions, such as the ‘Undo/reset’ control for the end-users.

2.3 User Test

The overall satisfaction in the use of LARTE tool, measured through SUS, equals 78.88 %. Participants who interacted with LARTE tool in the CAVE were more satisfied (85 %) than people who interacted with the zSpace (75 %). The t-test analysis showed that there were no significant differences between the groups of participants who interacted with the zSpace or in the CAVE in terms of usability of LARTE tool.

The percentage of usability issues discovered by the end-users in the CAVE ranged from 87 % to 89 %, while in the zSpace end-users discovered from 86 % to 93 % of the interaction problems.

Figures 2 and 3 showed outcomes of the SUS for the CAVE and zSpace users. As reported in Fig. 2, LARTE tool in the CAVE is perceived by end-users as a little bit complex in terms of ease of use (point 2, 35 %) although users have minimal issues in learning how to use the tool functions (point 10, 32 %). The lack of a wizard on how to manipulate the interface, and use the controllers brings the end-users to say that they will need assistance to learn how to use the system (point 4, 57 %).

Fig. 2.
figure 2

Users’ report about the usability of the CAVE

Full size image
Fig. 3.
figure 3

Users’ report about the usability of the zSpace

Full size image

People who interacted with the zSpace (see Fig. 3) declared that they will need of assistance to learn how to use the LARTE system (point 4, 54 %), moreover, for these participants LARTE resulted complex (point 2, 35 %), and awkward (point 8, 33 %). Nevertheless, participants experienced only minimal issues in learn how to use the tool (point 10, 36 %).

3 Discussion

The overall outcomes of the formative assessment of LARTE application showed that from the end-users’ point of view this application was satisfactory. By using the standard of SUS outcomes, proposed by Sauro and Lewis [23], LARTE was usable at level B (75 %) when experienced through the zSpace, and at level A (85 %) when used in the CAVE. However, there are not significant differences in terms of number of problems experienced during the interaction with the prototype, nor in terms of satisfaction scores. Only 10 problems out of 17 are considered severe (or moderate severe) issues by experts. The remained seven issues were considered small problems that designers may solve before the first release. These 7 issues were aggregated to the indications coming from SUS, for instance, the lack of a wizard about the LARTE tool and its main functions. All the remained issues and the feedbacks of the end users will be used to define the next version of LARTE application, before the summative evaluation.

4 Conclusion

Usability assessment of 3D interactive tools is a growing topic in the community of virtual training. The use of standardised tools and reliable approaches and methods of assessment can deeply enhance the overall user experience of the 3D environments, by concurrently increasing the effectiveness of the training experienced in virtual world. This study, by proposing a framework in which AGILE and user centred design principles are mixed together to simplify the redesign of the tool, showed how classic approaches of assessment can be applied to serve the scope of gathering reliable and structured evidences to identify the lacks of a product, and to proceed or discard specific lines of design. Further usability studies are needed in virtual training fields, especially today with portable 3D tools, such as zSpace or Oculus Rift, available on the market. In fact, these portable solutions are opening up the possibility for manufacturers to train their operators with alternative devices (and comparative low costs) to classic CAVE systems.