A comparison between expert and beginner learning for motor skill development in a virtual reality serious game

Abstract

In order to be used for skill development and skill maintenance, virtual environments require accurate simulation of the physical phenomena involved in the process of the task being trained. The accuracy needs to be conveyed in a multimodal fashion with varying parameterisations still being quantified, and these are a function of task, prior knowledge, sensory efficacy and human perception. Virtual reality (VR) has been integrated from a didactic perspective in many serious games and shown to be effective in the pedological process. This paper interrogates whether didactic processes introduced into a VR serious game, by taking advantage of augmented virtuality to modify game attributes, can be effective for both beginners and experts to a task. The task in question is subjective performance in a clay pigeon shooting simulation. The investigation covers whether modified game attributes influence skill and learning in a complex motor task and also investigates whether this process is applicable to experts as well as beginners to the task. VR offers designers and developers of serious games the ability to provide information in the virtual world in a fashion that is impossible in the real world. This introduces the question of whether this is effective and transfers skill adoption into the real world and also if a-priori knowledge influences the practical nature of this information in the pedagogic process. Analysis is conducted via a between-subjects repeated measure ANOVA using a $2 \times 2$ factorial design to address these questions. The results show that the different training provided affects the performance in this task ($N=57$). The skill improvement is still evidenced in repeated measures when information and guidance is removed. This effect does not exist under a control condition. Additionally, we separate by an expert and non-expert group to deduce if a-priori knowledge influences the effect of the presented information, it is shown that it does not.

1 Introduction

Clay pigeon shooting is a hobby sport that comprises the task of firing a shotgun at a disc of clay that is propelled through the air by a firing mechanism called a clay trap. There is inherent skill in this process, and skilled practitioners can focus on the target whilst simultaneously determining the path and speed of the target. They can then predict the point at which a collision would occur between the shot from the shotgun and the target in question. By aiming the shotgun at this collision point and firing, a hit is achieved. Both the target and indeed the shot have varying velocities and trajectories in their lifespan whilst airborne making this task quite challenging. As skill develops and is maintained by expert participants, the distances involved between target and shooter can be increased and the velocity of the target can also be increased to increase the difficulty. Targets are often designed on grounds that facilitate this hobby to account for a variety of skill levels, with experts traditionally being challenged by erratic targets or multiple targets at once reducing the amount of time available for decision-making.

Proficient coaches are rare, and it can be a hobby which is prohibitively expensive to learn. Instructor experience is invaluable as each participant may see things differently and indeed attribute different distances and velocities to objects. So there is no static way of communicating how to hit a target, instead instructors must be fluid and adapt to new techniques in order to convey how to hit a target. Simply saying “shoot another three feet in front” can mean different things to different people as distance and direction is perceived differently. Hits are obvious in the sport, the clay disc shatters, and the way in which this shatters can be used to determine the accuracy of the shot. Whether the shot cloud hits on the leading edge or a little higher can be useful instruction. However, when no shot hits the target, the feedback is almost non-existent. It is sometimes possible to see the shot cloud and where the miss occurred, but a trained eye is required for this. Whilst locating the miss is difficult, identifying the cause of the miss is harder still. Assuming the shotgun is mounted to the shoulder correctly, the gun was kept smoothly moving as the trigger was pulled, the trigger was squeezed so as not to alter the line of the swing, the lead assigned was correct, the shotgun was not canted, and the shot was fired into the correct location—it is still possible to miss. This is because the shot spreads after it leaves the shotgun and this is a function of the shot type, compression of the shotgun barrels (called choke) and bore of the gun. This spread can leave holes in the shot cloud that may mean the disc is not hit. A virtual shooting environment provides an opportunity to give accurate debugging information retrospectively to a shot which can aid in analysis of these myriad of factors.

Accurate simulation of this task depends on two pieces of information [11]: the location and orientation of the shotgun in the environment when the shot is fired and the velocity and position of the target(s) when the shot is fired. With this information, the simulation can emulate the paths each pellet in the pellet cloud will take and by simulating time steps of the physics for these pellets it is possible to predict intersections and collisions on a frame-by-frame basis. Replaying the simulation in slow-motion allows the user to visualise the shot placement and use this information to improve their technique, for a variety of different or bespoke targets. This feedback-loop training should aid their performance. This type of virtual solution enables teaching aids that are otherwise impossible in the real-world context: immediate and detailed visual shot feedback and a visual aid on shot placement. Additionally, shotgun techniques in the real-world context make heavy use of binocular vision. Monocular vision, when shooting, is considered a dramatically inferior technique [11]. VR provides a platform to address this concern.

Theories on frameworks for gamified learning as a process have been proposed by Landers [30]. It is suggested that serious game approaches via manipulation of game attributes affect the learning process without influencing the behavioural moderators and mediators of learning as posited by the Bedwell taxonomy [7]. This suggests that serious games are an effective pedagogic tool and with correct use of game attributes, this can be safer and more practical than real-world training. However, this is only true so long as the didactical principles of video games are integrated in the process [37]. This includes a preference for tangible games with multimodal-based interaction [32].

The main contributions of this work are as follows:

• an implementation of an efficient and accurate pellet cloud ballistics model and a target trajectory model both elucidated from literature,

• the ability to provide detailed feedback to the user on both pellet cloud and target trajectories to help diagnose misses and/or evaluate the quality of a hit/miss,

• a visualisation, in real time, of the lead required to hit the target (“sight picture”),

• a stereographic visualisation of the virtual range to allow the user to interact with the system in three dimensions and 360$^\circ$,

• a subjective evaluation of two types of training for this task and the efficacy of these VR-based training modes,

• an evaluation of expert versus non-expert learning in this task.

It should be noted that this article is an extended version of our conference paper, for this see Harvey et al. [28].

2 Related work

2.1 Background

Clay pigeon shooting was originally developed as a training aid and a substitute for shooting live quarry. Since the invention of this sport it has evolved and there now exist a number of branches of clay pigeon shooting and some of these are competed at the Olympic Games. It is one of the fastest growing sports in the UK, a report from 2014 suggested as many as 600,000 people within the UK shoot live quarry, clay pigeons or targets [42]. There are economic and agricultural benefits to this; nearly two million hectares of land are actively managed for conservation as a result of shooting and this is worth £2 billion to the UK economy [42].

Shotguns, defined by UK law, are smooth barrelled guns comprised by barrel(s) of length at least 24 in. and a diameter not exceeding 2 in. Clay pigeon shooting is more restrictive and barrels must be 12 bore or less (equivalent to 0.729 in.) diameter and between 26 and 32 in. in length. A shotgun fires a cartridge containing propellant, a wad and a number of pellets. The number of pellets depends on the size of the pellets, and these vary. When the cartridge is fired the propellant expands forcing the wad and subsequently the pellets out of the gun barrel. This emission can be seen in the simulation created in Fig. 1.

Fig. 1

A screenshot of the simulator showing the shot cloud having recently been emitted from the barrel and is highlighted in red at the top of the image

The pellets, depending upon their size and some other factors, leave the barrel at approximately 400 ${\hbox {ms}}^{-1}$ and spread as their distance from the gun increases, forming a cloud [13, 45]. This spread of the pellets can be controlled by the tightening of the barrel at the end of a gun, referred to as a choke. Figure 2 shows a 2D plate pattern representing the locations the pellets hit after 21 yards. This pellet cloud spread gives the shooter more room for error when aiming and shooting at a moving target. The cartridge can vary greatly; shot size and shot load weight are just two varying factors [11, 35].

Fig. 2

A pattern plate showing the variance in pellet distributions at a range of 21 yards

Clay targets can be broken by just one pellet; the target breaks more easily the more pellets successfully hit the target. The practitioner must shoot into a location which predicts where the target will be when the pellet cloud reaches that location, this allows for the flight time of the pellet cloud to the clay. This is referred to as “lead” and can be seen in Fig. 3. They are made to very exact specifications; the most common is “standard” and must weigh 105 g, be 110 mm diameter and 25–26 mm in height. Clay traps launch the clay pigeons. Traps can be modified to alter target launch speeds and trajectories.

Fig. 3

In order to be able to hit a moving target, it is necessary to aim in front of the target compensating for both speed of projectiles and the speed of the target in question. This correlates to “lead” and is dependent on a number of factors but mainly the distance to the target

2.2 VR simulators

There have been several VR clay pigeon simulators; these have been mainly created for use in arcades and shooting ranges. These have been rarely studied or investigated rigorously, and only manufacturer information provides context to the ability of these systems to improve task performance. Winchester Total Recoil [20] is a VR system that uses a Winchester 101 shotgun. There are various types of shooting that are emulated by this program, “Trap Master” is the main program to simulate clay pigeon shooting. Released in 1995, the technical specifications of this system are low relative to today’s capabilities, using dual 100 MHz Motorola chips with some dedicated graphics hardware. DryFire [22] is a home clay pigeon simulation product. The user’s own gun can be employed to fire at a dot projected on a wall. The ballistics model provided with this package allows for variable shot size, number of pellets and barrel choke size. However, as this system projects a dot onto the wall, this cannot provide the depth cues required for extrapolation into a real-world environment. ShotPro 2000 [43] employs lasers to detect the angle at which the gun is fired. It uses several projectors to display a shooting range scene in 2D to enhance the shooting experience. The user may use their own gun loaded with a special laser cartridge. Again 2D planar projection limits the practicality of this solution as not all cues are being delivered accurately in the training paradigm.

Bradley [9] developed a system which allowed for the visualisation of shotgun ballistics. Hardware was a limiting factor to the models proposed. The model used in this work is an adaptation, for efficiency, of the principal model proposed in this work whereby drag coefficients for pellets of different shapes and sizes are mapped to an analytical solution. Denton [21] created a 3D model and visualisation of a clay pigeon shooting range, this was created for a standard desktop monitor, and the ability to shoot targets was controlled with standard peripherals. As a result the cues that VR provide for this particular task were not integrated. Coulson [16] developed a system for tracking, in real time, the position and motion data of a practitioner for use in simulated clay pigeon environments. Standard commodity hardware provides this capability now for VR.

2.3 Serious games as training aids

In recent years, there has been increasing interest in applying computer gaming to health and rehabilitation. It is posited that incorporating computer games can provide a more engaging context and better motivation during practice [12]. Numerous global studies have concluded that playing serious games and computer games can have a positive effect on one’s hand-eye coordination and dexterity [2,3,4, 26, 44, 46].

With a specially simulated environment, VR could provide a fully controllable environment for training and development for a given task, many examples of this exist [39,40,41]. Controlling the virtual avatar using some peripherals, the user is able to interact with the virtual world. The position and orientation of the actuators in the virtual world can be affected by this control schema. However, as the user is moving his/her hands in a real environment, but experiencing the interaction between the virtual avatar and the virtual environment, there is a disconnection in the interaction between the physical and virtual elements.

It can be observed that within these VR-based systems, specific tasks with multimodal feedback (e.g. visual, tactile, audio) are designed in a bespoke fashion to engage the user effectively. However, the disconnection between the real world and the virtual objects still exists and often results in underestimation of distance, especially in large screen VEs [17]. This disconnection would affect the effectiveness of any training system in the development of motor skills. Performance measures of VR related tasks have however been shown to trend beginners towards experts during training in VEs [18]. It has also been shown that it is possible to transfer motor skills learned in VEs to real practice [17].

3 Ballistics and physics model

Ballistics of pellet flight have previously been measured to assist in the development and evaluation of non-toxic shot [24]. As part of this study, an extremely accurate statistical model of pellet distribution and flight has been developed for several different types of material, size and propellant. This is based on a representation of each pellet as a sphere travelling at either transonic or subsonic speeds.

To model the pellet cloud, parameters for every pellet within this cloud are generated. The pellet direction is determined by two angles $\theta$ and $\phi$. $\theta$ is the angle between the direction of the gun barrel and the direction of the pellet flight and is normally distributed around zero with a constant standard deviation for a specific gun [34]. $\phi$ is the angle of rotation from the vertical of the gun to the direction of the offset from the gun aim and is uniformly distributed. This function of pellet emission vector $\mathbf {v} = f(\theta , \phi )$ can be approximated via a statistical distribution and is demonstrated visually in Fig. 4.

Fig. 4

Pellet angle distributions. $\mathbf {v}$ is the vector of the pellet emission parameterised by the two angles $\theta$ and $\phi$, and $\mathbf {b}$ is the vector of the shotgun barrel facing direction from which $\theta$ and $\phi$ deviate

Each pellet has a diameter generated from a normal distribution based on the cartridge type used. This diameter determines the deceleration constant of the specific pellet. The deceleration constant of the leading (largest) and trailing (smallest) pellets are specified by the model and values between these two are linearly interpolated using the diameter of the pellet [25, 33, 34].

The time taken to travel a given distance is given by differential equations for modelling sphere motion. Equation 1 models the transonic (speeds between mach 0.5 and mach 1) and Eq. 2 the subsonic (speeds below mach 0.5) flight [24, 25].

$$\begin{aligned}&\frac{\delta v}{\delta t} = -kv^3 \end{aligned}$$

(1)

$$\begin{aligned}&\frac{\delta v}{\delta t} = -kv^2 \end{aligned}$$

(2)

where v is velocity, t is time and k is the deceleration constant.

As a clay pigeon simulation and training tool, it is very important that the ballistics are accurately modelled. Both the clay and the pellet flight must be statistically correct and the collision detection between the two exact. The pellet flight will be calculated using the model given by Compton [15] and Bradley [9]. The complexity of modelling the clay could vary greatly, but it should achieve the smooth motion that any experienced clay shooter would expect.

The number of pellets in a cloud can be anything up to 1100, dependent on the cartridge and pellet diameter being used, but is typically around 300 and calculating the location and intersections of the pellets will be required every frame. Evaluating this for high numbers of pellets with one or more targets is computationally expensive so efficiency is paramount to ensure real-time calculations.

To exploit the analytical solution to Eqs. 1 and 2, which give the flight time of a pellet as a function of distance travelled, we use Eqs. 3 and 4.

$$\begin{aligned} t= & {} \frac{1}{kv_0}(e^{kR}-1) \end{aligned}$$

(3)

$$\begin{aligned} t= & {} \frac{1}{v_0} R + \frac{k}{2} R^2 \end{aligned}$$

(4)

where k is the deceleration constant, $v_0$ is the initial velocity, R is the distance travelled by the pellet, and t is the time taken. Rearranged to give the distance travelled by the pellet from the subsonic and transonic flight time respectively, Eqs. 5 and 6:

$$\begin{aligned} R= & {} \frac{\ln (tv_0k+1)}{k} \end{aligned}$$

(5)

$$\begin{aligned} R= & {} \frac{-\frac{1}{v_0}+\sqrt{\frac{1}{v_0^2}+2kt}}{k} \end{aligned}$$

(6)

To determine which equation to use, the point at which the pellet passes from transonic to subsonic velocity is needed. This is also derived from differential equations modelling sphere motion [6], shown in Eq. 7.

$$\begin{aligned} v = \frac{v_0}{1+kv_0R}. \end{aligned}$$

(7)

3.1 Pellet drag coefficient

The major parameter of the drag coefficient is the Mach number, which is the ratio of the speed of the pellet sphere to the velocity of sound in the surrounding medium. From previous experiments, it has been shown that a sphere’s drag coefficient C obeys different force laws depending on its velocity for a Reynolds number of $5\times 10^4$ [5, 6, 10, 14]. It has been shown that at subsonic velocities below Mach 0.5, the drag coefficient is constant and a square law of air resistance [38]. This is obtained from Eq. 8:

$$\begin{aligned} F = -\frac{1}{2}\rho Av^2C \end{aligned}$$

(8)

where F is the drag force, A is the cross-sectional area of the sphere, $\rho$ is the atmospheric density, v is the instantaneous velocity of the sphere, and C the drag coefficient.

The force of air resistance is proportional to the square of the magnitude of the velocity. At transonic velocities between Mach 0.5 and Mach 1.4 the drag coefficient is approximately proportional to the velocity. At higher supersonic velocities (greater than Mach 1.4) the drag coefficient becomes approximately constant again.

3.2 Target physics

The flight of the clay is modelled after clay pigeon dynamics. The model uses a drag factor that reduces the forward velocity over time. In addition, gravity acts downwards on the clay and a lift factor reduces this as the clay begins to descend. This gives the effect of the clay hanging in the air as it reaches the top of its flight path. These factors can easily be changed for different effects and are shown in Fig. 5.

Fig. 5

A clay target is affected by a constant drag factor and gravity when moving upwards, when the clay starts to descend lift factor is introduced which increases the “hang time” of the clay. These are variables which control the realistic behaviour of the clay target

4 Subjective experiment

The psychophysical experiment outlined in this section intends to validate the efficacy of virtual reality training when applied to complex Ocular-Motor tasks and compare whether this efficacy remains true for experts versus non-experts to the task.

4.1 Design

The independent variables for the research are the following: guidance and information provided and expert or non-expert. These are separated by group for whether an Aim Assist tool is provided (A) and whether the participant has had at least two years or prior experience with the task (E). The dependent variable being monitored is the performance of the participant. Specifically performance metrics gathered are the total number of hits scored out of 50 possible hits. Additionally, a measure of the closest a pellet came to a target in the event of a miss was recorded for each shot taken.

This research is to be tested across two conditions, control and treatment. These groups represent, respectively, either the presence or absence of guidance in the virtual reality simulator. The independence of the groups means a participant placed in the control group conducts two runs of the simulator training, both without guidance. The treatment group will be exposed to guidance in the first run and receive the same conditions as the control in the second run. The approach of independent groups, and between-subject designs, where the participant is only exposed to one condition, was employed in response to the increased bias, and confounding factors presented in the alternative design of within subject design [23].

The null hypothesis is given as $\mathbf H _0^{\prime }$, that all conditions are equal under testing ($\mathbf H _0^{\prime }:\pi _i=\frac{1}{2}$). The alternative being that not all the conditions $\pi _i$ are equal. This is considered to be that “guidance and information via the virtual reality simulator does not affect the performance of the task”. A number of alternative hypotheses were considered based on the literature:

$\mathbf H _a$: the control and treatment conditions would produce different results.

$\mathbf H _b$: experts and non-experts will produce different results.

$\mathbf H _c$: guidance and information provided would influence usability.

Usability Questionnaires, which are used by 38% of studios [36], are used to evaluate the system’s usability. Specifically, the study uses the IBM PSSUQ [31]. The questions used in the study are as follows:

• 1. Overall, I am satisfied with how easy it is to play this game

• 2. It was simple to play this game

• 3. I could effectively complete the objectives and challenges

• 4. I was able to complete objectives and challenges quickly

• 5. I was able to efficiently complete objectives and challenges

• 6. I felt comfortable using this system

• 7. It was easy to learn to play this game

• 8. Whenever I make a mistake in the game, I recover easily and quickly

• 9. The organisation of information on the game screens is clear

• 10. The interface of this game is pleasant

• 11. I like using the interface of this game

4.2 Participants

A total of 31 non-expert participants (11 male, 20 female) and 26 expert participants (20 male, 6 female), for a total of 57 (31 male, 26 female) participants took part in this experiment. The expert set E was deduced based upon whether a participant had more than two years prior experience with the hobby of clay pigeon shooting. This relied upon each participant self-reporting their prior experience. Testing of their ability was not conducted. Participants reported no hearing difficulties and normal or corrected-to-normal vision. The age range of participants was between 19 and 66, with an average age of 35.

4.3 Materials

A significant number of materials and parameters have been used for the user study, and they are discussed here.

Fig. 6

The design and printed versions of the vive controller holder. This is based on a design by Thingiverse user BManx2000 [8]. This has been modified for more adjustable gun stock height with a stock that does not extend quite so far to ensure that the head-mounted display does not interfere with the stock and inhibit being able to look down the virtual barrel. The final modification was to thicken the connection from the stock to the rear controller as this portion would significantly flex making it unusable, even when printing in PLA

The simulation was developed using Unity version 2017.3.0f3 Personal edition. Steam’s VR plugin version 1.2.3 was used to interface with the HTC Vive for VR support. Binaural headphones were used for audio delivery. The ballistics model relied upon the integrated Unity Physics engine. Physics calculations are not constrained to a fixed update but are run on a frame-by-frame basis. The reason for this is due to the tunnelling effect from high-speed objects, they appear to teleport. Collision detection made use of the continuous detection feature provided by Unity Physics to negate the influence of this effect. Calculations are modulated by the frame-rate and as such are invariant to this.

A game scene was modelled to scale by designing the game elements to fit a scale of one metre to one Unity unit. Six clay traps were placed within the scene with varying distances, height and shooting angles to simulate a typical clay shooting ground. Within the scene, the shotgun was attached to the right controller inheriting its position and rotation. The left controller was attached to the 3D printed gun stock to provide aiming support and simulate holding the barrel of a shotgun. The 3D printed gunstock can be seen in Fig. 6.

An Aim Assist feature was created to show the visual lead to the user which is shown in Fig. 7. This information and guidance acted as another independent variable for the experiment design, and set A represented those participants provided with this Aim Assist feature in the first round of training. To correct for common misalignment, canting and incorrect mount errors from beginners, the Aim Assist tool provided a laser sight from the end of the shotgun to indicate the direction of the barrels in 3D space. The Aim Assist tool also calculated the correct place to shoot in front of the target by $\mathbf {l} = \mathbf {c} + \mathbf {c_v} \cdot t$ where $\mathbf {l}$ is the 3D position of the predicted lead intersection point, $\mathbf {c}$ is the 3D position of the clay target, $\mathbf {c_v}$ is the instant velocity of the clay target at the current frame, and t is the time taken for the shot to travel the distance to the target. t is evaluated from Eqs. 3 and 4. This predicted lead was used to visually display where a shot was necessary to be fired at in order for the pellets to collide with the clay disc. The ordering of A for the experiment was assigned randomly for set $\mathbf E =0$; the same assignment order of A was used for set $\mathbf E =1$. A screenshot of the deployed game is shown in Fig. 8.

Fig. 7

A screenshot of the Aim Assist feature active. A red laser is visible from the barrel of the shotgun which clearly indicates where the user is aiming. The green circle in front of the orange clay displays the visual lead, showing the user where to aim and at which position the actual target will be when the shot stream would reach it given current velocities

Fig. 8

A screenshot of the test Virtual Environment showing the shotgun, environment and traps. The traps are colour coded to provide a visual indication of which trap will fire a clay next. Red indicates the trap will not fire, green the opposite

The hair trigger on the left controller allows the user to manually fire a clay. There is a timer set to 3.5 seconds to stop the user from firing multiple clays at once. Once 50 clays have been fired, the application will quit. The hair trigger on the right controller is used to fire the shotgun. The shotgun can be fired up to two times before a timer is set to simulate reloading. The first shot instantiates the pellet stream from the top barrel of the shotgun, and the second shot instantiates the pellet stream from the bottom barrel. The menu button on both controllers displays the main game menu which allows the user can toggle the shot camera, Aim Assist, detail settings and brightness. This menu can be seen in Fig. 9.

Fig. 9

A screenshot showing the menu for set-up of the conditions for each trial. Graphics and brightness were always set to high for consistency. Aim Assist was turned on or off as required by the random assignments for the training phase and was always off for the testing phase

One of the main deficiencies of previous shotgun simulators has been their inability to deliver accurate depth cues through stereo rendering. VR provides a great solution to this; however, it has been shown that subjects underestimate depth cues provided by VR [1, 17]. Nonetheless, it has been shown that discrimination of depth order is very possible and that the variance of distance misjudging when applied to the velocities of a shotgun cartridge would only influence the outcome on rare occasions or for very distant targets outside of a shotguns effective range. The stereo render targets for various phases of the processing pipeline are shown in Fig. 10.

Fig. 10

(t): Showing the stereo nature of the binocular rendering provided for the VR headset. This particular image shows the shotgun mounted for a left-eyed shooter, with the gun aligned down the left eye’s render target. This has no barrel distortion applied to compensate for the lenses inside the VR headset yet. (b): This image shows the stereo render for a right-handed shooter, with the barrel distortion applied

4.4 Procedure

Participants were all asked to familiarise themselves with the controls of the system. This process was controlled to 3 min within the system where they could ask questions about the control scheme and test this out for themselves. Participants were then randomly assigned to a group. The group determined whether they received the training of Aim Assist ($\mathbf A =1$) or the control, no Aim Assist ($\mathbf A =0$). This only influenced exposure in the training phase. The testing phase was group invariant, and $\mathbf A =0$ for all participants in this testing phase. This procedure is shown in Fig. 11. Each round of training and testing comprised of 50 clay targets to be fired at with a limit of two shots allowed to be taken at each target.

Fig. 11

Procedure for the experiment highlighting the control and treatment for each participant and how this applied to each round of training they were exposed to

The set-up for the experiment is shown in Fig. 12.

Fig. 12

The experiment was set up in such a way that the participants could put on the equipment and become accustomed to the environment for 3 min in advance of the sessions. This was consistent across all participants. The participants were all asked to stand, and enough room was given to be able to explore the environment in 360$^\circ$

5 Results

Results for every participant were captured by non-invasive means and programmatically so as to ensure that these performance measures were not known to the participant. Two main performance measures are evaluated: number of successful hits where at least one pellet connected to the target (H) and the minimum pellet distance (d) so it is possible to identify how close the participants are on average to the target in the case of a miss. This is useful in the context of misses as it helps to elucidate information as to the context of the miss. In some cases, the accuracy can be good and in fact the target can fly through the shot cloud.

The minimum pellet distance is recorded for each shot taken during testing. This is the closest point that any single pellet reaches in relation to the clay the user is targeting. This measure allows for shots recorded as a “miss” to be evaluated giving a more accurate perception of the user’s level of skill. The minimum distance is calculated every frame using Eq. 9. However, being constrained to a calculation every frame means that the closest distance in continuous time may not be represented by this performance measure, given the simulation is in discrete time steps. Interpolation, however, can improve this metric, but this increases computation time and results in performance implications. The accuracy improvement yielded by interpolation was deemed to be insignificant in regard to metric accuracy. However, when considering this variable for the training and testing conditions it should be considered that the calculation considers distance to the centre of the clay of a fixed diameter (so the radius can be subtracted), that the values are averages over all shots and that the frame-by-frame calculations are not interpolated so are not true minima if the simulation was representative of continuous time.

$$\begin{aligned} d = \min _{\forall p \in P} \sqrt{ \sum \nolimits _{j}^{3} (\mathbf {c_j} - \mathbf {p_{ij})^2}} \end{aligned}$$

(9)

where d is the minimal distance that a pellet in a shot cloud reaches to the centre of a clay, $p_i$ is the i’th pellet in the set of pellets P, c is the position of the centre of the clay target, and j is the index in Euclidian space.

The results are presented in totality, for each participant in Table 1. These are then analysed in Sect. 6.

Table 1 Results from the testing phase

6 Analysis

Analysis is conducted via a between-subjects repeated measure ANOVA using a $2 \times 2$ (A $\times$ E) factorial design. A denotes the treatment received in the first round of training, Aim Assist turned on or off. This analysis was conducted for $N=57$, where A of $0, N=29$ was composed of participants receiving no treatment in the initial round and A of $1, N=28$ was composed of participants receiving treatment via the Aim Assist. E denotes the a-priori knowledge of the task by the participants for $N=57$, where E of $0, N=31$ was composed of participants who had little or no prior knowledge of the task and E of $1, N=26$ was composed of subjects who had at least two years of experience in the task. Descriptive statistics are presented in Table 2.

Table 2 Descriptive statistics for the variables A and E showing means ($\mu$), standard deviation ($\sigma$) and number of participants (N) delineated by performance measure in each phase of testing ($H_{\mathrm{tr}}$, $H_{\mathrm{te}}$, $d_{\mathrm{tr}}$ and $d_{\mathrm{te}}$)

6.1 Overall results

Tests of the between-subject effects were undertaken as follows:

Aim Assist, (A): The main effect of A was significant for the dependent variable of hits scored, H: $F_{1,56} = 38.55$, $p <0.001$, indicating a difference in results when the Aim Assist gamified tool is applied to the didactical process. The observed power of this finding, using $\alpha = 0.05$ was 1.0. The dependent variable, number of hits recorded, is recorded at an average of 19.38 for $H_{\mathrm{tr}}, \mathbf A = 0$ and 31.18 for $H_{\mathrm{tr}}, \mathbf A = 1$. This effect persisted through the treatment and into the controlled second exposure $H_{\mathrm{te}}$. For this, number of hits is recorded at averages of 18.83 for $H_{\mathrm{te}}, \mathbf A = 0$ and 26.36 for $H_{\mathrm{te}}, \mathbf A = 1$.

Fig. 13

Interaction of A showing the effect the training treatment has on the performance measure H across the control exposures

The main effect of A was significant for the dependent variable of the average minimum pellet distance from the target, d: $F_{1,56} = 34.494$, $p <0.001$, indicating a difference in results when the Aim Assist gamified tool is applied to the didactical process. The observed power of this finding, using $\alpha = 0.05$ was 1.0. The dependent variable, average minimum pellet distance, is recorded at an average of 1.07 for $H_{\mathrm{tr}}, \mathbf A = 0$ and 0.86 for $H_{\mathrm{tr}}, \mathbf A = 1$. This effect persisted through the treatment and into the controlled second exposure $H_{\mathrm{te}}$. For this, d is recorded at averages of 1.05 for $H_{\mathrm{te}}, \mathbf A = 0$ and 0.9 for $H_{\mathrm{te}}, \mathbf A = 1$.

Fig. 14

Interaction of A showing the effect the training treatment has on the performance measure d across the control exposures

These results indicate acceptance of $\mathbf H _a$: the control and treatment conditions would produce different results. These effects can be seen in Figs. 13 and 14.

Experts, (E): The main effect of E was significant for the dependent variable of hits scored, H: $F_{1,56} = 28.611$, $p <0.001$, indicating a difference in results between beginners to the task and experts with prior knowledge. The observed power of this finding, using $\alpha = 0.05$ was 0.999. The dependent variable, number of hits recorded, is recorded at an average of 21.97 for $H_{\mathrm{tr}}, \mathbf E = 0$ and 29 for $H_{\mathrm{tr}}, \mathbf E = 1$. This effect persisted through the treatment and into the controlled second exposure $H_{\mathrm{te}}$. For this, number of hits is recorded at averages of 18.19 for $H_{\mathrm{te}}, \mathbf E = 0$ and 27.69 for $H_{\mathrm{te}}, \mathbf E = 1$.

Fig. 15

Interaction of E showing the effect the training treatment has on the performance measure H on the sets of experts and non-experts

The main effect of E was significant for the dependent variable of the average minimum pellet distance from the target, d: $F_{1,56} = 4.767$, $p <0.05$, indicating a difference in results between beginners to the task and experts with prior knowledge. The observed power of this finding, using $\alpha = 0.05$ was 0.573. The dependent variable, average minimum pellet distance, is recorded at an average of 1.00 for $H_{\mathrm{tr}}, \mathbf E = 0$ and 0.92 for $H_{\mathrm{tr}}, \mathbf E = 1$. This effect persisted through the treatment and into the controlled second exposure $H_{\mathrm{te}}$. For this, d is recorded at averages of 1.00 for $H_{\mathrm{te}}, \mathbf E = 0$ and 0.94 for $H_{\mathrm{te}}, \mathbf E = 1$.

Fig. 16

Interaction of E showing the effect the training treatment has on the performance measure d on the sets of experts and non-experts

These results indicate acceptance of $\mathbf H _b$: experts and non-experts will produce different results. This confirms that skill in the real world translates into skill in the virtual world in this task. It also suggests that the virtual task is a good physically based representation of the task in reality. These effects can be seen in Figs. 15 and 16.

Aim Assist $\times$ Experts, (A $\times$ E): The main effect of A $\times$ E was not significant for the dependent variable of hits scored, H: $F_{1,56} = 1.256$, $p =0.267$, indicating no separation in the affect of the Aim Assist tool between beginners to the task and experts with prior knowledge. The observed power of this finding, using $\alpha = 0.05$ was 0.196.

Fig. 17

Interaction of A $\times$ E showing the effect the training treatment has on the performance measure H across the control exposures on the sets of experts and non-experts

The main effect of A $\times$ E was not significant for the dependent variable of the average minimum pellet distance from the target, d: $F_{1,56} = 0.000$, $p =0.99$, indicating no separation in the affect of the Aim Assist tool between beginners to the task and experts with prior knowledge for the dependant variable d either. The observed power of this finding, using $\alpha = 0.05$ was 0.05.

Fig. 18

Interaction of A $\times$ E showing the effect the training treatment has on the performance measure d across the control exposures on the sets of experts and non-experts

Table 3 Descriptive statistics for the usability ratings showing means ($\mu$), standard deviation ($\sigma$) and number of participants (N) across between-subject factors E and A

When aiming to assess whether the effect of either factor influences the other, it is shown that it does not, indicating that when the Aim Assist gamified tool is applied to the didactical process, the effect it provides is invariant of whether the subject is an expert or a beginner. These effects can be seen in Figs. 17 and 18.

6.2 Usability results

Usability was analysed post hoc and delineated by between-subject factors to try to investigate $\mathbf H _c$: guidance and information provided would influence usability. This was considered for a number of reasons:

• success could influence usability ratings

• the training provided may have (in)directly influenced usability

• prior knowledge of the task may influence usability due to the realism of the simulation

Usability ratings are shown per group in Table 3.

The main effect of A was not significant $F_{1,56} = 0.006$, $p = 0.939$, indicating no difference in the usability of the system across A. The performance measure, usability calculated from the SUS, was recorded at an average of 81.79 for the Aim Assist group compared to the non-Aim Assist group at 81.72. The main effect of E was not significant $F_{1,56} = 0.718$, $p = 0.401$, indicating no difference in the usability of the system across E. Usability was recorded at an average of 82.45 for the non-expert group compared to the expert group at 80.92. The main effect of A $\times$ E was not significant $F_{1,56} = 2.014$, $p = 0.162$, indicating no difference in the usability of the system can be elucidated between E and A. These results indicate rejection of $\mathbf H _c$: guidance and information provided would influence usability. This interaction can be seen in Fig. 19.

Fig. 19

Interaction of A $\times$ E showing the effect the training treatment has on the usability of the system across the sets of experts and non-experts

7 Conclusion and future work

We demonstrate an analysis of the pedagogic function of VR technology, in the context of a serious game, to a complex motor task (clay pigeon shooting). This didactic method is evaluated across two independent variables, guidance provided via augmented virtuality and efficacy on experts and beginners. This paper introduces a state-of-the-art simulation of clay pigeon shooting in a VR environment with tangible and multimodal-based interaction methods. The simulation relies upon pre-validated ballistic and target physic models derived from the literature. The information and guidance that VR allows us to convey in the context of this complex motor task is shown to have a significant difference on the recorded performance measures of this task. Beginners with guidance perform as well as unguided experts (see Fig. 17); the learning from this guidance persists across repeated measures and this is verified against a control group (see Fig. 13). Additionally, the guidance given does not just improve the performance of beginners to the task, but improves the performance of experts too (see Fig. 15).

As presence and immersion has been shown to be a factor of cross-modal influences [27, 29], future work will investigate the effect of spatial sound in conjunction with this training task, as performance in locating targets can be attributed to not just the visual domain but the acoustic too [19]. To be a truly representative simulator, other external influences such as wind and weather need to be considered as intrinsic into both the ballistic and target physic models. Additionally, further work will investigate the practicality of this result being extended into the real world. We aim to investigate whether performance and skill developed in the virtual world extends into the real world. If so, this work will investigate whether these transferrable skills are a one-to-one relationship or harbour some other intrinsic relationship. This work has already gone some extent to show that a one-way relationship exists in that experts in the task perform better in this simulation than beginners to the task. Can learning from this process be extrapolated out to a real-world condition? This of course may be dependent upon the domain of the skill, familiarity and a-priori conditioning, and research should aim to elucidate the influence of each of these variables.