1 Introduction

Traditionally, the early years of a medical school consist of building foundations in the basic medical sciences. With the evident benefits of early introduction of clinical skills to bridge the transition from theory to practice, many medical schools have since incorporated clinical skills training within the pre-clinical curriculum [1,2,3].

Clinical skills taught in these early years classically consisted of basic generic clinical skills such as, rapport building in a doctor-patient relationship, history taking, counselling and physical examination [4]. Training sessions are conducted using standardised patients (SP) simulating a disease to create a safe learning environment for the students to practice on. An SP is a normal person, usually an actor, trained to act as a patient in a clinical skills session. In the case where a paid actor is not used, peers practice usually on each other and effectively act as SPs for one another.

In the scope of physical examination, there are limitations as to the variety of signs an SP can simulate. Maintenance of a bank of SPs with stable physical findings is time-consuming and costly [5, 6]. This requires previous identification and monitoring of SPs with similar physical findings [4]. In preference to experiencing the ‘real thing’, some students suggest incorporating videos or computer simulation into teaching [1].

Simulation-based learning with computer simulations and physical model is innovative but removes the patient-doctor human interaction element, resulting in little to no empathy skills training, which is crucial for the doctor-patient relationship.

Real patients, on the other hand, present several issues. The demand for these patients as subjects for physical examinations has far exceeded supply for even undergraduate clinical placements let alone pre-clinical students for real patient encounters [7] especially in countries like Malaysia with over 40 medical schools. Time constraints prevent optimal learning in the clinical setting [2, 4]; patient priorities take precedence where quality of teaching is inconsistent. Patients are often too sick and with acute conditions, at times with multiple problems causing distraction or distorted results during examination. Having yet developed professional empathetic attitudes towards these signs of interest, patients are at risk of feeling undignified as a novice’s learning aid [8]; these patients may become conflicted and refuse their requests to learn.

Augmented reality (AR) holds promise in countering these limitations by augmenting clinical findings onto an SP or physical model in our shared space. AR technology, “A system that supplements the real world with virtual (computer-generated) objects that appear to coexist in the same space as the real world” has been an advent for industries for years and now made popular to the mainstream audience with the game Pokémon GO™. Recent advances enabled AR technology to be readily accessible on applications on smartphones and tablet devices via various AR software-development kits (SDK) such as Vuforia, Layar and ARtoolkit - many of which are open source and free-to-use.

This paper contributes to the current implementation of AR in teaching clinical skills at the early years (pre-clinically) of the medical curriculum by discussing a review of literature on AR technologies that have been developed for clinical skills education and their associated problems. The major contribution of this paper is to highlight a pilot project developed by the present authors to aid authentic learning of clinical skills in early years of medical education by simulating the clinical environment and how this review of literature has informed and guided this novel AR application development.

2 Methods

A literature survey was conducted to ascertain the current use of AR in the delivery of clinical skills education, their methods and associated challenges.

A search on databases DBLP, OvidMedline, CINAHL plus, Embase, Scopus, Web of Science, Google Scholar and ResearchGate were performed using the search terms ‘augmented reality’ OR ‘AR’ AND ‘clinical skills’ OR ‘physical examination’ OR ‘competencies’. Exclusion criteria included studies with projects beyond the early year curricula as the focus of the present paper was towards clinical skills training for pre-clinical students.

A detailed table of the review matrix has been included as an Appendix (Table 1).

Table 1. Studies of application of augmented reality technologies in clinical skills education
Full size table

3 Results

3.1 AR in Clinical Skills Education

Augmenting Sounds in Physical Examination.

Early works by Mckenzie et al. [9] in 2004 demonstrated pre-recorded pathological heart and lung sounds that play through headphones when a stethoscope is placed over designated areas of auscultation on a mannequin, tracked by a sensor.

In a later work, pre-recorded fine crackles and real breath sounds were combined and played out through a modified electric-stethoscope [10]. To play the sounds over specified locations on chest auscultation, the SP cooperates to signal the correct timing of respiration clicking the controller when the student auscultates over that location.

Similar to the previous study, another study devised an innovative method of tracking where the stethoscope is placed, based on ECG signals to play augmented heart murmur sounds on the SPs correctly [11].

AR Mobile Learning.

With readily available resources, AR developer kits and mobile devices, designing an AR application has become easily accessible and cost-effective for educators. In one project, Layar, as an AR developer platform, was utilized to develop a mobile AR heart murmur simulator for the Android system [12]. A shirt with imprinted AR tracking markers was worn and scanned to play heart murmurs through an electronic stethoscope and display visualization of the audio data in AR on the device.

Another project also opted for Layar, as their AR platform for its cost, support, simplicity and availability [13]. A range of equipment was tagged so students could scan the tags using their smartphones or tablet computers, pressing ‘buttons’ in AR that linked to various multimedia resources about the equipment for the students to explore.

Enhancing Engagement with a Mixed-Reality Environment.

AR can be integrated into clinical simulations with existing physical models or modified ones to create a more immersive environment.

In one study [14], an AR headset, Google Glass, was used to incorporate video into a clinical simulation of acute asthma exacerbation for training of pre-licensure nursing students [14]. The video, pre-recorded with an SP, is played in their field of vision while they performed clinical skills onto a mannequin. Realism is markedly enhanced in creating an authentic environment for the students, increasing engagement with the mannequin as a patient, thus motivating problem-solving skills.

For an AR clinical simulation, Daher designed a unique interactive Physical-Virtual Head that is interactive to touch [15]. For visual effects, it uses projection of imagery onto the shell of a human head shape. Infrared cameras and lights track fingers that touch the shell and send graphic changes to the imagery projected. This creates a mixed-reality environment of a visually augmented patient within a shared physical space.

Incorporating AR Feedback.

In a case study of the efficacy of real-time feedback [16], the study presents a novel pedagogy of clinical breast examination on a mannequin in AR. A video was streamed from a webcam input of the user’s hands upon the augmented physical breast model, transformed into a virtual human. The virtual human was able to speak with pre-recorded human speech and was interactive to the learner’s touch based on sensor data. Real-time feedback is visualized in AR with construction of touch maps of the pressure and coverage of the learner’s palpation. A pattern-of-search map indicates the sequence of palpation during the examination.

Gap in Literature.

The previous discussion highlights that though various specific AR applications for a particular skills training in one area (e.g. heart murmur or breast examination) exist, there is a need for one that acts as a common AR tool to aid clinical skills learning in all systems and which can simulate all the various signs on a SP or a patient superimposed on the exact location where such signs are manifested. Traditionally this is what early year preclinical students are taught, (in contrast to specialist training in clinical years) and for which the above AR application, as discussed may be applicable.

3.2 Clinical Augmented Reality Objects in Physical Examination (CAROPE)

This review of literature has led to incorporating some of the design process of AR applications to develop a clinical skills aid for clinical skills education in training early years’ medical education.

The present authors have developed a pilot AR project, called Clinical Augmented Reality Objects in Physical Examination (CAROPE) that augments actual images of clinical signs superimposed on the skin. A simple design of readily available resources to demonstrate how physical signs can be presented visually with AR has been used.

Based on a standard clinical examination textbook [17], CAROPE began as a repository of signs associated with the gastrointestinal (GI) system. The GI system was chosen as a pilot due to its great variety of clinical signs encompassing various regions of the human body. One of the authors, a medical graduate familiar with the curriculum of the same university, selected the applicable signs.

The images of clinical signs available were collected from pictures available for non-commercial use from Open Educational Resources (OER) under Creative Commons licenses. Using photo-editing techniques for the present CAROPE development, these photos were enhanced and the transparency of the images was set. Following this, a 3D editing program was used to create 3D objects, applying these images as an overlay.

An AR application was elected based upon its simplicity and ease of use. Utilizing a common AR applications’ image recognition technology, photographic image markers were used to launch the 3D objects within a mobile phone that were previously tagged. Once the marker is identified, the object appears on the screen of the mobile phone once the AR application is launched.

These markers are printed on stickers and adhered onto designated areas of the body to display these signs when scanned. Some are printed with information as to where and how it should be placed in relation to landmarks such as the umbilicus (belly button). For example, when the skin surrounding the umbilicus is augmented to display a bruise over it, the student can now describe it in detail during ‘Inspection’ and would be expected to perform ‘Palpation’ with care (Fig. 1).

Fig. 1.
figure 1

Using CAROPE: Left: marker/trigger (black box) is scanned (red line). Right: Cullen’s sign appears as an AR visual overlay on abdomen of a mannequin. (Color figure online)

Full size image

To add a visual guide to ‘Palpation’ of the abdomen, the AR application is utilised to create a ‘magic lens’, similar to Blum et al.’s concept of a ‘magic mirror’ [18]. This serves to indicate to the user the precise location of underlying organ visually when the hand of the user is placed over it (Fig. 2). The mobile device camera acts as a ‘magic lens’. For its position to be precise, the coordinates and size of the 3D object are adjusted to match true-to-life measurements and proportions. Akin to body painting, the authors hope that this can facilitate mental visualisation of positions and locations of the underlying organs of the different regions of the abdomen.

Fig. 2.
figure 2

A screenshot of the mobile device illustrating the magic lens concept to guide palpation of the liver on a mannequin

Full size image

4 Discussion

Blending of Realities.

AR has been well documented for its application in medical education [19]. Its ability to blend the real and virtual could, especially for basic clinical skills, bridge the gap we see here between pre-clinical and clinical years by providing a training ground for novice students. As described by Vaughn, with a simple addition to the scenario - the sights and sounds of a real patient in AR, students felt more ‘connected’ with the mannequin [14]. This breaks down the boundaries of the disbelief simulations usually come with.

Students are motivated to think and learn actively in an AR setting. Unlike conventional teaching with the unanimated mannequins, students using AR can build interpersonal and communication skills as they learn empathy through the augmented encounters. By integrating augmented signs into the practice of physical examination, students can anticipate pathology they should be aware of and be able to identify and describe them accordingly. This contrasts with the current pedagogy of rote memorization within the pre-clinical years.

Supplementing Current Methods of Clinical Skills Training.

On top of the benefit of peer physical examination in which students become more interested when the hands-on skill requires personal involvement, this can be further enhanced in AR as students can experience being their own learning subjects (SPs) by simulating signs onto each other.

In AR applications, such as in CAROPE, signs can be simulated without the need of disrobing and this can aid in the examination of sensitive areas such like the groin and genitals. However, it is true that discomfort may still exist for some students even when fully clothed. In the study which used a t-shirt with markers, the study reported that students felt quite exposed and uneasy wearing the shirt and being scanned by others [12].

Encouraging Self-directed Learning.

As described in two studies, mobile learning enables the students to learn at their own convenience. It could be a form of distance education where the resource becomes more accessible and applicable in different contexts.

In Peyton’s four-step approach of learning, medical curriculum concentrates on first two steps, ‘Demonstration’ and ‘Deconstruction’ of a skill by the clinical skills instructor [4]. By facilitating practice with AR, learners could find it easier to proceed to the next step of ‘Comprehension’ and followed by ‘Performance’. Real-time feedback in the form of interactivity and live mapping of performance can be greatly enhanced with AR technology, resulting in improvement of the learner’s performance comparable to an experienced clinician and the ability to perform independently [16].

Easily Accessible Resources.

Like two studies, we found that integrating our educational content with AR technologies was a low-cost innovation due to the numerous available resources and the students’ ability to navigate new technology easily. Students entering medical school presently have had experience using the Internet for both learning and social activity; these millennials have possessed advanced fluency in technology for most of their lives. Little training would be required for them to integrate most new applications of technology in their education.

In our project, we applied this into a readily-made AR platform - the application was designed for businesses and creatives to present 3D models in the real space with no pre-knowledge in programming. However, in order to effectively supplement live demonstration during classes, some training is necessary for clinical skills instructors [13] for its full integration.

Technical Limitations.

Technical difficulties are a recurring theme in the application of AR technology. Frequently, scanning problems, Internet connectivity and compatibility are cited to be among the most common issues [13]. As for headsets, unfamiliarity with the device, short battery life and limited number of devices become limiting factors of their use [14].

When mobile learning is used, the learner is unable to use both hands to perform examination when holding a device to view the augmented form. When the mobile phone becomes the interface for AR, it lacks the immersive quality of an AR headset. In CAROPE, students need to close a displayed sign and scan again for the next marker. Mobile phone AR applications may be more accessible for the masses but may not provide the seamlessness required for realistic blending of realities.

As for our current work, the visualization of the sign is difficult to be seamlessly tracked when markers move or change in appearance. In our development of CAROPE, development of the project required experience in 3D modelling and photo editing software. As the objects are set in a definite size and position, not every augmented sign is accurately displayed for every body shape, size and skin appearance without adjustment within the application.

Further Works.

It is suggested that with expansion to other clinical signs, the AR system could become an integral part of clinical skills education [10]. Further research is recommended to determine how AR can enhance simulation-based learning [14].

AR’s potential in realistically augmenting SPs in clinical examinations needs to be evaluated [12]. To determine its true integration into the curriculum, measurement of independent usage by the students based on traffic of the application is possible.

In comparison to a fully virtual environment, it is argued that a mixed-reality interaction is important for an appreciable transfer of training with regards to patient interaction [9]. Further works are needed to compare augmented and virtual reality and each of their applicability to clinical skills pedagogy.

Limitations of Current Studies.

Most studies are limited by small sample studies as AR has not been integrated in full-scale but is still under development.

5 Conclusion

Many medical schools have incorporated clinical skills in the early years of medical curriculum. With a limited range of clinical signs simulated by SPs and issues with real patients, AR can helpful in assisting the learning of clinical skills. Our literature review has reiterated how AR holds potential in creating a mixed-reality environment conducive for active learning [20] with a few studies that utilised AR in clinical examinations per se. To fill this gap, the authors have introduced CAROPE, a simple AR application easily accessible for use through current available resources, to aid clinical skills pedagogy by visualising signs and guidance to palpation.

With AR, CAROPE, especially through recreating some of the skin manifestations of clinical signs, can potentially be a helpful companion for preclinical curricula, rather than as a specialist training. This AR, unlike that in a fully virtual reality setting, simultaneously, along with clinical examination, engages leaners in a shared space allowing development of communication skills through human interaction to simulate an authentic clinical environment as happens in a clinical/hospital setting. Further work is necessary in its development and the evaluation of its role in clinical skills education.