Introduction

Surgical training programs are beginning to rely more heavily on simulation to supplement traditional teaching methods [11]. Simulation-based training allows trainees to practice skills in a low-pressure environment, without risk to patients, and allows trainees to make and learn from mistakes before they occur in surgery [13]. Previous research has shown that simulation can be highly effective for learning technical and nontechnical surgical skills [3, 4, 9, 14–17] and that the skills learned can transfer to clinical settings [5]. Furthermore, simulation-based training can shorten the learning curve, enhance later intraoperative learning [8, 14], and improve patient outcomes [19]. For example, one study has shown that general surgery residents who have undergone simulation training for laparoscopic inguinal hernia repair have had higher intraoperative performance scores, fewer postoperative and intraoperative complications, and less in-hospital stay compared with residents who used a conventional surgical training curriculum [19].

Despite clear advantages, the costs associated with simulation-based training are a major limitation [19]. Sufficient financial support is required for infrastructure, operating materials, support staff, and faculty time [18]. Although research suggests that the cost is offset by enhanced learning and decreased clinical complications [2], there is a clear need for better cost-effectiveness studies [19]. Zendejas and colleagues [18] conducted a systematic review and found that only 1.6% of studies provided any cost comparison when examining simulation-based training methods compared with other instructional methods in medical education.

There is a paucity of data regarding the costs of using simulation in orthopaedic surgery. For the past 5 years at the University of Toronto, a novel competency-based curriculum has been implemented as a pilot, functioning as a parallel stream of training for residents in the Division of Orthopaedic Surgery [1, 6]. In each of the first 3 years of the pilot program, three of 12 residents who matched to the residency program at the University of Toronto were selected into the competency-based curriculum stream. In its fourth year, five of 12 residents entered the competency-based curriculum. In its fifth year, a decision was made to convert all incoming residents to the competency-based curriculum stream. The competency-based curriculum makes extensive use of high- and low-fidelity simulation as a fundamental element of teaching and assessment. Standardized patients are used for teaching clinical skills, low-fidelity models are used for basic technical skills, and high-fidelity cadaveric models are used for higher order skills. All trainees start residency training by taking part in a 1-month long “Boot Camp,” in which the basic surgical skills and knowledge relating to surgical residency are learned. Trainees then move on to completing specialty-specific modules (clinical rotations) in which simulation is typically used at the beginning of the module for teaching the skills and knowledge required for that module. Simulation is also used at the end of several modules to assess whether trainees have met the necessary competencies. In addition to simulation, the competency-based curriculum training program uses focused teaching curricula, intensive summative and formative feedback sessions, and multiple assessment tools in the clinical environment to assess trainee competence. Augmented learner and trainer satisfaction has been accompanied by direct evidence of improved and more effective learning outcomes in the competency-based curriculum residents [14, 15]. In previous work we have shown that enrolling trainees in the month-long, laboratory-based training course (Boot Camp) at the onset of the competency-based curriculum greatly enhances the knowledge and skills of new trainees up to the level of senior residents [12, 15]. These improvements have been shown to be maintained long after completion of the course [14]. Furthermore, this type of simulation-based teaching may privilege nontechnical skill development and later clinical learning [10]. As such, the “Boot Camp” is an essential component of residency training in Toronto. Despite the benefits of the intensive use of simulation, there is a need to consider its financial and human resource implications.

The present study compares the costs of simulation programs in the competency-based curriculum with the historic costs of running simulation programs in the regular stream of resident training both before and after the competency-based curriculum was implemented. In addition to the financial costs, the number of faculty hours dedicated to simulation training is evaluated.

Materials and Methods

To determine the financial costs of simulation in the competency-based curriculum [1, 6], both before and after the implementation of the pilot to the University of Toronto training program, all invoices that were submitted to the division’s administrative office for simulation training were reviewed for each academic year. For reference, the academic year of 2008 to 2009 was chosen to represent an academic year that preceded the introduction of the competency-based curriculum. During this year, two skills sessions using cadavers were given to the residents. The costs involved during these sessions included those related to the purchase and return of cadaver specimens, skills laboratory labor, and the use of associated materials (such as instruments, instrument resterilization, drapes, gowns, etc). The academic year of 2010 to 2011 was chosen to represent an academic year in which the competency-based training program was functioning parallel but separate to the regular stream of training. After 2010 to 2011 and starting with the academic year of 2011 to 2012, all simulation teaching and assessments that were used in the competency-based curriculum program were universally applied to all residents in the regular stream of the residency training program. This measure was implemented because our research demonstrated that, in our context, the augmented simulation experiences our competency-based curriculum residents were receiving were translating to better learning outcomes compared with their regular stream peers [12, 15]. The academic year of 2012 to 2013 was chosen to represent an academic year when simulation was used equally among the competency-based curriculum and regular stream residents.

Simulation training and assessment occurred either in a dedicated surgical skills laboratory (the Surgical Skills Centre, Mount Sinai Hospital, Toronto, Canada) or at various university-affiliated hospital teaching sites. All costs related to simulation were itemized into the categories of cadavers, artificial models, labor (the cost of support staff in the surgical skills laboratory), standardized patients, and other associated materials. No adjustments were made in response to inflation in our analyses.

To further understand the costs of simulation in the training program, a breakdown of the costs of simulation in each competency-based curriculum training module was performed. (The competency-based curriculum residency training program curriculum is comprised of 21 specific modules, of which 17 are clinical modules. To complete this training program, all residents must meet the level of competence at the end of each module.) Simulation is used in 11 of the modules. These include: Introduction to Basic Surgical Skills (also known as the Boot Camp, in which only basic surgical skills and knowledge are taught); Hip and Basic Fractures; Emergency Orthopaedic Surgery; Basic Sports; Arthroplasty; Foot and Ankle; Hand and Upper Extremity; Oncology; Complex Trauma; Complex Arthroplasty; and Advanced Sports. Each module uses simulation for teaching residents the necessary skills and knowledge related to the module at the beginning of the module, for assessing the residents’ level of competence at the end of the module, or both.

To generate the cost of simulation for training per resident for each module, the total cost of the training sessions was divided by the number of residents that attended the sessions (Table 1). The number of residents that attended the sessions in the academic year when the competency-based curriculum training program was functioning in parallel but separate to the regular stream of training was three. The maximum number of residents who attended the training sessions in the academic year when the competency-based curriculum training program was universally applied to all residents in the regular stream of the residency training program was 12. To generate the cost of simulation for evaluation per resident for each module, the total cost of the evaluation sessions was divided by the number of competency-based curriculum residents taking that module that year (Table 1). For the academic year of 2010 to 2011, the number was three, whereas in the academic year of 2012 to 2013, the number was five. The only exception to this was in the Boot Camp module and Sports modules, in which all 12 residents were assessed with simulation, and the Complex Trauma and Oncology modules, in which three residents were assessed with simulation. No regular stream residents used simulation as an evaluation tool during any other modules in that academic year because simulation was only used for competency-based curriculum residents.

Table 1 Determination of costs of simulation for training and assessment for each module per resident in the University of Toronto orthopaedic surgery residency training program before and after the implementation of the competency-based curriculum (CBC)
Full size table

In addition to competency-based curriculum-specific simulation, all Postgraduate Year 3 residents in the training program (competency-based curriculum and regular stream) have had their essential physician competencies, known as the CanMEDS roles [7] (analogous to the US’ Accreditation Council of Graduate Medical Education’s 6 competency framework), assessed in an annual “CanMEDS Objective Structured Clinical Examination (OSCE)” since 2011 to 2012. Here, residents are assessed in how they perform in standardized clinical scenarios with standardized patients. The costs of using standardized patients was determined by averaging the cost over the last 3 academic years.

The cost of using simulation in the competency-based curriculum does not just relate to direct financial outlay. Faculty are needed to teach and assess trainees during the simulation sessions.

To quantify faculty involvement, the number of hours spent teaching and assessing in each simulation session was totaled for the 3 same academic years used for the cost analysis: 1 year before the implementation of the competency-based curriculum (2008–2009), the year when the competency-based curriculum was functioning parallel but separate from the regular stream of training (2010–2011), and the year when simulation was used equally among the competency-based curriculum and regular stream residents for teaching and assessment (2012–2013). Faculty hours were not converted into a financial value, because we have no known accurate metric of measuring this in our training center.

In addition, other costs relating to the creation and maintenance of the competency-based curriculum were not entered into our analyses. We did not account for the costs of hiring a half-time administrative assistant, sending resident trainees to external review or specialty-specific weekend education courses that were related to the training module they were on, or paying for travel, lodging, and consultation costs for expert consultants in the initial design of the curriculum.

Results

The financial costs of using simulation for teaching and assessment substantially increased when the training program transitioned into having a competency-based curriculum run as a parallel stream of training compared with its traditional stream of training and when then training program transitioned into using simulation program-wide (Table 2). Although the total cost of using simulation to teach residents before the competency-based curriculum was implemented was CDN 10,090 (USD 11,140), the total cost of using simulation to teach and assess all residents in the competency-based curriculum and regular stream programs was approximately 15 times higher (CDN 155,750; USD 158,050).

Table 2 Costs of simulation training and assessment in the University of Toronto orthopaedic surgery residency training program before and after the implementation of the competency-based curriculum (CBC)*
Full size table

Analysis of the cost difference between the academic years when the competency-based curriculum program was functioning parallel but separate to the regular stream of training and when the competency-based curriculum training program was universally applied to all residents in the regular stream of the residency training program indicated that the increased cost was the result of the cost of the Boot Camp. When the Boot Camp was given to the three competency-based curriculum residents only, the total cost was CDN 55,060 (USD 60,790). When the Boot Camp was expanded to all 12 residents in the training program, the cost was CDN 106,720 (USD 108,290) (Table 3). The costs relating to using simulation in teaching stayed the same both before and after simulation were offered to the regular stream residents (because the total cost of cadavers, artificial models, labor, and associated materials remained the same).

Table 3 The costs of simulation in the competency-based curriculum modules as well as the CanMEDS OSCE*
Full size table

The cost breakdown for each training module and the CanMEDS OSCE once all simulation training sessions were shared with the regular stream residents indicated that the vast majority of the costs for simulation in the training program (68.5%) came from the month-long, simulation-based Boot Camp course, whose total cost was CDN 106,720 (USD 108,290) (Table 3). The remainder of the costs came from 10 clinical modules (CDN 47,620; USD 48,320 in total) and the CanMEDS OSCE (CDN 3000; USD 3040).

Once simulation was applied to all trainees in the program, the largest segment of cost was related to the use of cadavers (CDN 67,500/155,750; USD 68,500/158,050; 43%). The next largest segment of cost was related to the use of associated materials (such as gowns and gloves, suture materials, sawblades, instrument processing, etc) in the skills laboratory (CDN 42,660/155,750; USD 43,290/158,050; 27.4%). After this, costs related to the labor of surgical skills laboratory staff were 18.8% (CDN 29,330/155,750; USD 29,760/158,050); costs related to the use of artificial models (such as artificial bone and skin models) were 8.5% (CDN 13,250/155,750; USD 13,450/158,050). It is important to note that the costs of using the surgical skills laboratory, which otherwise could have been substantial, were waived as a result of a cooperative initiative between the laboratory and the Division of Orthopaedic Surgery.

The total number of hours spent teaching and assessing using simulation in the training program before the competency-based curriculum was implemented (2008–2009) was 96 hours (Table 4). The total number of hours spent teaching and assessing using simulation after the introduction of the CBC was 317 hours (Table 4). This represents an overall increase of approximately 3.3 times.

Table 4 The total number of faculty hours spent teaching and/or assessing resident trainees with simulation in the University of Toronto orthopaedic surgery residency training program before and after the implementation of the competency-based curriculum (CBC)
Full size table

Discussion

As surgical education programs begin to rely more heavily on simulation-based training, it is important to consider the financial implications. The present study compared the costs of simulation programs in the new competency-based curriculum at the University of Toronto with the historic costs of running simulation programs in the regular stream of residency training both before and after the competency-based curriculum was implemented. In addition to the financial costs, the number of faculty hours dedicated to sessions where simulation is used for training and assessment was evaluated.

Although the data presented are specific in terms of assessing cost and faculty hours, the main limitation of this study relates to the fact that the data reflect the costs associated with using simulation in one residency training center. However, despite the fact that these costs are specific to Toronto, any institution that seeks to implement similar simulation-based training programs will need to consider the same cost categories and potential financial relationships with the university, industry, and faculty partners in determining a budget. Although there may be variations in specific costs at individual training centers, we believe that the findings from this study are broadly generalizable.

The University of Toronto received government funding for this unique initiative in redesigning residency education. The financial arrangements that the University of Toronto has are likely specific to the Division itself and may not reflect all of the potential costs of using simulation in a competency-based curriculum training program in other centers. For example, one important cost not identified in the data presented is the true cost of renting the surgical skills laboratory for each hour of use. The Mount Sinai Surgical Skills Centre, which is the main skills laboratory for simulation in the training program, has waived the hourly rental rate (CDN 197.50, USD 200) because the Centre was developed as a cooperative initiative between the University’s Division of Orthopaedic Surgery and Department of Surgery. As a result of this relationship, the fixed cost of facility rental was waived. As such, all other costs for simulation were variable, the total value of which depended on the number of residents that were to use simulation at a specific time (including the numbers of cadavers, instrument sets needing preparation and resterilization, number of skills center staff assisting the trainees).

One important cost not identified is that of orthopaedic implants and instrument systems, which were donated through educational grants to the Division. If the costs of rent, plates, screws, nails, arthroplasty components, and other such elements were included, the budget would be substantially more than what was presented. Another indirect cost not identified is the financial cost of faculty time, which was delivered by geographic full-time faculty, fellows, and retired faculty. The number of hours faculty devote to teaching and assessment is significant. Furthermore, the amount of time listed does not include travel time to and from respective offices as well as the cost of transportation, parking, and opportunity cost of the faculty not being clinically active. It is important to note that this teaching and assessment is performed in addition to the teaching and assessment that is done at the hospital-specific level, where trainees are taught by their attending surgeons how to take care of patients in the clinic, operating room, inpatient ward, and emergency department. The last important cost not identified relates to the effect of having the residents away from their clinical duties. In all instances, coresidents, clinical fellows, and staff surgeons had to cover for the residents when they were involved in simulation training and assessment. Although we were unable to determine a dollar value for this, the effect on increasing the patient care workload on those physicians covering for the trainees was substantial.

Despite the clear advantages of using simulation derived from reports emanating from the medical education literature, the associated costs are a major limitation to implementation in mainstream practice [19]. As this study has identified, sufficient financial support is required for infrastructure, operating materials, support staff, and faculty time. Although research suggests that the costs are offset by enhanced learning and decreased clinical complications [2], there is a need for better return-on-investment studies [18, 19]. This is especially the case as it relates to orthopaedic surgery, where to our knowledge no publications exist on the topic. Such studies need to take into account not only the costs involved (including a categorical listing of all resources involved, the monetary value of each resource, and a breakdown of cost per learner for each resource) [17], but also the potential cost savings involved (which would include a cost breakdown of improvements in resource use, operating room efficiency, reduction in patient care errors and events, and improvements in patient safety) [18]. Performing this type of research is complex and will call for close collaboration among surgeons, educators, health economists, and statisticians. Without this evidence, surgical educators might not have the leverage necessary to convince funding bodies to invest in this type of training [18]. Funding bodies may vary from region to region, but may include hospitals, universities, accreditation bodies, medical insurance companies, private donors, and industry.

The current study identifies that there is a substantial cost of implementing simulation for teaching and assessment in a novel, competency-based training curriculum. The extensive use of high- and low-fidelity simulation-based modalities to train and assess residents over a course of 12 modules and one OSCE that assesses the CanMEDS roles has led to financial costs and hours of faculty time that did not exist in the University of Toronto training program before the competency-based curriculum was instituted. The cost of simulation increased by an increment of approximately 15.4 times. The amount of faculty hours dedicated increased by approximately three times. A more categorical analysis of cost has shown that the largest proportion is related to the use of cadavers followed by associated materials, labor, artificial models, and standardized patients. Despite these costs, we believe that they are worth the augmented learner and trainer satisfaction and enhanced learning outcomes [14, 15]. In fact, implementing a month-long, laboratory-based surgical skills training program (“Boot Camp”) at the onset of the competency-based curriculum program has been shown to greatly advance the knowledge and skills of new trainees [12, 15]. These improvements have also been shown to be maintained well into the training program and long after completion of the course [14]. In addition, all trainees who have used simulation in the training program have successfully passed their final licensing examinations.

It is also interesting to note that this type of training using simulation might not only benefit technical skill development and knowledge acquisition, but it may also privilege the development of ancillary abilities such as communication and collaboration skills [10]. By doing so, the use of simulation-based training, especially during the early stages of residency, may promote a more positive learning environment and greatly enhance the efficacy of later clinical learning [10].

Should the intensive use of simulation be used to train and assess residents in all surgical training programs? Although the financial costs and time demands on faculty in running the simulation program in the competency-based curriculum at the University of Toronto have been substantial, augmented learner and trainer satisfaction has been accompanied by direct evidence of improved and more efficient learning outcomes [12, 14, 15]. Although our division believes that the training is cost-effective, other training centers will have to decide individually as to whether the cost is worth the benefit. Future studies looking at the return on investment of simulation in teaching and evaluating trainees are needed, especially because simulation will be playing a larger role in medical education. This research, which should look not only at the financial and human resources costs involved, but also at the cost savings involved, will be complex. Without this information, surgical educators might not have the leverage necessary to convince funding bodies (such as hospitals, universities, accreditation bodies, medical insurance companies, private donors, and industry) to invest in this type of training [18].