INTRODUCTION

In response to a projected shortage of primary care physicians trained to effectively care for patients with or at risk for HIV, the Health Resources Service Administration (HRSA) offered competitive funding to develop residency training programs to train HIV primary care providers.1 After successfully competing for this 4-year grant, the Yale Department of Internal Medicine embarked on curriculum planning for the HIV Primary Care Training Track. We sought to develop the curriculum based on the expected professional activities of an ideally trained primary care provider for HIV-infected patients. A separate manuscript describing the implementation and evaluation of the Yale HIV Primary Care training program structure has been published.2

The Accreditation Council for Graduate Medical Education (ACGME) currently requires residency training programs to evaluate trainees in six core competencies. In internal medicine (IM), the ACGME and American Board of Internal Medicine (ABIM) developed 22 reporting developmental milestones that contextualize these competencies.3 Because assessing some competencies require high inference by faculty, there have been recent efforts to integrate these competencies into discrete, directly observable activities that are critical for deeming a trainee entrustable for independent practice.4, 5 These Entrustable Professional Activities (EPA) are now being developed across a wide variety of specialties5,6,7 as well as for undergraduate medical education.8, 9

We report on the development of HIV-specific training EPAs and our experience with evaluation “bundles” developed to inform these EPAs, our early experience with tracking individual progress towards entrustment, and how the EPAs informed trainee and program evaluation. It is important to note that this HIV-specific training is embedded within the residents’ 3-year Primary Care Residency (parent) program requirement and is therefore a supplement to core residency training. The core program is also EPA based and shared competencies within the HIV training EPAs were included in the parent program biannual resident evaluations. The HIV Primary Care Training Track curriculum is accomplished during ambulatory immersion blocks, longitudinal clinic sessions, select HIV inpatient ward months, and a variety of other curricular activities.2

SETTING AND PARTICIPANTS

This educational innovation occurred within the Yale Primary Care Program with clinical and didactic experiences both in the inpatient setting at Yale New Haven Hospital and its embedded HIV ambulatory clinic (Nathan Smith Clinic). Eleven residents were recruited to the Yale Primary Care Program HIV Training Track over a 5-year period (2012–2017).

METHODS

We developed the HIV EPAs in an iterative process. The criteria for a clinical activity to be considered an EPA5 were disseminated to five Yale University School of Medicine AIDS Program faculty as well as to two Infectious Diseases faculty members. Twenty-five EPAs were identified after the first round of input. Four faculty members assigned to develop the HIV Training Track (DD, LB, MG, JT) winnowed the number by strictly applying the EPA criteria5, prioritizing the activities, and considering the number of EPAs manageable to track. This core group of faculty also determined and mapped the applicable curricular and reporting milestones for each EPA. The resulting document was programmed into our residency education management service (MedHub; Ann Arbor, MI) to generate a document that could automatically track current ACGME-required reporting milestones. Educational activities were selected to assist the resident in attaining entrustment in each EPA and were “bundled” with each EPA on MedHub for resident and advisor tracking and documentation.

RESULTS

  1. A.

    HIV EPAs—Twelve HIV Training Track EPAs were identified. The salient knowledge, skills, and attitudes required for each HIV EPA are summarized in Table 1 along with the associated activity and evaluation bundles.

  2. B.

    Activity bundles—For each EPA, a variety of activities were designed to help the trainee gain knowledge and skills/experiences necessary to progress towards entrustment within that activity. All content and resources were organized and available on the training track website (https://medicine.yale.edu/intmed/hivtraining/).

    1. (1)

      EPA-specific HIV modules—Twenty-four topics were identified for case creation by the cohort of HIV training track residents. Residents were each responsible for writing two to three modules per year (formatting based on Yale Office Based Medicine, a nationally distributed curriculum with which the residents were very familiar) until all 24 were developed. Additionally, residents accessed computer-based interactive cases (International Aids Society-Cases on the Web and National HIV Curriculum) highlighting any additional content of each EPA not covered in resident-developed cases. After completing each case, residents were asked to document completion and comprehension by taking the post-test the successful completion of which generates a CME certificate (scanned into their educational portfolio).

    2. (2)

      Deliberate practice with standardized patients (SP)—For two of the EPAs, scripts and SP roles were developed to exemplify (1) history-taking skills/difficult news-exploring risks for HIV acquisition emphasizing sex and drug-use history and giving positive results (EPA #1) and (2) initiating new medications (EPA #6 and others). These SP sessions were completed in the first 3 months of the first year with structured faculty observation and formative feedback/debriefing.

    3. (3)

      HIV preclinic conference—Weekly HIV-related topics that directly relate to EPAs were discussed in a small group setting.

    4. (4)

      Experiential—residents attended one half day per week in an HIV continuity clinic for 3 years with the aim of an eventual patient panel of 25–30 patients. In addition, residents had immersion blocks in which they spent 3 four-week blocks in the ambulatory setting. Residents were also assigned to the inpatient HIV ward 4–6 weeks per year. The details outlining activities and experiences afforded during the ambulatory immersion and inpatient HIV medicine blocks are outlined in our program description.2

  1. C.

    Evaluation

Table 1 12 HIV EPAs and Activity and Assessment Bundles
Full size table

Progress towards and determination of entrustment for each HIV EPA were accomplished formatively by a combination of direct bedside/clinic setting observation by faculty preceptors (utilizing frameworks focused on patient-centered interviewing skills, general history-taking skills, physical exam skills, breaking bad news, and initiating new medications, Table 1), patient management discussion and chart review, and patient and staff feedback (surveys). Following clinical encounters, residents logged the case on MedHub which generated the appropriate corresponding EPA form. Faculty completed the evaluation form which placed the resident into the developmental framework and provided specific behavioral feedback and suggestions (action plan). Documentation of structured observations was therefore saved to the trainee MedHub portfolio. Formative feedback was given immediately after observing residents.

While the HIV-TT trainees were included in the larger Yale Primary Care residency biannual review and ACGME reporting, we instituted a parallel HIV Training Track Clinical Competency Committee (CCC) to more comprehensively review trainee progress in the 12 HIV EPAs. Results from this biannual review were used for formative feedback to the individual resident and were used for program evaluation. This committee consisted of the academic advisor, other HIV clinic faculty, nursing, and the Program Director (non-voting). The final meeting of the CCC (end of year 3) for that resident served as the summative determination of entrustment. The HIV CCC pooled together HIV EPA logs and direct observation forms by clinic preceptors to plot progress in each EPA using narrative milestones (Table 2). Not surprisingly, the number of logged observations varied widely by particular EPA and by PGY level. Most residents who completed training or were in their final year (n = 7) reported exposure and content experience with all 12 HIV EPAs. To date, seven residents have completed the HIV TT Primary Care residency and data for the two most recent graduating cohorts (n = 5) on EPA performance is shown in Fig. 1. The level determined to be consistent with being “ready for independent practice” (i.e., entrustable) was set at 4 with a 5 defined as “aspirational” (consistent with terminology and behaviors outlined in ABIM Internal Medicine Milestones3). One hundred percent of the graduating residents were entrustable or aspirational in 91% (i.e., 11 out of 12) of the EPAs.

Table 2 Example HIV EPA Mapping and Behavioral Descriptors (See Appendix for Full Set)
Full size table
Figure 1
figure 1

At the completion of the 3-year HIV Primary Care Residency, final assessment of competency (scale 1–5 as defined in the text) was determined for each of the 12 EPAs. Data is shown for the first two evaluable cohorts of residents (n = 5). The average number of observations per resident per EPA was 7.7 (range 2–15). There were no observations done for two graduating residents on one out of the 12 (8%) EPAs. With a competency level of “4” (ready for unsupervised practice) = entrustable

Full size image

Information on gaps in skills and/or opportunities were identified through this process. The number of work-based assessment and mini-CEX events logged per resident per EPA was variable. Consistent with expected clinical work of routine HIV care, some EPAs were performed frequently (risk reduction, preventive care, ARV adherence/monitoring, mental health/substance use care) and some less (pregnant women with HIV, end-of-life/palliative care). Except for these latter EPAs (#9, 10) all five assessable residents who completed training had an average of 7.7 observations (2–15) per EPA.

DISCUSSION

We describe the first 5 years of our experience with trainee evaluation in a novel HIV Training Track residency program that is embedded in a northeastern US Primary Care Residency Program. The Entrustable Professional Activities construct gave us the opportunity to drive curricular planning to ensure that graduates of this HIV Training Track would possess the requisite skills to provide primary care for people living with HIV and/or those at risk. The strength of this Training Track is our a priori identification of these necessary professional activities (EPAs) to allow residents to gain these skills. Residents understood how learning activities (patient care, classes, modules, talks, SP sessions) contributed to the development of these EPAs and the basis by which they were assessed. Progress along the HIV EPAs provided formative feedback to residents and faculty to adjust and add experiences to assure adequate exposure. Faculty development needs were limited given the intuitive nature of EPAs, the availability of content, and the small core faculty precepting cohort had extensive prior experience with curriculum development, direct observation and feedback, and program evaluation.

A limitation to resident evaluation has been the cumbersome logging process that residents undertake to log patients (which generates the evaluation form sent to faculty who observed) by the EPA encountered after each visit. This limited our ability to consistently capture all of the EPAs that the residents actually performed. A more palatable smartphone platform for logging clinical experiences is in development as are the workplace-based assessments (WBA) for each. Strengthening this loop will afford more robust program evaluation and curricular adjustments going forward. Additionally, the small number of trainees each year does not allow for robust comparison of standardized test scores, passage of HIVMA certification test, and post-graduation competency.

With respect to EPAs, limited exposure to patients with HIV at the end of life may continue to affect our ability to train and assess competency and, because deaths due to HIV have thankfully plummeted, it may be removed as a specific EPA. Conversely, inconsistent exposure to HIV-infected women with reproductive issues has caused us to have our male residents comanage the pregnant women with HIV (who had requested female providers) when possible. Additionally, we note that prevention of HIV through the local development of a robust PrEP program had not been on the radar when developing the initial core EPA list and has subsequently become a significant clinical activity of great importance. We feel the knowledge and skills exemplified in EPA #3 (risk reduction counseling) and 2 (initiating ARV) encompass this new activity.

Despite not reaching 100% compliance with patient logs, the current system does allow faculty and participating residents to gain experience and track progress in most of the 12 HIV EPAs. Future plans include implementation of mobile patient logging and WBA, monitoring performance on summative knowledge assessments (HIVMA certificate exam), and post-graduate career choice and survey regarding how effective the program is in preparing graduates for a career in HIV primary care.

While patient outcomes were not explicitly evaluated, the residents increased clinical capacity and provided care comparable to the faculty providers in several performance measures and patient satisfaction.2, 10

The implementation of this HIV EPA-based curriculum has enabled us to explicitly prepare our trainees to provide the clinical care that HIV-infected patients deserve.

Our early results support the continued use of these EPAs, allow us to strengthen gaps in the curriculum, and may provide a framework for other residency programs wishing to offer enhanced training in HIV primary care.