Utility of self-competency ratings during residency training in family medicine education-emerging countries: findings from Japan
- 807 Downloads
Family medicine education-emerging countries face challenges in demonstrating a new program’s ability to train residents in womb-to-tomb care and resident ability to provide such care competently. We illustrate the experience of a new Japanese family medicine program with resident self-competency assessments.
In this longitudinal cross-sectional study, residents completed self-competency assessment surveys online during 2011–2015. Each year of training, residents self-ranked their competence using a 100-point visual analog scale for 142 conditions: acute (30 conditions), chronic (28 conditions) women’s health (eight conditions), and geriatrics/home (12 conditions) care; procedures (38 types); health promotion (21 conditions).
Twenty residents (11 women, 9 men) participated. Scores improved annually by training year from baseline to graduation; the mean composite score advanced from 31 to 65%. All subcategories showed improvement. Scores for care involving acute conditions rose from 49 to 75% (26% increase); emergency procedures, 46–65% (19% increase); chronic care, 33–73% (40% increase); women’s health, 16–59% (43% increase); procedural care, 26–56% (30% increase); geriatrics care-procedures, 8–65% (57% increase); health promotion, 21–63% (42% increase). Acute care, chronic care, and health promotion achieved the highest levels. Women’s health care, screenings, and geriatrics experienced the greatest increase. Health promotion gains occurred most dramatically in the final residency year.
A resident self-competency assessment provides a simple and practical way to conduct an assessment of skills, to monitor skills over time, to use the data to inform residency program improvement, and to demonstrate the breadth of family medicine training to policymakers, and other stakeholders.
KeywordsResidency training Evaluation Competency Developing countries Japan Family medicine General medicine Post-graduate education Primary care
family medicine education emerging
Japan Accreditation Council for Medical Education
Accreditation Council for Graduate Medical Education
family medicine training year
basic life support
advanced cardiac life support
pediatric advanced life support
advanced life support obstetrics
Japan Advanced Trauma Evaluation and Care
Shizuoka FM Program
Organized general/family medicine (FM) as a recognized medical specialty has a decades’ long history in multiple countries, dating back to the 1950s in the United Kingdom , Canada  and Australia  and to 1969 in the United States . Still, in many countries, FM is unrecognized or just emerging as a specialty [5, 6, 7, 8, 9]. In such places FM programs often lack resources for teaching and evaluating residents using the types of intensive one-on-one teaching procedures and complex competency or milestone evaluations that characterize FM education in more experienced countries.
FM programs may emerge because of policy changes that encourage or mandate their development. In this report, we refer to countries just beginning to develop systems or lacking official government-approved FM programs as Family Medicine Education Emerging (FMEE) countries. Emerging emphasizes that sociopolitical forces have reached a level where the determination across various constituencies to launch FM training is sufficient, relatively few faculty exist, and previous FM teaching experience is limited. FMEE academic programs face many barriers in launching a new discipline that impacts training at the student, resident, and faculty levels.
In Japan, a FMEE country, FM has not been recognized by the Japanese government as a medical specialty despite the efforts of advocates since the 1980s . However, some recent crises have created a sociopolitical climate favoring FM development. Japan is a superaging society, and the rapid growth in the elderly population has highlighted significant problems: fragmentation of care; polypharmacy; need for after-hours care, including home and palliative care; and sky-rocketing health care costs.
In addition to the need for FM specialists  and status reports [11, 12], other literature emphasizes the potential role of FM in the community , academic medical centers , and especially to support women’s health . Acknowledging Japan as a superaging society with fragmented medical care, the Japan Ministry of Health Labour and Welfare has articulated the clear need the need for general/FM physicians who can manage care efficiently, competently handle multiple medical problems, coordinate care, and address biopsychocial issues , and laid out plans for establishing general/FM as the 19th official specialty in Japan effective 2017 .
In light of new World Federation of Medical Education standards, the newly formed Japan Accreditation Council for Medical Education (JACME) will require medical schools to comply with new accreditation standards by 2023, including a newly required general medicine clinical rotation. Moreover, dedicated health tracks in medical schools focused on placing young physicians in health care shortage areas have seen limited success, despite lenient admission criteria, tuition waivers, and scholarships to individuals from health care shortage areas. These factors are forcing academicians to accept community and general medicine training as needed components of medical school training. The push from FM advocates and the crises confronting policymakers and academic leaders have provided the sociopolitical environment needed. In 2013, the Japanese government announced, as part of its restructuring of the specialist certification process, that a generalist specialty physician, called sogou shinryoui, will be recognized as the 19th Japanese medical specialty, starting in 2017 .
Despite national policy interest and academic medical institutions’ motivation to maintain international accreditation by offering clinical teaching in family/general medicine in Japan, some policy makers, academicians and subspecialists have been reluctant to change the status quo. Without previous experience working with or training family physicians, there may be apprehension as to whether FM residents can be trained across multiple disciplines and acquire the breadth of knowledge and skills required to treat the full spectrum of common conditions. Apprehension like this, which is not unique to Japan, challenges FMEE countries to provide evidence relatively quickly, despite limited resources, about the impact of FM training and to demonstrate programs’ ability to produce residents with the skills needed to provide competent womb-to-tomb care.
In medical residency training, competency is considered “an observable ability of a health professional, integrating multiple components such as knowledge, skills, values, and attitudes . As a part of their Outcome Project, in 1999 the U.S. Accreditation Council for Graduate Medical Education (ACGME) defined six core competencies expected of medical residents: patient care, medical knowledge, interpersonal and communication skills, professionalism, and practice-based medicine . Resident training programs must demonstrate that residents are developing these competencies by providing educational experiences and valid and reliable tools to use as evaluation instruments . Along with contextualizing the concept of competency, the ACGME, together with the American Board of Medical Specialties, published a suggested “toolbox” of assessment methods, including Chart Stimulated Recall; objective structured clinical examinations (OSCEs), and 360° Global Ratings . Recent movement surrounding resident assessment focuses on achievement of particular milestones . Previous studies describe the creation of an evaluation instrument in their resident-training program, and analyze the results as indicative of the effectiveness of the instrument and the training program [24, 25, 26].
In FM education-experienced environments, a gold-standard assessment would involve direct evaluation of a resident’s performance in patient care. A competency assessment would involve direct observation (e.g., Mini-CEX ) or review by a fully qualified family physician. Another option would be to examine board scores on a shelf examination each year (i.e., previously used test). Unfortunately, both procedures also require considerable infrastructure, including the availability of sufficient faculty time, shelf copies of a board examination (if one even exists) and a system to administer, score, and provide feedback to programs and residents. Unfortunately, both approaches are highly labor and cost intensive. A novel alternative would be a yearly resident self-competency evaluation. Using such an evaluation would provide a simple way of tracking individual development, and cumulatively, resident development in a program. This could thus be used to the extent newly developed FM residency training programs in FMEE countries are successfully teaching their residents the full breadth of care and providing reassurance that FM training program graduates with sufficient confidence in their knowledge and skills for FM practice.
In 2010 the University of Michigan Department of Family Medicine, along with three local hospitals, three mayors, and other supporters in Shizuoka Prefecture, Japan, received funding from the Prefectural Government to start a three-year FM training program unique in Japan for womb-to-tomb spectrum of care . This research illustrates a longitudinal approach to self-competency assessments based on 5 years’ experience with the system in a new FM program in the FMEE country of Japan.
The study utilized a longitudinal cross-sectional self-competency assessment survey conducted yearly during 2011–2015. The survey was distributed using an online survey system (Qualtrics) each spring, after the academic year ended, and repeated yearly over four years. The University of Michigan Institutional Review Board and Shizuoka FM Program (SFM), Shizuoka, Japan, approved this educational research.
The research was conducted with SFM program residents. Nine responses were collected in 2011, 12 in 2012, 13 in 2013, 12 in 2014, and 9 in 2015. Residents were asked to complete the self-assessment survey online in the spring, at the beginning of each residency year, and on completion of the program. At matriculation into the program, 12 residents started at postgraduate year (PGY) 3; 4 at PGY4; 1 at PGY5; 1 at PGY 6; and 2 at PGY9. This variation occurs as there is not a matching system beyond preliminary training programs . All SFM residents participated voluntarily. To avoid confusion with their PGY, residents were categorized by their FM training year (FMY), e.g., FMY1 means first year of FM training.
In conjunction with U.S. and Japanese faculty, a pilot paper survey with 92 conditions was tested in the inaugural SFM class, 2010. When we initiated the study there was no “list of common” conditions in primary care in Japan. Hence, we adapted a list for piloting based on 87 common conditions reflecting womb-to-tomb conditions commonly seen in acute care, chronic care, minor procedures, and preventive care/health promotion. We used as a reference common conditions recommended for medical students to learn about during their 4 week, third year clinical rotation at the University of Michigan, along with 5 emergency procedure protocols used commonly in Japan. Over the ensuing years, some edits for clarification, and additions and deletions, were made (further details available on request). Since 2014, the final instrument used for analysis contained 142 items: 30 items regarding acute care conditions, 28 on chronic care, 8 on women’s health, 38 on procedures (including three physical examination skills of the male/female genitalia/breast), 21 on health promotion (including newborn/young child, adolescent, and adult male/female), 12 on geriatrics and home care, and 5 emergency procedures protocols [basic life support (BLS), advanced cardiac life support (ACLS), pediatric advanced life support (PALS), advanced life support in obstetrics (ALSO), Japan advanced trauma evaluation and care (JATEC)]. Participants ranked their confidence for each domain or skill using a 100-point visual analog scale; 0 represents “can’t do at all” and 100 indicates “can do satisfactorily/completely.” Most respondents completed the survey in 10–15 min.
Completed surveys were downloaded from Qualtrics and imported into Microsoft Excel for organization before further analysis with Statistical Computer Software R. Means for individual residents and classes were calculated for each item and group to determine an average score per year and for the class as a whole. Mean scores for each category were compared by year. To compare progress over the residency, mean scores were plotted by FM residency program year (FMY) using box-plots.
Twenty residents completed one or more of the 59 surveys received over the 5 years of instrument administration. Eleven respondents were women; 9 were men. By Japanese regulation, all residents must complete a two-year preliminary rotating residency before entering an advanced residency such as the SFM program. As not all residents took the survey all years, the number of surveys taken differed by FMY; 16 trainees completed the survey at baseline (matriculation), 17 trainees responded after completing FMY1, 11 after FMY2, and 11 after FMY3. Four trainees completed the survey only once, another five completed two surveys, three completed three surveys, and eight completed four surveys. For privacy reasons, data on age and previous residency training were not collected.
Acute conditions and emergency procedures
Chronic women’s health conditions
Geriatrics, home care
Preventive Services: Health Promotion, Screenings
These findings illustrate that if a FM residency training program provides sufficient training experiences, self-assessed competencies for a broad range of conditions and procedures can progressively improve over the course of training. The progressive improvements on the self-competency assessment findings suggesting successful skills in FM are corroborated by a passing performance by all residents who completed the program on the actual official national board examination conducted by the Japan Primary Care Association. This provides at least a minimal level of external validity of progressively improving self-competency assessments. For policy makers desiring FM womb-to-tomb training in three years, these data suggest FM residents can learn progressively on a year-to-year basis the content and skills necessary to feel confident in their ability to manage the acute care, chronic care, preventive care, and procedural care needed by patients in primary care.
Implications for FM in Japan
Despite policy maker and educator lack of familiarity about FM residency programs’ ability to teach content outside of general internal medicine in Japan, these data provide compelling evidence that, in an appropriately structured program, residents can learn material and develop sufficient confidence in content areas, including women’s health, children’s health, and minor surgical specialties necessary for womb-to-tomb care. Of particular interest, the benefits of training in full-spectrum womb-to-tomb care were similar regardless of whether the learner had years of experience and practice in another non-generalist specialty or had just completed the basic clinical training through the 2-year mandatory rotating internship. Even residents at PGY5-9 at entry in the program initially scored low in areas of focus by family physicians, for example, preventive services, outpatient procedures, and women’s health.
These data are particularly timely as a requirement for women’s health care in sougou shinryoui training is under intense debate in Japan. There, many argue that women’s health care is the domain of obstetrician-gynecology providers exclusively. A plateau effect in progression occurred in the third year of residency. This probably occurred due to: limited opportunities to provide women’s health care in the outpatient FM clinics as FMY3s , a limited number of attending physicians trained in women’s health care, and patients’ being unaware that women’s health care services are provided in the FM clinics. Nonetheless, the evidence is clear that, given the opportunity, FM residents can develop the skills to provide competently women’s health care in Japan.
Implications for FMEE countries
These self-competency assessments can be used potentially for residents to see their own progress and for programs to see the progress of residents, individually and across the board. Persistently low assessments by many residents might suggest a problem with rotations, internal or external. While other procedures regarded as gold standards for assessing resident progress and competency—for example, individualized assessments during patient care and monitoring of progress on shelf copies of mock examinations—are plausible, these approaches may be difficult to utilize in FMEE countries, where resources are limited. Regarding the performance of the instrument, conducting a concurrent validity assessment, by using faculty evaluations, was infeasible; however, the perception of faculty that residents were progressively improving in their ability to provide the breadth of care as expected with advancement in the program does suggest external validity of the instrument. The findings also are corroborated by a passing performance by all residents who completed the program on the national board examination conducted by the Japan Primary Care Association.
Although these surveys were conducted online, paper-based assessments would be possible. Additionally, considerable adjustments were necessary to make the categories relevant to Japan. Adding some procedures—for example, regarding the management of home-care conditions—was easy, while dropping others, e.g., vasectomy, was fairly obvious. Decisions regarding some conditions and procedures were difficult, however, because faculty were of different training backgrounds and faculty in the program come from diverse backgrounds; some instructors received training abroad, whereas others are self-styled family physicians who received traditional internal medicine training that excluded surgical care involving ophthalmology, otolaryngology, obstetrics/gynecology, urology, or general surgery. The biggest the self-competency assessment procedures challenge of using in another country would be developing the specific items in each category.
The mean scores reported represent a composite. Attrition or nonparticipation of certain residents over time precludes our inferring precise levels of competency, though making such pinpoint measurement was not the purpose of this study. Second, as rankings are self-assessments, no ideal score exists, particularly because of regional variations in services needed and provided. Third, there is no consensus in Japan and many FMEE countries about what FM residents should become competent to do. As the assessment using 142 items was informed by considerable U.S. experience, and adapted iteratively with input from Japanese FM colleagues, we believe it adequately represents most common problems and procedures in primary care in Japan. Home-care items were added to the survey to reflect national and prefectural priorities, though in future iterations programs may want to add items on complementary and alternative/integrative medicine, treatments commonly used in Japan, or other particular conditions or procedures emphasized. Fourth, breaking up the data was sometimes challenging, as some items related to women’s health were grouped with procedures and preventive health, while others are chronic conditions. Fifth, formal validity testing has not been conducted for this instrument, though face validity is self-evident.
These data suggest that the SFM program provides a training experience that produces physicians confident to practice the breadth of care needed by family physicians. The findings further illustrate that appropriately structured FM training programs can produce graduates prepared to provide the full spectrum of FM care. A resident self-competency assessment provides a simple and practical way to conduct resident self-assessments of skills, to monitor this over time, to use the data to inform residency program improvement, and to demonstrate the breadth of family medicine training to policymakers, and other stakeholders.
MDF Contributed to the conception and design, acquisition of data, analysis and interpretation, drafting of the manuscript, and critically revising it for intellectual content. SM Contributed to the conception and design, acquisition of data, analysis and interpretation, drafting of the manuscript, and critically revising it for intellectual content. LI Contributed to the analysis and interpretation, drafting of the manuscript, and critically revising it for intellectual content. KN Contributed to the conception and design, analysis and interpretation, drafting of the manuscript, and critically revising it for intellectual content. KS Contributed to the conception and design, acquisition of data, and critically revising it for intellectual content. MT Contributed to the conception and design, acquisition of data, drafting of the manuscript, and critically revising it for intellectual content. TT Contributed to the conception and design, acquisition of data, and critically revising the manuscript for intellectual content. MI Contributed to the analysis and interpretation, drafting of the manuscript, and critically revising it for intellectual content. All authors contributed to drafting the manuscript. All authors agree to be accountable for all aspects of the work related to accuracy and integrity. All authors read and approved the final manuscript.
The authors gratefully acknowledge the participation of the residents who took the survey. The administrative support of Ms. Morita and Ms. Miyashita made the data collection possible.
The authors declare that they have no competing interests.
Ethics approval and consent to participate
This research was reviewed by the University of Michigan Institutional Review Board and ruled as exempt (HUM00047926).
The authors further acknowledge the support of the Shizuoka Prefectural Government for supporting the SFM program and the Shizuoka–University of Michigan Advanced Residency Training, Education, and Research (SMARTER FM) Project.
- 1.Stevens R. Changes in general practice: the question of status. In: Stevens R, editor. Medical practice in modern England: the impact of specialization and state medicine. New Brunswick: Transaction Publishers; 2003. p. 153–68.Google Scholar
- 2.College of Family Physicians of Canada. College History. http://www.cfpc.ca/CollegeHistory/. Accessed 14 Nov 2016.
- 3.Royal Australian College of General practitioners. College History: Australian General Practice. http://www.racgp.org.au/yourracgp/organisation/history/college-history/australian-general-practice/. Accessed 14 Nov 2016.
- 4.American Board of Family Medicine. History of the Specialty. https://www.theabfm.org/Search.aspx?q=history. Accessed 14 Nov 2016.
- 5.Ng JC, Teng LC, Abdullah A, Wong HC, Hanafi SN, Phoa SS. The status of family medicine training programs in the Asia Pacific. Fam Med. 2016;48:194–202.Google Scholar
- 9.Gossa W, Wondimagegn D, Mekonnen D, Eshetu W, Abebe Z, Fetters MD. Key informants’ perspectives on development of Family Medicine training programs in Ethiopia. Adv Med Educ Pract. 2016;6:1–9.Google Scholar
- 13.Fetters MD, Kiyota A, Sano K. 地域における家庭医療の社会的役割:家庭医を専門医として理解するために (The social role family practice plays in the community: understanding the family physician as a specialist). Jpn J Prim Care. 2004;27(1):29–35.Google Scholar
- 14.Fujioka Y, Fetters MD. アメリカ医科大学における家庭医療 (Family Medicine in American medical schools). Hospitals (病院) (Byouin). 2008;67(10):893–6.Google Scholar
- 15.Fetters MD, Fujioka Y. なぜ、日本の家庭医学研修に婦人医療研修が含まれるべきなのか?(Why women’s health training is needed during family medicine residency training in Japan). Jpn J Fam Pract. 2009;15(1):44–51.Google Scholar
- 16.Ministry of Health, Labour and Welfare. 総合診療専門医に関する委員会のまとめ (Committee Summary on a General Medicine Specialty–authors’ translation). http://www.mhlw.go.jp/file/06-Seisakujouhou-10800000-Iseikyoku/0000081678.pdf; 2014. Accessed 14 Nov 2016.
- 17.Ministry of Health, Labour and Welfare. 「総合医」「総合診療医」等に関する論点整理(案) (Discussion points about “Generalists, general practitioners” authors’ translation). http://www.mhlw.go.jp/stf/shingi/2r9852000002rudp-att/2r9852000002ruvt.pdf; 2012. Accessed 14 Nov 2016.
- 18.Ministry of Health Labour and Welfare. 専門医の在り方に関する検討会 報告書 (Report of the Committee on the Status of Specialists. http://www.mhlw.go.jp/stf/shingi/2r985200000300ju.html; 2013. Accessed 14 Nov 2016.
- 19.American Association of Medical Colleges. Core Entrustable Professional Activities For Entering Residency: Curriculum Developers Guide. https://members.aamc.org/eweb/upload/core%20EPA%20Curriculum%20Dev%20Guide.pdf. Accessed 14 Nov 2016.
- 22.Accreditation Council for Graduate Medical Education and American Board of Medical Specialties. ACGME competencies: suggested best methods for evaluation. https://www.partners.org/Assets/Documents/Graduate-Medical-Education/ToolTable.pdf. Accessed 14 Nov 2016.
- 23.Accreditation Council for Graduate Medical Education and The American Board of Family Medicine. The Family Medicine Milestone Project. http://www.acgme.org/portals/0/PDFs/Milestones/FamilyMedicineMilestones.pdf. Accessed 14 Nov 2016.
- 25.Golnik C, Beaver H, Gauba V, Lee AG, Mayorga E, Palis G, Saleh GM. Development of new valid, reliable, and internationally applicable assessment tool of residents’ competence in ophthalmic surger (an American Ophthalmological Society thesis). Trans Am Ophthalmol Soc. 2013;111:24–33.PubMedPubMedCentralGoogle Scholar
- 27.The ABIM Mini-Clinical Evaluation Exercise (Mini-CEX). http://www.abim.org/program-directors-administrators/assessment-tools/mini-cex.aspx. Accessed 14 Nov 2016.
- 28.Terada M. Shizuoka family medicine training program. J Jpn Munic Hosp Assoc. 2011;50(6):867–73.Google Scholar
- 29.Fetters MD, Yokoyama I. Medical education in Japan. In: The SAGE handbook of modern Japanese studies. Thousand Oaks; 2014, p. 299–319.Google Scholar
- 30.Shultz CG, Chu MS, Yajima A, Skye EP, Kiyoshi S, Inoue M, Tsuda T, Fetters MD. The cultural context of teaching and learning sexual health care examinations in Japan: a mixed methods case study assessing the use of standardized patient instructors amony Japanese family physician trainees of the Shizuoka Family Medicine Program. Asia Pac Fam Med. 2015;14:8.CrossRefPubMedPubMedCentralGoogle Scholar
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.