There is increasing requirement to develop guidelines using transparent, standardized, and rigorous methods. Consequently, a better understanding of the knowledge, skills, and expertise necessary for guideline development is needed. The aim of this manuscript is to describe a theoretical framework of knowledge and skills that are required for individuals to serve on a guideline panel in varying capacities.
Based on an iterative process and review of published manuscripts focused on guideline development, we identified competencies, subcompetencies, and milestones.
Using a competency-based approach to training and the Dreyfus model of skill acquisition, we identified three core competencies: (1) facilitate the development of guideline structure and setup, (2) make judgments about the quality or certainty of the evidence, and (3) transform evidence to a recommendation. Level 1 focuses on recognizing and acknowledging the importance of a specific skill or behavior. Levels 2 and 3 require learners to demonstrate progressive acquisition of knowledge and application to specific behaviors. Level 4 represents the individual who has acquired the requisite knowledge and can function independently, while level 5 represents the mastery/aspirational level.
We propose a preliminary competency-based education framework that will (1) help standardize the qualifications needed for individuals to serve on guideline panels in varying capacities or (2) help with curricula development for teaching and training of guideline panel members. This framework can also help enable guideline-producing organizations to identify guideline methodologists with the relevant and appropriate level of knowledge and skills to lead guidelines. Validation of the framework and further refinement of the competencies and milestones will be required before widespread adoption.
The past 20 years has witnessed an exponential increase in the number of clinical practice guidelines produced by various national and international organizations. Simultaneously, there has been increasing pressure on guideline-producing organizations to adhere to more rigorous guideline development approaches. Resulting from the evidence-based medicine movement in the 1990s, there have been major advances in guideline development as underscored by the introduction and widespread adoption of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach which has become the most common framework used by guideline-producing organizations in the USA and internationally.1 These developments have been paralleled by increased public scrutiny and desire for greater transparency, as witnessed by the establishment of standards for trustworthy guidelines by the Academy of Medicine (formerly Institute of Medicine) and Guidelines International Network, as well as the indexing of guidelines that met such standards in the National Guideline Clearing House (NGC).2, 3
The requirement for greater methodological rigor has increased the pressure on organizations invested in guideline development to include individuals with specific expertise in health research methods, clinical trial design, epidemiology, and evidence synthesis. Recent reports have highlighted the shortcomings of guidelines developed by specialty professional organizations and provide further rationale for strengthening the leadership role of methodologists within guideline panels.4, 5 Research has demonstrated that guideline development groups that include a methodologist are associated with a decreased rate of recommendations labeled as “expert opinion” and more transparent management of potential conflicts of interest.6, 7
Norris et al. have previously outlined four main tasks related to guideline development and provide a set of minimum skills and experience to perform these tasks which include development of key questions, assessment of certainty of effect estimates, development of recommendations, and teaching GRADE.8 Furthermore, other resources are available such as the Journal of Clinical Epidemiology series, which explains the GRADE approach, and the GIN-McMaster Guideline Development Checklist, which outlines the various steps in guideline development.9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28 However, the specific skillset and expertise needed by different members of a guideline development team have not been well defined and extend beyond learning resources. Furthermore, greater detail around the process of skill acquisition and an understanding of what expected behaviors a guideline methodologist must demonstrate to fulfill this role have also not been clearly delineated. To meet this unfulfilled need, we propose a theoretical framework of competency-based learning for individuals involved in guideline development.
The primary objective of this manuscript is to outline and describe a theoretical framework of competencies and educational milestones that illustrate the requisite knowledge and skills that an individual should seek to acquire as part of a guideline development training program. We propose that such a framework will help (1) standardize the qualifications needed for individuals to serve on guideline panels, help lead guidelines, or serve as guideline methodologists; and (2) develop curricula for teaching and training of guideline panel members. This framework can also help enable organizations to identify guideline developers with the relevant and appropriate level of knowledge and skills.
Competency-Based Medical Education Framework
We used a competency-based medical education (CBME) framework, commonly used in the U.S. medical education and graduate medical education system, to define specific competencies and milestones. Competency-based medical education is defined as “an outcomes-based approach to the design, implementation, assessment, and evaluation of a medical education program using an organizing framework of competencies” with competencies being defined as the essential minimal set of a combination of attributes, such as applied knowledge, skills, and attitudes, that enable an individual to perform a set of tasks to an appropriate standard efficiently and effectively.29, 30 Educational milestones are defined under the umbrella of each competency and provide narrative descriptors of the competencies and subcompetencies along a developmental continuum. In graduate medical education (GME), milestones, defined as “developmentally based achievements that residents are expected to demonstrate at established intervals as they progress through training,” are reported to the Accreditation Council for Graduate Medical Education (ACGME) and used to determine whether a trainee is meeting the requisite requirements to be deemed competent.31,32,33,34
In accordance with the ACGME, we relied on the Dreyfus model to attribute competencies related to guideline methodology training. Originally developed by Hubert and Stuart Dreyfus who were commissioned by the U.S. Air Force to describe the development of the knowledge and skills of a pilot, this model consists of five stages: (1) novice, (2) advanced beginner, (3) competent, (4) proficient, and (5) expert.30, 35 They later identified a similar process of development in the chess player, the adult learning a second language, the adult learning to drive an automobile, and many others.35 When adapted to medicine, these levels outline progressive stages of educational growth and development through a dedicated educational curriculum with higher stages representing levels at which the learner has demonstrated mastery (Table 1).
Applying the CBME Framework and Dreyfus Model to Guideline Development
Six faculty from the U.S. GRADE Network (USGN; MHM, PD, RAM, RLM, SS, YFY) with extensive expertise as guideline methodologists helped develop the framework and competencies. The USGN is one of 16 established GRADE Working Group Networks and Centers around the world. It is the main organization in the USA focused on teaching guideline development using the GRADE approach through guideline development and systematic review workshops. Over the last 5 years, the USGN faculty has taught > 20 guideline development workshops attended by over 400 participants with a wide range of specialties (clinical, public health, environmental) and backgrounds (e.g., academic, health systems, policy) from a variety of professional societies and governmental entities.
An initial face-to-face meeting was held in 2017 and was attended by USGN members and two methodologist trainees (who have attended several training workshops). After consensus was reached to use a CBME model, two of the USGN members (SS and RAM) derived an initial set of competencies based on a review of GRADE guidance manuscripts developed by the GRADE Working Group (published in the Journal of Clinical Epidemiology), the GIN-McMaster Guideline Development Checklist, and the work by Norris et al..8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28 Subsequently, the group participated in focused discussions via teleconferences, and using an iterative process, the group developed a consensus-based set of proposed competencies, subcompetencies, and milestones.
Using the CBME model, we defined three core competencies: (1) facilitate the development of guideline structure and setup, (2) make judgments about the quality or certainty of the evidence, and (3) transform evidence to a recommendation. The competencies focused on standards necessary for ensuring the development of trustworthy guidelines, for example explaining the importance of multistakeholder input or developing clear and actionable recommendations. These competencies are further divided into 12 subcompetencies with proposed milestones (expectations of knowledge and skills), for each subcompetency (see Table 2).
Different Levels and Types of Competency May Be Necessary Depending on One’s Role in the Guideline Development Process
Acknowledging the different roles within a guideline panel, including that of the chair, methodologist, or panel members, some competencies may be more relevant and pertinent to fulfilling the expected role. Panel members may need to be most familiar with competencies related to transforming evidence to recommendations and limited competencies focused on making judgments about the certainty of evidence, whereas guideline methodologists may need to demonstrate most, if not all, of the core competencies. Guideline panel chair roles and expectations may vary depending on the organization’s guideline development process and may span across all three domains or may include a very small subset of competencies. As such, this framework has tried to capture as many of the potential behaviors and skills that different panel members may need, recognizing that different organizations may have variations in their guideline development processes with predefined roles and expectations that may be unique to their organization.
Not only will the set of competencies vary depending on the individuals’ role but the level of achievement may also be variable. To progress along the development continuum, individuals may need to attend workshops with dedicated training in guideline development, engage in self-directed learning activities, and/or demonstrate involvement in guideline efforts. At a given level, the expectation is that the learner substantially demonstrates that milestone as well as those in lower levels (if applicable). Level 1 represents the entry level characterized by expectations that focus on recognizing and acknowledging the importance of a specific skill or behavior relevant to the guideline development process. All panel members regardless of their role should reach level 1. We expect that the majority of learners who attend dedicated guideline development workshops or training programs are at level 1. The expectations for levels 2 and 3 require individuals to demonstrate progressive acquisition of knowledge and application of this knowledge. Individuals may demonstrate this behavior by their involvement in the development of “real-life” clinical practice or public health guidelines. Guideline panel members may only need to reach level 2 or perhaps level 3 for select competencies, whereas a guideline methodologist may be expected to demonstrate level 3 or 4 competency for most of the applicable competencies. For example, for the subcompetency of “formulate a healthcare question,” a guideline panel member who is the clinical or content expert may only need to demonstrate level 1 mastery (can explain how a standardized format can be used to structure healthcare questions in a guideline) or level 2 mastery (can explicitly describe the population, the interventions, and the comparators for each guideline question; can identify the outcomes (both benefits and harms); and rate the relative importance of the outcomes), whereas a guideline panel member serving as the methodologist may need to demonstrate mastery up to level 4 (can lead a guideline group to develop structured PICO questions with explicit considerations of population, intervention, comparator, and outcomes and in the selection and rating of outcomes) (Table 1). For guideline methodologists, progression to independent functioning (level 4) may only occur after working under, or alongside, a more experienced methodologist who can directly observe and account for the completed guideline product. Level 5 is the mastery or aspirational level and is applicable to a small subset of individuals who have advanced their own skills through their work on the development of a number of clinical practice guidelines across a spectrum of topics, are active in educational teaching activities or training workshops at a national or international level, or are dedicated to advancing the field through methodological and scholarly work in the practice of guideline development. Individuals at this level are expected to be master educators or mentors who can educate other learners who are at the beginning stages of training. In Figure 1, we outline the knowledge, skills, and expertise necessary for individuals to progress along a spectrum of performance levels.
This framework may be useful to various stakeholders including guideline panel members (by providing explicit expectations about knowledge and skills), organizations involved in guideline training (by facilitating training activities, curriculum development, and tools to competencies), and finally guideline-producing organizations (particularly those seeking guideline methodologists with the requisite relevant knowledge and skills to oversee guideline development) (Table 3).
We propose an initial theoretical framework for the development of a competency-based curriculum for learning and teaching guideline development. Competencies offer a common shared language for defining and specifying observable abilities, which can be expressed as actual behavior and can be assessed to ensure that learners have acquired them successfully.30 Implementation of competency-based education involves the identification of competencies, designing of curriculum and teaching programs that are mapped to specific competencies, and developing assessment tools that provide a valid and reliable evaluation of the outlined competencies.
As the majority of guideline methodology training is focused around attendance of guideline methodology workshops, development of competencies and operationalization of milestones can be used to structure learning, training, and assessment of guideline training activities. Although this framework describes the skills necessary for guideline development, every learner may not need to master every skill, behavior, or educational outcome. Depending on the role that an individual aims to take in the guideline development process, different levels of competency will be required. While we expect measurable progression of learners though the competency framework, we recognize that learners may advance at varying speeds or demonstrate competency levels that fall across a spectrum within each competency. Norris and colleagues have suggested three basic approaches to determine methodologist expertise level8: first, self-report of knowledge, skills, and experience using a standardized “guideline curriculum vitae (CV)”; second, demonstration of skills using worked examples; and third, a formal evaluation using a written or oral test. Additional assessment tools such as surveys or direct observation of skills within structured tutor trainee models of learning is likely to provide additional information.
Strengths and Limitations
To our knowledge, this is the first proposed framework for defining a set of competencies and milestones specific for guideline development akin to the five levels of milestones used in medical education.34 We view publication of this initial framework as a critical step toward establishing foundational competencies for various individuals involved in guideline development. Important limitations of this proposed framework are the lack of broad-based validation by a large and international group of established guideline methodologists and that the use of a competency-based framework may not be ideal for organizations that are not familiar with this educational framework. This framework was developed by members of the U.S. GRADE Network and relies on the GRADE approach for evaluating the certainty of evidence and strength of recommendations but also includes competencies that are considered standards for rigorous guideline development in general (irrespective of framework). For example, guideline setup, conflict of interest management, rating certainty in evidence, and developing clear and actionable recommendations are all critical steps in any guideline approach. We also acknowledge that the specific competencies for each of the panel members in their various roles have not been explicitly defined in this preliminary framework, but our goal was not to be overly prescriptive or impose this framework on organizations that may have already set processes and expectations for guideline panel members. However, we hope to conduct future studies to prioritize and build on this framework using the cumulative experience of national and international guideline developers, methodologists and learners and work toward synergy in establishing standards for competencies associated with training in guideline development. We envision additional studies to validate the competencies and gain consensus on the final working and operationalization of milestones. We will also seek to further refine the milestones and outline which set of competencies and what level of mastery should be expected from panel members in different roles. Future research may demonstrate that the initial set of competencies may be too specific or broad or that the described behaviors or knowledge outcomes may not be reflective of the specified competency. Despite this limitation, this framework can support capacity building by providing a set of skills and behaviors to structure a future curriculum or guide the educational experience of individuals involved in guideline development.
Future research efforts should focus on validation, evaluation, and further refinement of this framework. We encourage educators and other stakeholders to pilot test this framework to assess usability, ensure appropriate definitions of educational outcomes and behaviors, and improve the comprehensiveness of this framework. We also encourage the development of validated instruments to track the progress of learners and to assess if an individual has achieved educational milestones or mastered a competency. Additionally, future studies should be carried out to determine the minimal number of guidelines necessary to accomplish certain levels of progression using specific competencies, although the degree of guideline complexity (e.g., including diagnostic strategies and other additional guidance topics) will likely have an impact as well.
In conclusion, we propose a theoretical framework for a competency-based training program for individuals serving on guideline panels in various capacities including clinical or content experts, guideline methodologists, or chairs; this will help standardize and ensure adequate training by organizations that offer educational workshops, allow for a formal system to measure mastery of the competencies, and help guideline-producing organizations recognize the skill set needed to identify individuals with expertise to assist with guideline development in their respective fields.
Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924-6.
Murad MH. Clinical practice guidelines: a primer on development and dissemination. Mayo Clin Proc 2017;92(3):423-33.
Institute of Medicine. Clinical practice guidelines we can trust. Washinton, DC: National Academies Press; 2011; pp. 1-300.
Rosenfeld RM, Wyer PC. Stakeholder-driven quality improvement: a compelling force for clinical practice guidelines. Otolaryngol Head Neck Surg 2018;158(1):16-20.
Ioannidis JPA. Professional societies should abstain from authorship of guidelines and disease definition statements. Circ Cardiovasc Qual Outcomes 2018;11(10):e004889.
Ponce OJ, Alvarez-Villalobos N, Shah R, Mohammed K, Morgan RL, Sultan S, et al. What does expert opinion in guidelines mean? A meta-epidemiological study. Evid Based Med 2017;22(5):164-9.
Guyatt G, Akl EA, Hirsh J, Kearon C, Crowther M, Gutterman D, et al. The vexing problem of guidelines and conflict of interest: a potential solution. Ann Intern Med 2010;152(11):738-41.
Norris SL, Meerpohl JJ, Akl EA, Schunemann HJ, Gartlehner G, Chen Y, et al. The skills and experience of GRADE methodologists can be assessed with a simple tool. J Clin Epidemiol 2016;79:150-8 e1.
Zhang Y, Coello PA, Guyatt GH, Yepes-Nunez JJ, Akl EA, Hazlewood G, et al. GRADE guidelines: 20. Assessing the certainty of evidence in the importance of outcomes or values and preferences-inconsistency, imprecision, and other domains. J Clin Epidemiol 2018.
Zhang Y, Alonso-Coello P, Guyatt GH, Yepes-Nunez JJ, Akl EA, Hazlewood G, et al. GRADE guidelines: 19. Assessing the certainty of evidence in the importance of outcomes or values and preferences—risk of bias and indirectness. J Clin Epidemiol 2018.
Schunemann HJ, Cuello C, Akl EA, Mustafa RA, Meerpohl JJ, Thayer K, et al. GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence. J Clin Epidemiol 2018.
Schunemann HJ, Mustafa R, Brozek J, Santesso N, Alonso-Coello P, Guyatt G, et al. GRADE guidelines: 16. GRADE evidence to decision frameworks for tests in clinical practice and public health. J Clin Epidemiol 2016;76:89-98.
Andrews JC, Schunemann HJ, Oxman AD, Pottie K, Meerpohl JJ, Coello PA, et al. GRADE guidelines: 15. Going from evidence to recommendation—determinants of a recommendation’s direction and strength. J Clin Epidemiol 2013;66(7):726-35.
Andrews J, Guyatt G, Oxman AD, Alderson P, Dahm P, Falck-Ytter Y, et al. GRADE guidelines: 14. Going from evidence to recommendations: the significance and presentation of recommendations. J Clin Epidemiol 2013;66(7):719-25.
Guyatt GH, Thorlund K, Oxman AD, Walter SD, Patrick D, Furukawa TA, et al. GRADE guidelines: 13. Preparing summary of findings tables and evidence profiles—continuous outcomes. J Clin Epidemiol 2013;66(2):173-83.
Guyatt GH, Oxman AD, Santesso N, Helfand M, Vist G, Kunz R, et al. GRADE guidelines: 12. Preparing summary of findings tables—binary outcomes. J Clin Epidemiol 2013;66(2):158-72.
Guyatt G, Oxman AD, Sultan S, Brozek J, Glasziou P, Alonso-Coello P, et al. GRADE guidelines: 11. Making an overall rating of confidence in effect estimates for a single outcome and for all outcomes. J Clin Epidemiol 2013;66(2):151-7.
Brunetti M, Shemilt I, Pregno S, Vale L, Oxman AD, Lord J, et al. GRADE guidelines: 10. Considering resource use and rating the quality of economic evidence. J Clin Epidemiol 2013;66(2):140-50.
Guyatt GH, Oxman AD, Sultan S, Glasziou P, Akl EA, Alonso-Coello P, et al. GRADE guidelines: 9. Rating up the quality of evidence. J Clin Epidemiol 2011;64(12):1311-6.
Guyatt GH, Oxman AD, Kunz R, Woodcock J, Brozek J, Helfand M, et al. GRADE guidelines: 8. Rating the quality of evidence—indirectness. J Clin Epidemiol 2011;64(12):1303-10.
Guyatt GH, Oxman AD, Kunz R, Woodcock J, Brozek J, Helfand M, et al. GRADE guidelines: 7. Rating the quality of evidence—inconsistency. J Clin Epidemiol 2011;64(12):1294-302.
Guyatt GH, Oxman AD, Montori V, Vist G, Kunz R, Brozek J, et al. GRADE guidelines: 5. Rating the quality of evidence—publication bias. J Clin Epidemiol 2011;64(12):1277-82.
Guyatt GH, Oxman AD, Vist G, Kunz R, Brozek J, Alonso-Coello P, et al. GRADE guidelines: 4. Rating the quality of evidence—study limitations (risk of bias). J Clin Epidemiol 2011;64(4):407-15.
Balshem H, Helfand M, Schunemann HJ, Oxman AD, Kunz R, Brozek J, et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol 2011;64(4):401-6.
Guyatt GH, Oxman AD, Kunz R, Atkins D, Brozek J, Vist G, et al. GRADE guidelines: 2. Framing the question and deciding on important outcomes. J Clin Epidemiol 2011;64(4):395-400.
Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables. J Clin Epidemiol 2011;64(4):383-94.
Guyatt GH, Ebrahim S, Alonso-Coello P, Johnston BC, Mathioudakis AG, Briel M, et al. GRADE guidelines 17: assessing the risk of bias associated with missing participant outcome data in a body of evidence. J Clin Epidemiol 2017;87:14-22.
Guyatt GH, Oxman AD, Kunz R, Brozek J, Alonso-Coello P, Rind D, et al. GRADE guidelines 6. Rating the quality of evidence—imprecision. J Clin Epidemiol 2011;64(12):1283-93.
Frank JR, Snell LS, Cate OT, Holmboe ES, Carraccio C, Swing SR, et al. Competency-based medical education: theory to practice. Med Teach 2010;32(8):638-45.
Carraccio C, Englander R, Van Melle E, Ten Cate O, Lockyer J, Chan MK, et al. Advancing competency-based medical education: a charter for clinician-educators. Acad Med 2016;91(5):645-9.
Weinberger SE, Pereira AG, Iobst WF, Mechaber AJ, Bronze MS, Alliance for Academic Internal Medicine Education Redesign Task F, II. Competency-based education and training in internal medicine. Ann Intern Med 2010;153(11):751-6.
Long DM. Competency-based residency training: the next advance in graduate medical education. Acad Med 2000;75(12):1178-83.
Holmboe ES, Call S, Ficalora RD. Milestones and competency-based medical education in internal medicine. JAMA Intern Med 2016;176(11):1601-2.
Accreditation Council of Graduate Medical Education website. http://www.acgme.org/acgmeweb/tabid/430/ProgramandInstitutionalAccreditation/NextAccreditationSystem/Milestones.aspx. Accessed January 2, 2019
Dreyfus H, Dreyfus SE. Five Steps From Novice to Expert: Mind Over Machine. New York: Free Press; 1988.
The authors acknowledge the participation and contributions of Ms. Madelin Seidler and Dr. Raj Shah in the development of this manuscript.
Conflict of Interest
The authors do not have relevant financial conflicts of interest. All the authors are members of the GRADE Working Group and the U.S. GRADE Network. The authors provide educational activities and workshops on guideline development.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Sultan, S., Morgan, R.L., Murad, M.H. et al. A Theoretical Framework and Competency-Based Approach to Training in Guideline Development. J GEN INTERN MED 35, 561–567 (2020). https://doi.org/10.1007/s11606-019-05502-9
- clinical practice guideline
- competency-based training
- guideline panel