Skip to main content
Log in

Rigorous Development does not Ensure that Guidelines are Acceptable to a Panel of Knowledgeable Providers

  • Original Article
  • Published:
Journal of General Internal Medicine Aims and scope Submit manuscript



Rigorous guideline development methods are designed to produce recommendations that are relevant to common clinical situations and consistent with evidence and expert understanding, thereby promoting guidelines’ acceptability to providers. No studies have examined whether this technical quality consistently leads to acceptability.


To examine the clinical acceptability of guidelines having excellent technical quality.


We selected guidelines covering several musculoskeletal disorders and meeting 5 basic technical quality criteria, then used the widely accepted AGREE Instrument to evaluate technical quality. Adapting an established modified Delphi method, we assembled a multidisciplinary panel of providers recommended by their specialty societies as leaders in the field. Panelists rated acceptability, including “perceived comprehensiveness” (perceived relevance to common clinical situations) and “perceived validity” (consistency with their understanding of existing evidence and opinions), for ten common condition/therapy pairs pertaining to Surgery, physical therapy, and chiropractic manipulation for lumbar spine, shoulder, and carpal tunnel disorders.


Five guidelines met selection criteria. Their AGREE scores were generally high indicating excellent technical quality. However, panelists found 4 guidelines to be only moderately comprehensive and valid, and a fifth guideline to be invalid overall. Of the topics covered by each guideline, panelists rated 50% to 69% as “comprehensive” and 6% to 50% as “valid”.


Despite very rigorous development methods compared with guidelines assessed in prior studies, experts felt that these guidelines omitted common clinical situations and contained much content of uncertain validity. Guideline acceptability should be independently and formally evaluated before dissemination.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others


  1. Mosca L, Linfante AH, Benjamin EJ, et al. National study of physician awareness and adherence to cardiovascular disease prevention guidelines. Circulation. 2005;111(4):499–510.

    Article  PubMed  Google Scholar 

  2. Bauer MS. A review of quantitative studies of adherence to mental health clinical practice guidelines. Harv Rev Psychiatry. 2002;10(3):138–53.

    Article  PubMed  Google Scholar 

  3. Cruz-Correa M, Gross CP, Canto MI, et al. The impact of practice guidelines in the management of Barrett esophagus: a national prospective cohort study of physicians. Arch Intern Med. 2001;161(21):2588–95.

    Article  PubMed  CAS  Google Scholar 

  4. Switzer GE, Halm EA, Chang CC, Mittman BS, Walsh MB, Fine MJ. Physician awareness and self-reported use of local and national guidelines for community-acquired pneumonia. J Gen Intern Med. 2003;18(10):816–23.

    Article  PubMed  Google Scholar 

  5. Christakis DA, Rivara FP. Pediatricians’ awareness of and attitudes about four clinical practice guidelines. Pediatrics. 1998;101(5):825–30.

    Article  PubMed  CAS  Google Scholar 

  6. Cabana MD, Rand CS, Powe NR, Wu AW, Wilson MH, Abboud PA. Why don't physicians follow clinical practice guidelines? A framework for improvement. JAMA. 1999;282(15):1458–65.

    Article  PubMed  CAS  Google Scholar 

  7. Ward MM, Vaughn TE, Uden-Holman T, Doebbeling BN, Clarke WR, Woolson RF. Physician knowledge, attitudes and practices regarding a widely implemented guideline. J Eval Clin Pract. 2002;8(2):155–62.

    Article  PubMed  Google Scholar 

  8. Cabana MD, Ebel BE, Cooper-Patrick L, Powe NR, Rubin HR, Rand CS. Barriers pediatricians face when using asthma practice guidelines. Arch Pediatr Adolesc Med. 2000;154(7):685–93.

    PubMed  CAS  Google Scholar 

  9. Committee to Advise the Public Health Service on Clinical Practice Guidelines, Institute of Medicine. In: Field MJ, Lohr KN, eds. Clinical Practice Guidelines: Directions for a New Program. Washington, DC: National Academies Press; 1990.

    Google Scholar 

  10. Shaneyfelt TM, Mayo-Smith MF, Rothwangl J. Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature. JAMA. 1999;281(20):1900–5.

    Article  PubMed  CAS  Google Scholar 

  11. The AGREE Collaboration. The Appraisal of Guidelines for Research & Evaluation (AGREE) Instrument. Available at: Accessed March 21, 2007.

  12. The AGREE Collaboration. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care. 2003;12(1):18–23.

    Article  Google Scholar 

  13. Burgers JS. Guideline quality and guideline content: are they related? Clin Chem. 2006;52(1):3–4.

    Article  PubMed  CAS  Google Scholar 

  14. Shiffman RN, Shekelle P, Overhage JM, Slutsky J, Grimshaw J, Deshpande AM. Standardized reporting of clinical practice guidelines: a proposal from the Conference on Guideline Standardization. Ann Intern Med. 2003;139(6):493–8.

    PubMed  Google Scholar 

  15. United States National Guideline Clearinghouse Inclusion Criteria. Available at: Accessed July 6, 2006.

  16. State of California. Assembly Bill 749 (Calderon), Workers’ Compensation: Administration and Benefits; 2002.

  17. State of California. Senate Bill 228 (Alarcón), Workers’ Compensation; 2003.

  18. State of California. Senate Bill 899 (Poochigan), Workers’ Compensation; 2004.

  19. Hutchings A, Raine R, Sanderson C, Black N. An experimental study of determinants of the extent of disagreement within clinical guideline development groups. Qual Saf Health Care. 2005;14(4):240–5.

    Article  PubMed  CAS  Google Scholar 

  20. Nuckols TK, Wynn BO, Lim Y, et al. Evaluating Medical Treatment Guideline Sets for Injured Workers in California. Santa Monica, CA: RAND Corporation; 2005. MG-400-ICJ.

  21. Grol R, Cluzeau FA, Burgers JS. Clinical practice guidelines: towards better quality guidelines and increased international collaboration. Br J Cancer. 2003;89(Suppl):S4–8.

    Article  PubMed  Google Scholar 

  22. Watine J, Friedberg B, Nagy E, et al. Conflict between guideline methodologic quality and recommendation validity: a potential problem for practitioners. Clin Chem. 2006;52(1):65–72.

    Article  PubMed  CAS  Google Scholar 

  23. Vlayen J, Aertgeerts B, Hannes K, Sermeus W, Ramaekers D. A systematic review of appraisal tools for clinical practice guidelines: multiple similarities and one common deficit. Int J Qual Health Care. 2005;17(3):235–42.

    Article  PubMed  Google Scholar 

  24. The AGREE Collaboration. The Appraisal of Guidelines for Research & Evaluation (AGREE) Instrument Training Manual. Available at: Accessed March 22, 2007.

  25. Fitch K, Bernstein SJ, Aguilar MD, et al. The RAND/UCLA Appropriateness Method User’s Manual. Santa Monica, CA: RAND Corporation; 2001. MR-1269-DG-XII/RE. Available at:

  26. Shekelle PG. The appropriateness method. Med Decis Making. 2004;24(2):228–31.

    Article  PubMed  Google Scholar 

  27. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–74.

    Article  PubMed  CAS  Google Scholar 

  28. Shekelle PG, Kahan JP, Bernstein SJ, Leape LL, Kamberg CJ, Park RE. The reproducibility of a method to identify the overuse and underuse of medical procedures. N Engl J Med. 1998;338(26):1888–95.

    Article  PubMed  CAS  Google Scholar 

  29. Shekelle PG, Chassin MR, Park RE. Assessing the predictive validity of the RAND/UCLA appropriateness method criteria for performing carotid endarterectomy. Int J Technol Assess Health Care. 1998;14(4):707–27.

    Article  PubMed  CAS  Google Scholar 

  30. Higashi T, Shekelle PG, Adams JL, Kamberg CJ, Roth CP, Solomon DH, Reuben DB, Chiang L, MacLean CH, Chang JT, Young RT, Saliba DM, Wenger NS. Quality of care is associated with survival in vulnerable older patients. Ann Intern Med. 2005 Aug 16;143(4):274–81.

    PubMed  Google Scholar 

  31. Wynn BO, Bergamo G, Shaw RN, Mattke S, Dembe A. Medical Care Provided California’s Injured Workers: An Overview of the Issues. Santa Monica, CA: RAND Corporation; 2006. WR-394-ICJ.

  32. Hasenfeld R, Shekelle PG. Is the methodological quality of guidelines declining in the U.S.? Comparison of the quality of U.S. agency for health care policy and research (AHCPR) guidelines with those published subsequently. Qual Saf Health Care. 2003;12(6):428–34.

    Article  PubMed  CAS  Google Scholar 

  33. Brook RH, McGlynn EA, Cleary PD. Quality of health care. Part 2: measuring quality of care. N Engl J Med. 1996;335(13):966–70.

    Article  PubMed  CAS  Google Scholar 

  34. Donabedian A. Explorations in Quality Assessment and Monitoring, Vol. 1. The Definition of Quality and Approaches to its Assessment. Ann Arbor, MI: Health Administration Press; 1980.

  35. Donabedian A. Explorations in Quality Assessment and Monitoring, Vol. 2. The Criteria and Standards of Quality. Ann Arbor, MI: Health Administration Press; 1982.

  36. Donabedian A. Explorations in Quality Assessment and Monitoring, Vol. 3. The Methods and Findings of Quality Assessment and Monitoring: An Illustrated Analysis. Ann Arbor, MI: Health Administration Press; 1985.

  37. Shekelle PG, Woolf SH, Eccles M, Grimshaw J. Clinical guidelines: developing guidelines. BMJ. 1999;318(7183):593–6.

    PubMed  CAS  Google Scholar 

  38. Maue SK, Segal R, Kimberlin CL, Lipowski EE. Predicting physician guideline compliance: an assessment of motivators and perceived barriers. Am J Manag Care. 2004;10(6):383–91.

    PubMed  Google Scholar 

  39. Mottur-Pilson C, Snow V, Bartlett K. Physician explanations for failing to comply with “best practices”. Eff Clin Pract. 2001;4(5):207–13.

    PubMed  CAS  Google Scholar 

  40. Grol R, Dalhuijsen J, Thomas S, Veld C, Rutten G, Mokkink H. Attributes of clinical guidelines that influence use of guidelines in general practice: observational study. BMJ. 1998;317(7162):858–61.

    PubMed  CAS  Google Scholar 

  41. Burgers JS, Grol RP, Zaat JO, Spies TH, van der Bij AK, Mokkink HG. Characteristics of effective clinical guidelines for general practice. Br J Gen Pract. 2003;53(486):15–9.

    PubMed  Google Scholar 

  42. Shiffman RN, Dixon J, Brandt C, et al. The GuideLine Implementability Appraisal (GLIA): development of an instrument to identify obstacles to guideline implementation. BMC Med Inform Decis Mak. 2005;5:23–34.

    Article  PubMed  Google Scholar 

  43. Shekelle PG, Ortiz E, Rhodes S, et al. Validity of the Agency for Healthcare Research and Quality Clinical Practice Guidelines: how quickly do guidelines become outdated? JAMA. 2001;286(12):1461–7.

    Article  PubMed  CAS  Google Scholar 

  44. American Academy of Orthopaedic Surgeons. Guidelines and support documents for hip pain, knee injury, knee osteoarthritis, low back pain/sciatica, shoulder pain and wrist pain. 2001–2003.

  45. American College of Occupational and Environmental Medicine. Occupational Medicine Practice Guidelines: Evaluation and Management of Common Health Problems and Functional Recovery in Workers, Second Edition. Beverly Farms, Massachusetts; 2004.

  46. Intracorp. Intracorp Optimal Treatment Guidelines. Philadelphia, Pennsylvania; 2003. Available at:

  47. McKesson. InterQual Care Management Criteria for Workers’ Comp & Clinical Evidence Summaries. (Formerly QualityFirst® Workers’ Compensation/Disability Guidelines). Newton, Massachusetts. 2004.

  48. Work Loss Data Institute. Official Disability Guides (ODG), Treatment in Workers’ Compensation. San Diego, CA; 2004. Available at:

Download references


Christine Baker, M.A., Executive Officer of the California Commission on Health and Safety and Workers’ Compensation and Anne Searcy, M.D., Medical Director of the California Division of Workers’ Compensation participated in the study conception and design (specifically, choosing the inclusion criteria for the guidelines). Paul G. Shekelle, M.D, Ph.D., The RAND Corporation, Santa Monica; Division of General Internal Medicine and Health Services Research, Department of Medicine David Geffen School of Medicine at the University of California, Los Angeles; and VA Greater Los Angeles Healthcare System, Los Angeles, California, provided comments on a draft of this manuscript. This study was sponsored by the California Department of Industrial Relations (the Division of Workers Compensation and the California Commission on Health and Safety and Workers’ Compensation). The RAND Corporation provided support for the preparation of this manuscript. Although Christine Baker and Anne Searcy, Division of Workers’ Compensation participated in the study conception and design, data acquisition, analysis, and preparation of this manuscript were completely independent of the funders.

Conflicts of Interest

None disclosed.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Teryl K. Nuckols MD, MSHS.

Additional information

This document includes content previously published in a RAND report, available online at:



Table 6 and 7

Table 6 Panel Judgments Regarding Perceived Comprehensiveness for 16 Selected Topics, Any Residual Content Aside from Those 16 Topics, and Overall (C = Comprehensive, I = Intermediate, NC = Not Comprehensive)
Table 7 Panel Judgments Regarding Perceived Validity for 16 Selected Topics, Any Residual Content Aside from Those 16 Topics, and Overall

Rights and permissions

Reprints and permissions

About this article

Cite this article

Nuckols, T.K., Lim, YW., Wynn, B.O. et al. Rigorous Development does not Ensure that Guidelines are Acceptable to a Panel of Knowledgeable Providers. J GEN INTERN MED 23, 37–44 (2008).

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: