European Archives of Oto-Rhino-Laryngology

, Volume 275, Issue 5, pp 1319–1325 | Cite as

Diagnostic flexible pharyngo-laryngoscopy: development of a procedure specific assessment tool using a Delphi methodology

  • Jacob Melchiors
  • Mikael Johannes Vuokko Henriksen
  • Frederik G. Dikkers
  • Javier Gavilán
  • J. Pieter Noordzij
  • Marvin P. Fried
  • Daniel Novakovic
  • Johannes Fagan
  • Birgitte W. Charabi
  • Lars Konge
  • Christian von Buchwald



Proper training and assessment of skill in flexible pharyngo-laryngoscopy are central in the education of otorhinolaryngologists. To facilitate an evidence-based approach to curriculum development in this field, a structured analysis of what constitutes flexible pharyngo-laryngoscopy is necessary. Our aim was to develop an assessment tool based on this analysis.


We conducted an international Delphi study involving experts from twelve countries in five continents. Utilizing reiterative assessment, the panel defined the procedure and reached consensus (defined as 80% agreement) on the phrasing of an assessment tool.


Fifty panelists completed the Delphi process. The median age of the panelists was 44 years (range 33–64 years). Median experience in otorhinolaryngology was 15 years (range 6–35 years). Twenty-five were specialized in laryngology, 16 were head and neck surgeons, and nine were general otorhinolaryngologists. An assessment tool was created consisting of twelve distinct items.


The gathering of validity evidence for assessment of core procedural skills within Otorhinolaryngology is central to the development of a competence-based education. The use of an international Delphi panel allows for the creation of an assessment tool which is widely applicable and valid. This work allows for an informed approach to technical skills training for flexible pharyngo-laryngoscopy and as further validity evidence is gathered allows for a valid assessment of clinical performance within this important skillset.


Flexible laryngoscopy Delphi-study Assessment tool Education Technical skills 



The participants of the Delphi panel who gave their time and knowledge to this project (listed in Appendix 1). Our esteemed colleagues Aaltonen LM, Weinstein G, Wei WI who provided invaluable assistance in the recruitment of the panel.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflicts of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Supplementary material

405_2018_4904_MOESM1_ESM.jpg (4.6 mb)
Supplementary material 1 (JPG 4676 KB)
405_2018_4904_MOESM2_ESM.jpg (11.5 mb)
Supplementary material 2 (JPG 11740 KB)


  1. 1.
    Garcia M (1854) Observations on the human voice. Proc R Soc Lond 7:399–410CrossRefGoogle Scholar
  2. 2.
    Regionernes Lønnings- og Takstnævn (2015) Danske Reg [Internet].
  3. 3.
    Sethi RKV, Kozin ED, Remenschneider AK, Lee DJ, Gray ST, Shrime MG et al (2014) Subspecialty emergency room as alternative model for otolaryngologic care: implications for emergency health care delivery. Am J Otolaryngol Head Neck Surg 35:758–765Google Scholar
  4. 4.
    Couch ME (2010) Cummings otolaryngology—head and neck surgery, Fifth Edit. Cummings Otolaryngology Head and Neck Surgery. Copyright © 2010, 2005, 1998, 1993, 1986 by Mosby, Inc. All Rights Reserved, pp 93–107Google Scholar
  5. 5.
    Paul BC, Rafii B, Achlatis S, Amin MR, Branski RC (2012) Morbidity and patient perception of flexible laryngoscopy. Ann Otol Rhinol Laryngol 121(11):708–713CrossRefPubMedGoogle Scholar
  6. 6.
    Ongkasuwan J, Yung KC, Courey MS (2012) The physiologic impact of transnasal flexible endoscopy. Laryngoscope 122(6):1331–1334CrossRefPubMedGoogle Scholar
  7. 7.
    Sethi RKV, Kozin ED, Remenschneider AK, Lee DJ, Gliklich RE, Shrime MG et al (2015) Otolaryngology-specific emergency room as a model for resident training. Laryngoscope 125(1):99–104CrossRefPubMedGoogle Scholar
  8. 8.
    Reznick RK (1993) Teaching and testing technical skills. Am J Surg 165(3):358–361CrossRefPubMedGoogle Scholar
  9. 9.
    Nasca TJ, Philibert I, Brigham T, Flynn TC (2012) The next GME accreditation system—rationale and benefits. N Engl J Med 366(11):1051–1056CrossRefPubMedGoogle Scholar
  10. 10.
    Swing SR, Clyman SG, Holmboe ES, Williams RG (2009) Advancing resident assessment in graduate medical education. J Grad Med Educ 1(2):278–286CrossRefPubMedPubMedCentralGoogle Scholar
  11. 11.
    Ghaderi I, Manji F, Park YS, Juul D, Ott M, Harris I et al (2015) Technical skills assessment toolbox. Ann Surg 261(2):251–262CrossRefPubMedGoogle Scholar
  12. 12.
    Messick S (1995) Validity of psychological assessment: validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. Am Psychol 50(22):741–749CrossRefGoogle Scholar
  13. 13.
    Cook D, Beckman TJ (2006) Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med 119(2):166.e7–166.e16CrossRefGoogle Scholar
  14. 14.
    Stefanidis D, Arora S, Parrack DM, Hamad GG, Capella J, Grantcharov T et al (2012) Research priorities in surgical simulation for the 21st century. Am J Surg 203(1):49–53CrossRefPubMedPubMedCentralGoogle Scholar
  15. 15.
    Robinson HF, Dennick R (2015) Teaching laryngeal endoscopy skills to speech and language therapists. Curr Opin Otolaryngol Head Neck Surg 23(3):197–201CrossRefPubMedGoogle Scholar
  16. 16.
    Herrmann IF, Domagk D, Strahl M, Arens C (2016) The transnasal videopanendoscopy (ViP): diagnostics of the upper aerodigestive tract using an anatomically correct model from the model to the patient. Eur Arch Otorhinolaryngol 273(3):749–754CrossRefPubMedGoogle Scholar
  17. 17.
    Laeeq K, Pandian V, Skinner M, Masood H, Stewart CM, Weatherly R et al (2010) Learning curve for competency in flexible laryngoscopy. Laryngoscope. 120(10):1950–1953CrossRefPubMedGoogle Scholar
  18. 18.
    Brook CD, Platt MP, Russell K, Grillone G, Aliphas A, Noordzij JP (2015) Time to competency, reliability of flexible transnasal laryngoscopy by training level: A Pilot Study. Otolaryngol Head Neck Surg 152(5):843–850CrossRefPubMedGoogle Scholar
  19. 19.
    Okoli C, Pawlowski SD (2004) The Delphi method as a research tool: an example, design considerations and applications. Inf Manag 42(1):15–29CrossRefGoogle Scholar
  20. 20.
    de Villiers MR, de Villiers PJT, Kent AP (2005) The Delphi technique in health sciences education research. Med Teach 27(7):639–643CrossRefPubMedGoogle Scholar
  21. 21.
    Murry JW, Hammons JO. Delphi (1995) A versatile methodology for conducting qualitative research. Rev High Educ 18(4):423–436CrossRefGoogle Scholar
  22. 22.
    Keeney S, Hasson F, McKenna H (2006) Consulting the Oracle: ten lessons from using the Delphi techniques in nursing research. J Adv Nurs 53(2):205–212CrossRefPubMedGoogle Scholar
  23. 23.
    Streiner DL, Norman GR, Cairney J (2015) Health measurement scales, vol 9. Oxford University Press, OxfordCrossRefGoogle Scholar
  24. 24.
    Walsh CM, Ling SC, Khanna N, Cooper MA, Grover SC, May G et al (2014) Gastrointestinal endoscopy competency assessment tool: development of a procedure-specific assessment tool for colonoscopy. Gastrointest Endosc 79(5):798–807.e5CrossRefPubMedGoogle Scholar
  25. 25.
    Madani A, Watanabe Y, Vassiliou M, Feldman LS, Duh QY, Singer MC et al (2016) Defining competencies for safe thyroidectomy: an international Delphi consensus. Surg (United States) 159(1):86–101Google Scholar
  26. 26.
    Maertens H, Aggarwal R, Macdonald S, Vermassen F, Van Herzeele I, Brodmann M et al (2016) Transatlantic multispecialty consensus on fundamental endovascular skills: results of a Delphi Consensus Study. Eur J Vasc Endovasc Surg 51(1):141–149CrossRefPubMedGoogle Scholar
  27. 27.
    Downing S, Yudkowsky R (2009) Assessment in health professions education. Routledge, New YorkGoogle Scholar
  28. 28.
    Ilgen JS, Ma IWY, Hatala R, Cook D (2015) A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment. Med Educ 49:161–173CrossRefPubMedGoogle Scholar
  29. 29.
    Tolsgaard MG, Todsen T, Sorensen JL, Ringsted C, Lorentzen T, Ottesen B et al (2013) International multispecialty consensus on how to evaluate ultrasound competence: a Delphi Consensus Survey. PLoS One 8(2):e57687. CrossRefPubMedPubMedCentralGoogle Scholar
  30. 30.
    Thomas W (2006) Teaching and assessing surgical competence. Ann R Coll Surg Engl 88(5):429–432CrossRefPubMedPubMedCentralGoogle Scholar
  31. 31.
    Miller GE (1990) The assessment of clinical skills/competence/performance. Acad Med 65(9):63–67CrossRefGoogle Scholar
  32. 32.
    Melchiors J, Todsen T, Nilsson P, Kohl AP, Bøttger M, Charabi B et al (2016) Self-directed simulation-based training of emergency cricothyroidotomy: a route to lifesaving skills. Eur Arch Otorhinolaryngol 273(12):4623–4628CrossRefPubMedGoogle Scholar
  33. 33.
    Mcgaghie WC (2015) Mastery learning: it is time for medical education to join the 21st Century. Acad Med 90(11):1438–1441CrossRefPubMedGoogle Scholar
  34. 34.
    Lineberry M, Soo Park Y, Cook D, Yudkowsky R (2015) Making the case for mastery learning assessments. Acad Med 90(11):1CrossRefGoogle Scholar
  35. 35.
    Melchiors J, Todsen T, Nilsson P, Wennervaldt K, Charabi B, Bøttger M et al (2015) Preparing for emergency: a valid, reliable assessment tool for emergency cricothyroidotomy skills. Otolaryngol Head Neck Surg 152(2):260–265CrossRefPubMedGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  • Jacob Melchiors
    • 1
    • 2
  • Mikael Johannes Vuokko Henriksen
    • 2
  • Frederik G. Dikkers
    • 3
  • Javier Gavilán
    • 4
  • J. Pieter Noordzij
    • 5
  • Marvin P. Fried
    • 6
  • Daniel Novakovic
    • 7
  • Johannes Fagan
    • 8
  • Birgitte W. Charabi
    • 1
  • Lars Konge
    • 2
  • Christian von Buchwald
    • 1
  1. 1.Department of Otorhinolaryngology-Head and Neck Surgery and AudiologyRigshospitaletCopenhagen EDenmark
  2. 2.Copenhagen Academy for Medical Education and SimulationCopenhagen, Capital RegionDenmark
  3. 3.Department of Otorhinolaryngology, Academic Medical CenterUniversity of AmsterdamAmsterdamThe Netherlands
  4. 4.Department of OtolaryngologyLa Paz University HospitalMadridSpain
  5. 5.Department of Otolaryngology-Head and Neck SurgeryBoston University School of MedicineBostonUSA
  6. 6.Department of Otorhinolaryngology-Head and Neck SurgeryMontefiore Medical CenterBronxUSA
  7. 7.Canterbury HospitalUniversity of Sydney Medical SchoolSt. LeonardsAustralia
  8. 8.Division of OtolaryngologyUniversity of Cape TownCape TownSouth Africa

Personalised recommendations