European Archives of Oto-Rhino-Laryngology

, Volume 276, Issue 5, pp 1525–1531 | Cite as

Inter-rater reliability between experienced and inexperienced otolaryngologists using Koo’s drug-induced sleep endoscopy classification system

  • Soo Kweon KooEmail author
  • Sang Hoon Lee
  • Tae Kyung Koh
  • Young Jun Kim
  • Ji Seung Moon
  • Ho Byung Lee
  • Geun Hyung Park



An ideal, drug-induced sleep endoscopy (DISE) classification system should cover all the upper airways, be simple and practical, and quantify the severity of any obstruction. Excellent validity and reliability are essential. We explored the inter-rater reliability of Koo’s DISE classification system in the hands of experienced and inexperienced otolaryngologists.


We retrospectively analyzed video images of 100 patients who underwent DISE examination in our hospital between 2015 and 2018. Three experienced and three inexperienced otolaryngologists reviewed and scored all images. We calculated the inter-rater reliabilities of the two groups of otolaryngologists.


Independent of the extent of experience with DISE, detection of retropalatal obstructions (overall agreement: 0.87; kappa value: 0.60), and the degree of such obstructions (overall agreement: 0.67; kappa value: 0.52) were more consistent than were the detection of retrolingual obstructions (overall agreement: 0.61, kappa value: 0.37) and the degree of retrolingual obstructions (overall agreement: 0.20, kappa value: 0.35). Inexperienced observers were in good agreement for palatal obstructions and experienced observers were in good agreement for tongue-base obstructions. All of the otolaryngologists found it difficult to detect a lateral pharyngeal wall obstruction at the retrolingual level.


Koo’s DISE classification system focuses on surgical treatment, especially by otolaryngologists, and the degree of agreement between the experienced and inexperienced observers was relatively high. The participants’ level of experience had a strong impact on scoring. The less-experienced otolaryngologists tended to overlook tongue-base obstructions, focusing instead on relatively simple retropalatal obstructions. In the future, development of a DISE classification system that can be accepted globally will be necessary.


Drug-induced sleep endoscopy Sleep apnea, obstructive Observer variation Reliability 


Compliance with ethical standards

Conflict of interest

We have no potential conflict of interest.

Research involving human participants and/or animals

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed consent

Informed consent was obtained from all individual participants included in the study. The study was approved by the Institutional Review Board of Busan Saint Mary’s Medical Center (Approval no. BSM-2018-13).


  1. 1.
    Kerizian EJ, Hohenhorst W, de Vries N (2011) Drug-induced sleep endoscopy: the VOTE classification. Eur Arch Otorhinolaryngol 268(8):1233–1236. CrossRefGoogle Scholar
  2. 2.
    Kezirian EJ, White DP, Malhotra A, Ma W, McCulloch CE, Goldberg AN (2010) Interrater reliability of drug-induced sleep endoscopy. Arch Otolaryngol Head Neck Surg 136(4):393–397. CrossRefGoogle Scholar
  3. 3.
    Pringle MB, Croft CB (1991) A comparison of sleep nasendoscopy and the Muller manoeuvre. Clin Otolaryngol Allied Sci 16(6):559–562CrossRefGoogle Scholar
  4. 4.
    Amos JM, Durr ML, Nardone HC, Baldassari CM, Duggins A, Ishman SL (2018) Systematic review of drug-induced sleep endoscopy scoring systems. Otolaryngol Head Neck Surg 158(2):240–248. CrossRefGoogle Scholar
  5. 5.
    Koo SK, Choi JW, Myung NS, Lee HJ, Kim YJ, Kim YJ (2013) Analysis of obstruction site in obstructive sleep apnea syndrome patients by drug-induced sleep endoscopy. Am J Otolaryngol 34(6):626–630. CrossRefGoogle Scholar
  6. 6.
    De Vito A, Carrasco Llatas M, Ravesloot MJ, Kotecha B, De Vries N, Hamans E et al (2018) European position paper on drug-induced sleep endoscopy: 2017 update. Clin Otolaryngol 43(6):1541–1552. CrossRefGoogle Scholar
  7. 7.
    Fujita S, Conway W, Zorick F, Roth T (1981) Surgical correction of anatomic abnormalities in obstructive sleep apnea syndrome: uvulopalatopharyngoplasty. Otolaryngol Head Neck Surg 89(6):923–934CrossRefGoogle Scholar
  8. 8.
    Koo SK, Kwon SB, Kim YJ, Moon JS, Kim YJ, Jung SH (2017) Acoustic analysis of snoring sounds recorded with a smartphone according to obstruction site in OSAS patients. Eur Arch Otorhinolaryngol 274(3):1735–1740. CrossRefGoogle Scholar
  9. 9.
    Koo SK, Ahn GY, Choi JW, Kim YJ, Jung SH, Moon JS, Lee YI (2017) Obstructive sleep apnea in postmenopausal women: a comparative study using drug-induced sleep endoscopy. Braz J Otorhinolaryngol 83(3):285–291. CrossRefGoogle Scholar
  10. 10.
    Jung SH, Koo SK, Choi JW, Moon JS, Lee SH (2017) Upper airway structural changes induced by CPAP in OSAS patients: a study using drug-induced sleep endoscopy. Eur Arch Otorhinolaryngol 274(1):247–252. CrossRefGoogle Scholar
  11. 11.
    Koo SK, Kwon SB, Moon JS, Lee SH, Lee HB, Lee SJ (2018) Comparison of snoring sounds between natural and drug-induced sleep recorded using a smartphone. Auris Nasus Larynx 45(4):777–782. CrossRefGoogle Scholar
  12. 12.
    Rodriguez-Bruno K, Goldberg AN, McCulloch CE, Kezirian EJ (2009) Test-retest reliability of drug-induced sleep endoscopy. Otolaryngol Head Neck Surg 140(5):646–651. CrossRefGoogle Scholar
  13. 13.
    Vroegop AV, Vanderveken OM, Hamans E, Dieltjens M, Michels NR, Hogenhorst W et al (2013) Observer variation in drug-induced sleep endoscopy: experienced versus nonexperienced ear, nose and throat surgeons. Sleep 36(6):947–953. CrossRefGoogle Scholar
  14. 14.
    Hernández-Gancedo C, Pestaña D, Peña N, Royo C, Pérez-Chrzanowska H, Criado A (2006) Monitoring sedation in critically ill patients: bispectral index, Ramsay and observer scales. Eur J Anaesthesiol 23(8):649–653. CrossRefGoogle Scholar
  15. 15.
    Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33(1):159–174CrossRefGoogle Scholar
  16. 16.
    Bachar G, Nageris B, Feinmesser R, Hadar T, Yaniv E, Shpitzer T et al (2012) Novel grading system for quantifying upper-airway obstruction on sleep endoscopy. Lung 190(3):313–318. CrossRefGoogle Scholar
  17. 17.
    Chan DK, Liming BJ, Horn DL, Parikh SR (2014) A new scoring system for upper airway pediatric sleep endoscopy. JAMA Otolaryngol Head Neck Surg 140(7):595–602. CrossRefGoogle Scholar
  18. 18.
    Pringle MB, Croft CB (1993) A grading system for patients with obstructive sleep apnoea-based on sleep nasendoscopy. Clin Otolaryngol Allied Sci 18(6):480–484CrossRefGoogle Scholar
  19. 19.
    Randerath WJ, Verbraecken J, Andreas S, Bettega G, Bodewyns A, Hamans E et al (2011) Non-CPAP therapies in obstructive sleep apnoea. Eur Respir J 37(5):1000–1028. CrossRefGoogle Scholar
  20. 20.
    Cho KS, Koo SK, Lee JK, Hong SL, Capasso R, Roh HJ (2014) Limited palatal muscle resection with tonsillectomy: a novel palatopharyngoplasty technique for obstructive sleep apnea. Auris Nasus Larynx 41(6):558–562. CrossRefGoogle Scholar
  21. 21.
    Chi JC, Chiang RP, Chou TY, Shu CH, Shiao AS, Lin CM (2015) The role of lateral pharyngoplasty in obstructive sleep apnea syndrome. Eur Arch Otorhinolaryngol 272(2):489–496. CrossRefGoogle Scholar
  22. 22.
    Güler İ, Kuzucu İ, Baklacı D, Kum RO, Kum NY, Özcan M (2018) Efficiency of expansion sphincter pharyngoplasty in the treatment of obstructive sleep apnea syndrome. Turk Arch Otorhinolaryngol 56(4):206–209. CrossRefGoogle Scholar
  23. 23.
    Miller SC, Nguyen SA, Ong AA, Gillespie MB (2017) Transoral robotic base of tongue reduction for obstructive sleep apnea: a systematic review and meta-analysis. Laryngoscope 127(1):258–265. CrossRefGoogle Scholar
  24. 24.
    Samutsakorn P, Hirunwiwatkui P, Chaitusaney B, Charakorn N (2018) Lingual tonsillectomy with palatal surgery for the treatment of obstructive sleep apnea in adults: a systematic review and meta-analysis. Eur Arch Otorhinolaryngol 275(4):1005–1013. CrossRefGoogle Scholar
  25. 25.
    Certal VF, Zaghi S, Riaz M, Vieira AS, Kushida C, Capasso R et al (2015) Hypoglossal nerve stimulation in the treatment of obstructive sleep apnea: a systematic review and meta-analysis. Laryngoscope 125(5):;1254–1264. CrossRefGoogle Scholar
  26. 26.
    Woodson BT (2017) Updated hypopharyngeal surgery for sleep apnea. Adv Otorhinolaryngol 80:81–89. Google Scholar
  27. 27.
    Lin HC, Weaver EM, Lin HS, Friedman M (2017) Multilevel obstructive sleep apnea surgery. Adv Otorhinolaryngol 80:109–115. Google Scholar
  28. 28.
    Boudewyns A, Verhulst S, Maris M, Saldien V, Van de Heyning P (2014) Drug-induced sedation endoscopy in pediatric obstructive sleep apnea syndrome. Sleep Med 15(12):1526–1531. CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department Otorhinolaryngology-Head and Neck SurgeryBusan Saint Mary’s HospitalBusanSouth Korea

Personalised recommendations