Advertisement

Video-based assessment for laparoscopic fundoplication: initial development of a robust tool for operative performance assessment

  • E. Matthew RitterEmail author
  • Aimee K. Gardner
  • Brian J. Dunkin
  • Linda Schultz
  • Aurora D. Pryor
  • Liane Feldman
2019 SAGES Oral
  • 19 Downloads

Abstract

Introduction

While better technical performance correlates with improved outcomes, there is a lack of procedure-specific tools to perform video-based assessment (VBA). SAGES is developing a series of VBA tools with enough validity evidence to allow reliable measurement of surgeon competence. A task force was established to develop a VBA tool for laparoscopic fundoplication using an evidence-based process that can be replicated for additional procedures. The first step in this process was to seek content validity evidence.

Methods

Forty-two subject matter experts (SME) in laparoscopic fundoplication were interviewed to obtain consensus on procedural steps, identify potential variations in technique, and to generate an inventory of required skills and common errors. The results of these interviews were used to inform creation of a task inventory questionnaire (TIQ) that was delivered to a larger SME group (n = 188) to quantify the criticality and difficulty of the procedural steps, the impact of potential errors associated with each step, the technical skills required to complete the procedure, and the likelihood that future techniques or technologies may change the presence or importance of any of these factors. Results of the TIQ were used to generate a list of steps, skills, and errors with strong validity evidence.

Results

Initial SMEs interviewed included fellowship program directors (45%), recent fellows (24%), international surgeons (19%), and highly experienced super SMEs with quality outcomes data (12%). Qualitative analysis of interview data identified 6 main procedural steps (visualization, hiatal dissection, fundus mobilization, esophageal mobilization, hiatal repair, and wrap creation) each with 2–5 sub steps. Additionally, the TIQ identified 5–10 potential errors for each step and 11 key technical skills required to perform the procedure. Based on the TIQ, the mean criticality and difficulty scores for the 11/21 sub steps included in the final scoring rubric is 4.66/5 (5 = absolutely essential for patient outcomes) and 3.53/5 (5 = difficulty level requires significant experience and use of alternative strategies to accomplish consistently), respectively. The mean criticality and frequency scores for the 9/11 technical skills included is 4.51/5 and 4.51/5 (5 = constantly used ≥ 80% of the time), respectively. The mean impact score of the 42/47 errors incorporated into the final rubric is 3.85/5 (5 = significant error that is unrecoverable, or even if recovered, likely to have a negative impact on patient outcome).

Conclusions

A rigorous, multi-method process has documented the content validity evidence for the SAGES video-based assessment tool for laparoscopic fundoplication. Work is ongoing to pilot the assessment tool on recorded fundoplication procedures to establish reliability and further validity evidence.

Keywords

Video-based assessment Assessment Fundoplication Content validity Test development 

Notes

Compliance with ethical standards

Disclosures

Dr. Ritter reports royalties from the Henry M. Jackson Foundation for the Advancement of Military Medicine outside the submitted work. Dr. Gardner reports fees for contracted consultant work from the SAGES Foundation during the conduct of the study. Dr. Dunkin reports fees for contracted consultant work from the SAGES Foundation during the conduct of the study and is currently employed by Boston Scientific. His employment with Boston Scientific occurred outside of the submitted work. Ms. Shultz has nothing to disclose. Dr. Pryor reports speaking fees from Ethicon, Gore Medical, Medtronic, and Stryker, all outside the submitted work. Dr. Feldman reports research Grants from Medtronic and Merck, both outside the submitted work.

References

  1. 1.
    Fecso AB, Szasz P, Kerezov G, Grantcharov TP (2017) The effect of technical performance on patient outcomes in surgery: a systematic review. Ann Surg 265:492–501CrossRefGoogle Scholar
  2. 2.
    Birkmeyer JD, Finks JF, O’Reilly A, Oerline M, Carlin AM, Nunn AR, Dimick J, Banerjee M, Birkmeyer NJ, Michigan Bariatric Surgery C (2013) Surgical skill and complication rates after bariatric surgery. N Engl J Med 369:1434–1442CrossRefGoogle Scholar
  3. 3.
    Hogg ME, Zenati M, Novak S, Chen Y, Jun Y, Steve J, Kowalsky SJ, Bartlett DL, Zureikat AH, Zeh HJ 3rd (2016) Grading of surgeon technical performance predicts postoperative pancreatic fistula for pancreaticoduodenectomy independent of patient-related variables. Ann Surg 264:482–491CrossRefGoogle Scholar
  4. 4.
    Mackenzie H, Ni M, Miskovic D, Motson RW, Gudgeon M, Khan Z, Longman R, Coleman MG, Hanna GB (2015) Clinical validity of consultant technical skills assessment in the English National Training Programme for laparoscopic colorectal surgery. Br J Surg 102:991–997CrossRefGoogle Scholar
  5. 5.
    Grenda TR, Pradarelli JC, Dimick JB (2016) Using surgical video to improve technique and skill. Ann Surg 264:32–33CrossRefGoogle Scholar
  6. 6.
    Tam V, Zeh HJ 3rd, Hogg ME (2017) Incorporating metrics of surgical proficiency into credentialing and privileging pathways. JAMA Surg 152:494–495CrossRefGoogle Scholar
  7. 7.
    Jones DB, Stefanidis D, Korndorffer JR Jr, Dimick JB, Jacob BP, Schultz L, Scott DJ (2017) SAGES University MASTERS Program: a structured curriculum for deliberate, lifelong learning. Surg Endosc 31:3061–3071CrossRefGoogle Scholar
  8. 8.
    Bilgic E, Al Mahroos M, Landry T, Fried GM, Vassiliou MC, Feldman LS (2019) Assessment of surgical performance of laparoscopic benign hiatal surgery: a systematic review. Surg Endosc.  https://doi.org/10.1007/s00464-019-06662-9 Google Scholar
  9. 9.
    American Educational Research Association APA, National Council on Measurement in Education (2014) Standards for educational and psychological testing. AERA, Washington, DCGoogle Scholar
  10. 10.
    Ag DC (1986) The vaidity of credentialing examinations. Eval Health Prof 9:137–169CrossRefGoogle Scholar
  11. 11.
    Kane MT (1985) The validity of licensure examinations. Am Psychol 7:911–918Google Scholar
  12. 12.
    Annett J (2003) Heirarchical task analysis. In: Hollnage E (ed) Handbook of cognitive task design. LEA, New Jersey, pp 17–36CrossRefGoogle Scholar
  13. 13.
    Brannick MT, Pearlman K, Sanchez JI (2017) Work analysis. In: Farr J, Tippins NT (eds) Handbook of employee selection. Routledge, New YorkGoogle Scholar
  14. 14.
    Nelson D (1994) Job analysis for licnsure and certification exams: science or politics. Educ Measurement 13:29–35CrossRefGoogle Scholar
  15. 15.
    Jc F (1954) The critical incident technique. Psychol Bull 51:317–358CrossRefGoogle Scholar
  16. 16.
    Downing SM, Haladyna TM (2004) Validity threats: overcoming interference with proposed interpretations of assessment data. Med Educ 38:327–333CrossRefGoogle Scholar
  17. 17.
    Ahmed K, Miskovic D, Darzi A, Athanasiou T, Hanna GB (2011) Observational tools for assessment of procedural skills: a systematic review. Am J Surg 202(469–480):e466Google Scholar
  18. 18.
    Downing SM (2003) Validity: on meaningful interpretation of assessment data. Med Educ 37:830–837CrossRefGoogle Scholar
  19. 19.
    Korndorffer JR Jr, Kasten SJ, Downing SM (2010) A call for the utilization of consensus standards in the surgical education literature. Am J Surg 199:99–104CrossRefGoogle Scholar
  20. 20.
    Miskovic D, Ni M, Wyles SM, Kennedy RH, Francis NK, Parvaiz A, Cunningham C, Rockall TA, Gudgeon AM, Coleman MG, Hanna GB, National Training Programme in Laparoscopic Colorectal Surgery in E (2013) Is competency assessment at the specialist level achievable? A study for the national training programme in laparoscopic colorectal surgery in England. Ann Surg 257:476–482CrossRefGoogle Scholar
  21. 21.
    Zevin B, Bonrath EM, Aggarwal R, Dedy NJ, Ahmed N, Grantcharov TP, ATLAS groups (2013) Development, feasibility, validity, and reliability of a scale for objective assessment of operative performance in laparoscopic gastric bypass surgery. J Am Coll Surg 216:955–965 (quiz 1029–1031, 1033) CrossRefGoogle Scholar

Copyright information

© This is a U.S. government work and not under copyright protection in the U.S.; foreign copyright protection may apply 2019

Authors and Affiliations

  1. 1.Department of SurgeryUniformed Services University of the Health Sciences and Walter Reed National Military Medical CenterBethesdaUSA
  2. 2.Baylor College of MedicineHoustonUSA
  3. 3.Houston MethodistWeill Cornell Medical CollegeHoustonUSA
  4. 4.Society of American Gastrointestinal and Endoscopic SurgeonsLos AngelesUSA
  5. 5.Department of SurgeryStony Brook University Renaissance School of MedicineStony BrookUSA
  6. 6.Department of SurgeryMcGill UniversityMontrealCanada

Personalised recommendations