Usability testing of EirV3—a computer-based tool for patient-reported outcome measures in cancer
- 126 Downloads
Eir version 3 (V3) is an electronic tool for administration of patient-reported outcome measures (Eir-Patient) that immediately presents patient scores on the physician’s computer (Eir-Doctor). Perceived usability is an important determinant for successful implementation. The aim of this study was to answer the following research question evaluated at the cancer outpatient clinics, in the patients’ home, and at general practitioners’ (GPs) offices: What are the number, type, and severity of usability issues evaluated by the patient (Eir-Patient module) and by the physician (Eir-Doctor module)?
A usability evaluation using observations, think-aloud sessions, individual interviews and focus group interviews in cancer patients and their physicians was conducted. Identified usability issues were graded on a severity scale from 1 (irritant) to 4 (unusable).
Overall, 73 Eir registrations were performed by 37 patients, and used by 17 physicians in clinical consultations. All patients were able to complete the Eir-Patient symptom registration. Seventy-two usability issues were identified. None of them were graded as unusable. For the Eir-Patient module, 62% of the identified usability issues was graded as irritant (grade 1), 18% as moderate (grade 2), and 20% as severe (grade 3). For the Eir-Doctor module, 46% of the identified usability issues were graded as irritant, 36% as moderate and 18% as severe.
In the updated Eir version, issues in the severe and moderate categories have been changed, to optimize the usability of using real-time PROMs in clinical practice.
KeywordsPatient-reported outcome measures PROMs Usability Feasibility Electronic patient-reported outcomes
We would like to thank all participating patients and physicians in this study, as well as the Coordination Unit Orkdal Region for their project management contribution.
The study was funded by the Regional Research Funds in Norway, Mid-Norway, and the European Palliative Care Research Centre, Department of Clinical and Molecular Medicine, NTNU, Norwegian University of Science and Technology, and St. Olavs Hospital, Trondheim University Hospital.
Compliance with ethical standards
Disclosure of potential conflicts of interests
Krogstad, Sundt-Hansen, Hjermstad, Hågensen, Raj, Steinsbekk, and Sand have declared no conflicts of interests. Eir Solutions AS was established in 2015 with Kaasa, Loge, and NTNU technology Transfer AS/Andersen as shareholders. No income, dividend, or financial benefits are related to the work presented here nor in relation to Eir in any way.
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Informed consent was obtained from all individual participants included in the study.
- 1.Stacey D, Legare F, Col NF, Bennett CL, Barry MJ, Eden KB, Holmes-Rovner M, Llewellyn-Thomas H, Lyddiatt A, Thomson R, Trevena L, Wu JH (2014) Decision aids for people facing health treatment or screening decisions. Cochrane Database Syst Rev (1):Cd001431. https://doi.org/10.1002/14651858.CD001431.pub4
- 3.Person-centred care made simple (2014) www.health.org.uk/publications/person-centred-care-made-simple
- 4.Basch E, Deal AM, Kris MG, Scher HI, Hudis CA, Sabbatini P, Rogak L, Bennett AV, Dueck AC, Atkinson TM, Chou JF, Dulko D, Sit L, Barz A, Novotny P, Fruscione M, Sloan JA, Schrag D (2016) Symptom monitoring with patient-reported outcomes during routine cancer treatment: a randomized controlled trial. J Clin Oncol Off J Am Soc Clin Oncol 34(6):557–565. https://doi.org/10.1200/jco.2015.63.0830 CrossRefGoogle Scholar
- 6.Strasser F, Blum D, von Moos R, Cathomas R, Ribi K, Aebi S, Betticher D, Hayoz S, Klingbiel D, Brauchli P, Haefner M, Mauri S, Kaasa S, Koeberle D (2016) The effect of real-time electronic monitoring of patient-reported symptoms and clinical syndromes in outpatient workflow of medical oncologists: E-MOSAIC, a multicenter cluster-randomized phase III study (SAKK 95/06). Ann Oncol 27(2):324–332. https://doi.org/10.1093/annonc/mdv576 CrossRefGoogle Scholar
- 8.Food and Drug Administration USDoHaHS (2009) Guidance for industry. Patient-reported outcome measures: use in medical product development to support labeling claims. https://www.fda.gov/downloads/drugs/guidancecomplianceregulatoryinformation/guidances/ucm193282.pdf
- 9.Zikos E, Ghislain I, Coens C, Ediebah DE, Sloan E, Quinten C, Koller M, van Meerbeeck JP, Flechtner HH, Stupp R, Pallis A, Czimbalmos A, Sprangers MA, Bottomley A (2014) Health-related quality of life in small-cell lung cancer: a systematic review on reporting of methods and clinical issues in randomised controlled trials. Lancet Oncol 15(2):e78–e89. https://doi.org/10.1016/S1470-2045(13)70493-5 CrossRefGoogle Scholar
- 10.Gravis G, Marino P, Joly F, Oudard S, Priou F, Esterni B, Latorzeff I, Delva R, Krakowski I, Laguerre B, Rolland F, Theodore C, Deplanque G, Ferrero JM, Pouessel D, Mourey L, Beuzeboc P, Zanetta S, Habibian M, Berdah JF, Dauba J, Baciuchka M, Platini C, Linassier C, Labourey JL, Machiels JP, El Kouri C, Ravaud A, Suc E, Eymard JC, Hasbini A, Bousquet G, Soulie M, Fizazi K (2014) Patients’ self-assessment versus investigators’ evaluation in a phase III trial in non-castrate metastatic prostate cancer (GETUG-AFU 15). Eur J Cancer 50(5):953–962. https://doi.org/10.1016/j.ejca.2013.11.034 CrossRefGoogle Scholar
- 13.Benze G, Nauck F, Alt-Epping B, Gianni G, Bauknecht T, Ettl J, Munte A, Kretzschmar L, Gaertner J (2017) PROutine: a feasibility study assessing surveillance of electronic patient reported outcomes and adherence via smartphone app in advanced cancer. Ann Palliat Med 6:705. https://doi.org/10.21037/apm.2017.07.05 CrossRefGoogle Scholar
- 14.Hjermstad MJ, Lie HC, Caraceni A, Currow DC, Fainsinger RL, Gundersen OE, Haugen DF, Heitzer E, Radbruch L, Stone PC, Strasser F, Kaasa S, Loge JH (2012) Computer-based symptom assessment is feasible in patients with advanced cancer: results from an international multicenter study, the EPCRC-CSA. J Pain Symptom Manag 44(5):639–654. https://doi.org/10.1016/j.jpainsymman.2011.10.025 CrossRefGoogle Scholar
- 15.Krogstad H, Brunelli C, Sand K, Andersen E, Garresori H, Halvorsen T, Haukland EC, Jordal F, Kaasa S, Loge JH, Løhre ET, Raj SX, Hjermstad MJ (2017) Development of EirV3: a computer-based tool for patient-reported outcome measures in cancer. JCO Clin Cancer Inf. https://doi.org/10.1200/CCI.17.0051
- 16.Wintner LM, Giesinger JM, Zabernigg A, Rumpold G, Sztankay M, Oberguggenberger AS, Gamper EM, Holzner B (2015) Evaluation of electronic patient-reported outcome assessment with cancer patients in the hospital and at home. BMC Med Inf Decis Mak 15:110. https://doi.org/10.1186/s12911-015-0230-y CrossRefGoogle Scholar
- 17.International Organization for Standardization/International Electrotehcnical Commission (ISO/IEC) (1998) Ergonomic requirements for office work with visual display terminals (VDTs)-Part 11- Guidance on usability 9241-11Google Scholar
- 20.European Palliative Care Research Centre (PRC), . www.ntnu.edu/PRC
- 21.Sigurdardottir KR, Kaasa S, Rosland JH, Bausewein C, Radbruch L, Haugen DF (2014) The European Association for palliative care basic dataset to describe a palliative care cancer population: results from an international Delphi process. Palliat Med 28(6):463–473. https://doi.org/10.1177/0269216314521264 CrossRefGoogle Scholar
- 25.Morin CM (1996) Insomnia: psychological assessment and managementGoogle Scholar
- 26.Common Terminology Criteria for Adverse Events (CTCAE) Version 4.0 (2009)Google Scholar
- 27.Rubin J, Chisnell D (2008) Handbook of usability testing: how to plan, design, and conduct effective tests. WileyGoogle Scholar
- 28.Lewis C (1982) Using the “think aloud” method in cognitive interface design. IBM: Technical report RC-9265, Yorktown HeightsGoogle Scholar
- 29.Stinson J, McGrath P, Hodnett E, Feldman B, Duffy C, Huber A, Tucker L, Hetherington R, Tse S, Spiegel L, Campillo S, Gill N, White M (2010) Usability testing of an online self-management program for adolescents with juvenile idiopathic arthritis. J Med Internet Res 12(3):e30. https://doi.org/10.2196/jmir.1349 CrossRefGoogle Scholar
- 33.Walker J, Bohnke JR, Cerny T, Strasser F (2010) Development of symptom assessments utilising item response theory and computer-adaptive testing—a practical method based on a systematic review. Crit Rev Oncol Hematol 73(1):47–67. https://doi.org/10.1016/j.critrevonc.2009.03.007 CrossRefGoogle Scholar
- 37.Jordan K, Aapro M, Kaasa S, Ripamonti CI, Scotté F, Strasser F, Young A, Bruera E, Herrstedt J, Keefe D, Laird B, Walsh D, Douillard JY, Cervantes A (2018) European Society for Medical Oncology (ESMO) position paper on supportive and palliative care. Ann Oncol 29(1):36–43. https://doi.org/10.1093/annonc/mdx757 CrossRefGoogle Scholar
- 40.Palliative Care Integrated in Oncology (PALLiON) (2017) https://pallion.no/
- 42.Brooke J (1996) SUS: a “quick and dirty” usability scale. In: Jordan PW, Thomas B, Weerdmeester BA, McClelland IL (eds) Usability evaluation in industry. Taylor & Francis, London, pp 189–194Google Scholar