The design and methodology of a usability protocol for the management of medications by families for aging older adults
Health research apps often do not focus on usability as a design priority. This is problematic when the population of interest is disproportionately underrepresented as users of mobile apps, especially observed with aging older adults (> = 75). Challenges with the adoption of health information technology (HIT) among this group are exacerbated by poor design and user interface/experience (UI/UX) choices. This protocol describes the testing and evaluation process of one HIT app for the family-based collaboration platform InfoSAGE.
We aim to recruit twenty subjects from both informal family-caregivers and aging older adults to examine the usability of the InfoSAGE mobile medication manager. Participants will be audio and visually recorded, in addition to the use of screen capture recordings, while ‘thinking aloud’ as they complete eight common use-case scenarios. Multiple independent reviewers will code video and audio recordings for thematic analysis and use problems will be evaluated. Success and failure of each scenario will be determined by completion of sub-events. Time-to-complete analysis will be used to ascertain the learning curve associated with the app.
Frequently observed problem areas will be used as the basis of further evolution of the app, and will further inform generalized recommendations for the design of HIT apps for research and public use. This study aims to improve the model of development for dual user populations with dissimilar technological literacy to improve retention and use. Results of this study will form the foundation of a design framework for mobile health apps.
KeywordsmHealth Usability Older adults Health information technology User interface design
Behavioral Observation Research Interactive Software
Health Information Technology
Information Sharing Across Generations and Environments
Longevity is increasing worldwide , and in the United States the ‘baby-boomer’ generation is rapidly approaching the age of ballooning medical costs that results from higher utilization due to age-related morbidities [2, 3, 4]. The challenges associated with these coming stressors will require scalable solutions throughout the healthcare spectrum. One area of promise is in the use of mobile health information (mHealth) technology to bring efficient, cost-effective, care to a wide audience [5, 6]. While elders are increasing using more mobile phones [7, 8, 9], challenges remain in adoption and technical literacy, especially as age rises [10, 11, 12]. The digital-divide has been decreasing in the last ten years, but is a scarcity of systematically designed studies for the evaluation of the usability of mHealth solutions in mixed-age populations, aimed at both informal caregivers and aging older adults.
Many studies have shown the potential negative impact of poorly designed information technology on facilitating medical errors [13, 14], and specific problems regarding regarding usability of mobile apps for medication [6, 15]. Usability testing has been applied in the assessment of health information system safety to identify and prevent medical errors and patient safety risks that may arise from the use of health information systems. Specifically, such methods have begun to be applied to the assessment of the impact of user interface features and design choices on medical error . One study  evaluated a prototype app that simulates medication tracking using an iPad and found that users struggled with screen glare, button activation, and the “drag and drop” function , which makes it difficult for the significant number of users with poor vision to correctly use those apps - an estimated 1 in 5 North American adults aged 75 or over have a self-reported “seeing diability” . Low health literacy is another barrier to effective interaction with technology - about one-half of North American adults have low literacy, meaning they lack the literacy skills needed for everyday life,  and between 46 and 60% also have low health literacy and struggle to “obtain process, and understand basic health information and services needed to make appropriate health decisions” [19, 20].
Usability problems in mobile apps lead to reduced utilization, lower rates of user retention, and increased user frustration  users who have difficulties navigating an app, understanding button configuration or layout, or find features to be too convoluted or complex are less likely to continue use [21, 22].
To these aims, this protocol describes a systematic method of evaluating a mobile medication management system within the InfoSAGE platform, usable on both iOS and Android operating systems. While the specifics are focused on the InfoSAGE app, the method and approach should be widely applicable to any app under evaluation for age-appropriate usability.
The result of this study will be an evaluation of common problems encountered by frail older adults and their informal caregivers using this mobile app, and a set of recommendations for the design of mobile apps for elders based on our observations and analysis.
Ethical approval and recruitment
The study is approved for ethical review by the Beth Israel Deaconess Medical Center Institutional Review board. Recruitment is on-going, beginning in April 2018, with a target convenience sample of 10 informal caregivers and 10 frail older adults drawn from local advertising, flyers, and online message boards, and a word-of-mouth and grassroots approach through collaborating partners.
Post-testing surveys and open-ended responses
Post-Testing Survey - Likert Scale (1–7)
Overall, I am satisfied with the ease of completing these tasks
Overall, I am satisfied with the amount of time it took me to complete these tasks
Overall, I am satisfied with the usefulness of the on-line help for completing these tasks
How would you rate the difficulty of completing the task scenarios?
Overall, after completing these tasks, I feel that this could potentially be used on a regular basis as part of my patient care, or my care recipient’s care, and for communicating my current list of medications with my care provider.
Did you understand the tasks that you were asked to complete?
Are there any tasks that you found particularly difficult to use?
What changes would you suggest to make the system easier to use?
Is there anything about this usability test process or tasks scenarios that can be improved?
Do you feel that this medication system could potentially improve your patient care?
Testing will take place in a controlled office environment, using supplied development iPads using the publicly available InfoSAGE app. Audio and video recording will be utilized, but video recording will be limited to hands only. Participants will be asked to ‘think aloud’ while interacting with the app during testing scenarios. No faces or other identifiers will be recorded. During testing, touches and actions performed on the development iPad will be captured by screen recording software. Participants will be supplied with a test account, with full logging enabled to remove potential problems in registration or log ins. Prior to testing, participants will be given a brief overview of the InfoSAGE platform, the tiered access system, and a brief overview of its main features. Participants will not be instructed in how to use the app and will only receive help if they are unable to continue with the testing process. Any help provided will be noted.
Scenario tasks and sub-events
Add Lisinopril 5 mg, 1 tablet once daily
Add Warfarin 5 mg, 1 tablet once daily
Add Acetaminophen, 500 mg tablet, 2 tablets by mouth every 6 h, as needed for pain
View the side-effects of Lisinopril
Review drug-drug interactions
Modify the dose of Warfarin to 7.5 mg once daily
Email medication list
Evaluation and analysis
Audio transcripts will be transcribed and coded for broad themes. A heuristic method of coding will be performed on the video recording and screen capture. Two analysts will individually evaluate the videos, using a shared code dictionary, noting the events and codes through the Behavioral Observation Research Interactive Software (BORIS, v.7) . Review of the coded videos will be conducted by the entire team. Initial cases will be used to further develop the analysis methodology, with codes undergoing assessment for applicability and appropriateness. This method will ensure the relevance and maximize the value of the categorizing and coding.
Each case will be reviewed by the team, and differences in codes and events noted. Although this is a qualitative study and we expect to have differences in coding between analysts, we will compare each event and code against the code definition for appropriateness. Incorrectly applied codes will be adjudicated as a group. Broadly, we expect the codes to fall into the following categories: data display visibility issues, navigation problems, data entry problems, content comprehension, attention problems or other cognitive confusion, and issues of health literacy. A qualitative assessment will be completed on the problems users had with cognitive confusion based on the verbal comments made by the test subjects.
Interrater correlation will be evaluated by a two-way, intraclass correlation coefficient . Individual codes will be thematically grouped and evaluated by scenario and subject. Additionally, time-series visualization will be used to compare clustering of all subjects’ aggregated events to identify commonly observed navigation or cognitive problems. Finally, intraclass correlation will be determined between coders for failure/success of each scenario.
Secondary to the evaluation of common problems identified, we will also examine the uptake of learning by comparing the differences in time to complete for scenarios one and two in aggregate, and by comparing the time to a group of familiar, expert users. Additionally, navigation errors, hesitation, and comments or observations of frustration and annoyance will be quantified between the two scenarios and analyzed as a condition of learning.
Post-testing feedback from participants will be used to inform future design decisions and will comments will be assessed along with observations and quantitative metrics to modify navigation flow and user interface/user experience. Demographics, Internet comfort-level and app expertise will be examined for correlation to age and tech literacy of the participants.
We aim to produce a generalizable set of recommendations for the future design of mHealth apps targeted towards mixed age populations of informal caregivers and their aging older adults. Through design and iteration, we hope to form the basis for a framework to support further usability testing of mobile health applications.
The authors would thank Warner Slack, Maxwell Gorenberg,and current and former members of the Division of Clinical Informatics at Beth Israel Deaconess Medical Center.
YQ, DF, JH, AM, contributed to the initial development of the protocol, and al authors were involved in writing the protocol. AA contributed statistical background. CS is the project’s principle investigator and leads the InfoSAGE research effort. All authors read and approved the final manuscript.
This study was made possible through the Agency for Healthcare Research and Quality (AHRQ) grants # R01 HS021495-01A1 and R18 HS24869–01. The authors are solely responsible for this document’s contents, findings, and conclusions.
Ethics approval and consent to participate
The institutional review board of Beth Israel Deaconess Medical Center approved this protocol (2014P000296). Informed consent will be obtained from all study participants using an approved online informed consent process.
Consent for publication
The authors have no competing interests to declare.
- 1.United Nations, Department of Economic and Social Affairs, population division. World population prospects: the 2015 revision, key findings and advance tables. Working paper no. ESA/P/WP.241 29 July 2015, New York. Available at URL: http://www.un.org/en/development/desa/publications/world-population-prospects-2015-revision.html [Last Accessed 9 January 2019].
- 2.US Census. 2017 National Population Projections Tables - Projections for the United States: 2017 to 2060. Available at URL: https://www.census.gov/data/tables/2017/demo/popproj/2017-summary
- 7.Pew Research Center (2014). Older Adults and Technology Use. April 3, 2014. Available at URL: http://www.pewinternet.org/2014/04/03/older-adults-and-technology-use/ [Last Accessed 9 January 2019].
- 8.Pew Research Center. Americans’ Internet Access: 2000–2015. June 26, 2015. Available at URL: http://www.pewinternet.org/2015/06/26/americans-internet-access-2000-2015/ [Last Accessed 9 January 2019].
- 9.Pew Research Center, May 2017, “Tech adoption climbs among older adults”. URL: http://www.pewinternet.org/2017/05/17/tech-adoption-climbs-among-older-adults/
- 11.Arcury TA, Sandberg JC, Melius KP, Quandt SA, Leng X, Latulipe C, Miller DP Jr, Smith DA, Bertoni AG. Older adult internet use and eHealth literacy. J Appl Gerontol. 2018t;24:733464818807468.Google Scholar
- 13.Kushniruk A, Triola MM, Stein B, Borycki EM, Kannry JL. The relationship of usability to medical error: an evaluation of errors associated with usability problems in the use of a handheld application for prescribing medications. InMedinfo. 2004;7:1073–6.Google Scholar
- 17.Lethbridge-Çejku M, Rose D, Vickerie JL. Summary health statistics for the US adults; national health interview survey; 2004.Google Scholar
- 18.Kutner M, Greenburg E, Jin Y, Paulsen C. The health literacy of America's adults: results from the 2003 National Assessment of adult literacy. NCES 2006–483. National Center for Education Statistics. 2006 Sep.Google Scholar
- 20.HP USDHHS. Understanding and improving health. Washington, DC: US Department of Health and Human Services, US Government Printing Office; 2000.Google Scholar
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.