Skip to main content
SpringerLink
Log in

Methods for Evaluating Online Health Information Systems

  • Reference work entry
  • First Online:
Handbook of Research Methods in Health Social Sciences

Abstract

This chapter will examine the rationale, strategy, and methods for systematically evaluating the effectiveness of online health information systems. Evaluation research will be framed as an essential activity for designing, refining, and sustaining robust health information systems. The best health information system evaluation research programs should include: (1) formative evaluation research activities, such as needs analysis and audience analysis, for designing responsive and appropriate systems; (2) process evaluation research activities to assess how well health information systems work with users, primarily through use of message testing, system usage analysis, and user feedback systems; as well as (3) summative evaluation research activities to assess the influences of the health information systems on important health outcomes, including costs and benefits. We will describe the use of multiple research methods as part of multimethodological designs for conducting health information system evaluation. These methods will include examinations of the applications of content analysis, interviews, focus groups, usability tests, cost-benefit analysis, user feedback systems, unobtrusive measures, the critical incidents method, and field experiments for evaluating health information systems.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 649.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 849.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Albu M, Atack L, Srivastava I. Simulation and gaming to promote health education: results of a usability test. Health Educ J. 2015;74(2):244–54.

    Article  Google Scholar 

  • Alpert J, Desens L, Krist A, Kreps GL. Measuring health literacy levels of a patient portal using the CDC’s Clear Communication Index. Health Promot Pract. 2016a. https://doi.org/10.1177/1524839916643703.

    Article  Google Scholar 

  • Alpert JM, Krist AH, Aycock BA, Kreps GL. Applying multiple methods to comprehensively evaluate a patient portal’s effectiveness to convey information to patients. J Med Internet Res. 2016b;18(5):e112. https://doi.org/10.2196/jmir.5451.

    Article  Google Scholar 

  • Alpert JM, Krist AH, Aycock BA, Kreps GL. Designing user-centric patient portals: clinician and patients’ uses and gratifications. Telemed e-Health., Ahead of print. 2016c. https://doi.org/10.1089/tmj.2016.0096.

    Article  Google Scholar 

  • Cho H, Salmon CT. Unintended effects of health communication campaigns. J Commun. 2007;57(2):293–317.

    Article  Google Scholar 

  • Chu LF, Chan BK. Evolution of web site design: implications for medical education on the internet. Comput Biol Med. 1998;28(5):459–72.

    Article  Google Scholar 

  • Finney Rutten L, Hesse B, Moser R, Kreps GL, editors. Building the evidence base in cancer communication. Cresskill: Hampton Press; 2011.

    Google Scholar 

  • Fonteyn ME, Kuipers B, Grobe SJ. A description of think aloud method and protocol analysis. Qual Health Res. 1993;3(4):430–41.

    Article  Google Scholar 

  • Green LW, Glasgow RE. Evaluating the relevance, generalization, and applicability of research: issues in external validation and translation methodology. Eval Health Prof. 2006;29(1):126–53.

    Article  Google Scholar 

  • Greenwood S, Perrin A, Duggan M (2016) Social media update 2016. Retrieved 16 March 2017, from http://www.pewinternet.org/2016/11/11/social-media-update-2016/

  • Hesse BW, Nelson DE, Kreps GL, Croyle RT, Arora NK, Rimer BK, Viswanath K. Trust and sources of health information. The impact of the internet and its implications for health care providers: findings from the first Health Information National Trends Survey. Arch Intern Med. 2005;165(22):2618–24.

    Article  Google Scholar 

  • Hornik R, Jacobsohm L, Orwin R, Piesse A, Klton G. Effects of the National Youth Anti-Drug media campaign on youths. Am J Public Health. 2008;98(12):2229–36.

    Article  Google Scholar 

  • James KJ, Albrecht JA, Litchfield RE, Weishaar CA. A summative evaluation of a food safety social marketing campaign “4-day throw-away” using traditional and social media. J Food Sci Educ. 2013;12(3):48–55.

    Article  Google Scholar 

  • Kreps GL. Evaluating new health information technologies: expanding the frontiers of health care delivery and health promotion. Stud Health Technol Inform. 2002;80:205–12.

    Google Scholar 

  • Kreps GL. The information revolution and the changing face of health communication in modern society. J Health Psychol. 2011a;16:192–3.

    Article  Google Scholar 

  • Kreps GL. Methodological diversity and integration in health communication inquiry. Patient Educ Couns. 2011b;82:285–91.

    Article  Google Scholar 

  • Kreps GL. Consumer control over and access to health information. Ann Fam Med. 2012;10(5). Available at: http://www.annfammed.org/content/10/5/428.full/reply#annalsfm_el_25148

  • Kreps GL. Evaluating health communication programs to enhance health care and health promotion. J Health Commun. 2014a;19(12):1449–59.

    Article  Google Scholar 

  • Kreps GL. Achieving the promise of digital health information systems. J Public Health Res. 2014b;3(471):128–9.

    Google Scholar 

  • Kreps GLR. Epilogue: lessons learned about evaluating health communication programs. J Health Commun. 2014c;19(12):1510–4.

    Article  Google Scholar 

  • Kreps GL. Communication technology and health: the advent of ehealth applications. In: Cantoni L, Danowski JA, editors. Communication and technology, Volume 5 of the Handbooks of communication science, p. 483–493, (Schulz PJ, Cobley P, General Editors). Berlin: De Gruyter Mouton Publications; 2015.

    Google Scholar 

  • Kreps GL, Neuhauser L. New directions in ehealth communication: opportunities and challenges. Patient Educ Couns. 2010;78:329–36.

    Article  Google Scholar 

  • Kreps GL, Neuhauser L. Artificial intelligence and immediacy: designing health communication to personally engage consumers and providers. Patient Educ Couns. 2013;92:205–10.

    Article  Google Scholar 

  • Krist AH, Nease DE, Kreps GL, Overholser L, McKenzie M. Engaging patients in primary and specialty care. In: Hesse BW, Ahern DK, Beckjord E, editors. Oncology informatics: using health information technology to improve processes and outcomes in cancer care. Amsterdam: Elsevier; 2016. p. 55–79.

    Chapter  Google Scholar 

  • Levine D, Madsen A, Wright E, Barar RE, Santelli J, Bull S. Formative research on MySpace: online methods to engage hard-to-reach populations. J Health Commun. 2011;16(4):448–54.

    Article  Google Scholar 

  • Moore G, Audrey S, Barker M, Bond L, Bonell C, Cooper C, Hardeman W, Moore L, O’Cathain A, Tinati T, Wight D, Baird J. Process evaluation in complex public health intervention studies: the need for guidance. J Epidemiol Community Health. 2014;68:101–2.

    Article  Google Scholar 

  • Murdough C. Social media measurement: It’s not impossible. J Interact Advert. 2009;10(1):94–9.

    Article  Google Scholar 

  • Neiger BL, Thackeray R, Van Wagenen SA, Hanson CL, West JH, Barnes MD, Fagen MC. Use of social media in health promotion purposes, key performance indicators, and evaluation metrics. Health Promot Pract. 2012;13(2):159–64.

    Article  Google Scholar 

  • Neuhauser L. Participatory design for better interactive health communication: a statewide model in the USA. Electron J Commun. 2001;11(3 and 4):43.

    Google Scholar 

  • Neuhauser L, Kreps G. Rethinking communication in the e-health era. J Health Psychol. 2003;8: 7–22.

    Article  Google Scholar 

  • Neuhauser L, Kreps GL. Online cancer communication interventions: meeting the literacy, linguistic, and cultural needs of diverse audiences. Patient Educ Couns. 2008;71(3):365–77.

    Article  Google Scholar 

  • Neuhauser L, Kreps G. Ehealth communication and behavior change: promise and performance. J Soc Semiot. 2010;20(1):9–27.

    Article  Google Scholar 

  • Neuhauser L, Kreps GL. Participatory design and artificial intelligence: strategies to improve health communication for diverse audiences. In: Green N, Rubinelli S, Scott D, editors. Artificial intelligence and health communication. Cambridge, MA: AAAI Press; 2011. p. 49–52.

    Google Scholar 

  • Neuhauser L, Kreps GL. Integrating design science theory and methods to improve the development and evaluation of health communication programs. J Health Commun. 2014;19(12): 1460–71.

    Article  Google Scholar 

  • Neuhauser L, Paul K. Readability, comprehension and usability. In: Communicating risks and benefits: an evidence-based user’s guide. Silver Spring/Bethesda: U.S. Department of Health and Human Services/Food and Drug Administration; 2011.

    Google Scholar 

  • Neuhauser L, Schwab M, Obarski SK, Syme SL, Bieber M. Community participation in health promotion: evaluation of the California wellness guide. Health Promot Int. 1998;13(3):211.

    Article  Google Scholar 

  • Neuhauser L, Constantine WL, Constantine NA, Sokal-Gutierrez K, Obarski SK, Clayton L, Desai M, Sumner G, Syme SL. Promoting prenatal and early childhood health: evaluation of a statewide materials-based intervention for parents. Am J Public Health. 2007;97(10):813–9.

    Article  Google Scholar 

  • Nguyen P, Gold J, Pedrana A, Chang S, Howard S, Ilic O, et al. Sexual health promotion on social networking sites: a process evaluation of the FaceSpace project. J Adolesc Health. 2013;53(1):98–104.

    Article  Google Scholar 

  • Nielsen J. Usability engineering: Amsterdam: Elsevier; 1994.

    Google Scholar 

  • Nielsen J. Designing web usability: the practice of simplicity. Indianapolis: New Riders Publishing; 1999.

    Google Scholar 

  • Nutbeam D. Evaluating health promotion – progress, problems, and solutions. Health Promot Int. 1998;13:27–44.

    Article  Google Scholar 

  • Rinegold DJ. Boomerang effects in response to public health interventions: some unintended consequences in the alcoholic beverage market. J Consum Policy. 2002;25:27–63.

    Article  Google Scholar 

  • Rootman I, Goodstadt M, McQueen D, Potvin L, Springett J, Ziglio E, editors. Evaluation in health promotion: principles and perspectives. Copenhagen: WHO; 2001.

    Google Scholar 

  • Rus HM, Cameron LD. Health communication in social media: message features predicting user engagement on diabetes-related Facebook pages. Ann Behav Med. 2016;50(5):678–89.

    Article  Google Scholar 

  • Sterne J. Social media metrics: how to measure and optimize your marketing investment. Hoboken: Wiley; 2010.

    Google Scholar 

  • Thackeray R, Neiger BL, Smith AK, Van Wagenen SB. Adoption and use of social media among public health departments. BMC Public Health. 2012;12(1):242.

    Article  Google Scholar 

  • van Wijngaarden JDH, Scholten GRM, van Wijk KP. Strategic analysis for health care organizations: the suitability of the SWOT-analysis. International journal of health planning and management. 2010. Available at: https://www.researchgate.net/profile/Jeroen_Wijngaarden/publication/45094861_Strategic_analysis_for_health_care_organizations_the_suitability_of_the_SWOT-analysis/links/541fc9860cf203f155c25f28.pdf

  • Webb EJ, Campbell DT, Schwartz RD, Sechrist L. Unobtrusive measures: nonreactive research in the social sciences. New York: Rand McNally & Company; 1972.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Communication, George Mason University, Fairfax, VA, USA

    Gary L. Kreps

  2. Department of Advertising, University of Florida, Gainesville, FL, USA

    Jordan Alpert

Authors
  1. Gary L. Kreps
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jordan Alpert
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gary L. Kreps .

Editor information

Editors and Affiliations

  1. School of Science and Health, Western Sydney University, Penrith, NSW, Australia

    Pranee Liamputtong

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Singapore Pte Ltd.

About this entry

Check for updates. Verify currency and authenticity via CrossMark

Cite this entry

Kreps, G.L., Alpert, J. (2019). Methods for Evaluating Online Health Information Systems. In: Liamputtong, P. (eds) Handbook of Research Methods in Health Social Sciences. Springer, Singapore. https://doi.org/10.1007/978-981-10-5251-4_111

Download citation

Publish with us

Policies and ethics

Search

Navigation