Methods for Evaluating Online Health Information Systems

  • Gary L. Kreps
  • Jordan Alpert
Reference work entry


This chapter will examine the rationale, strategy, and methods for systematically evaluating the effectiveness of online health information systems. Evaluation research will be framed as an essential activity for designing, refining, and sustaining robust health information systems. The best health information system evaluation research programs should include: (1) formative evaluation research activities, such as needs analysis and audience analysis, for designing responsive and appropriate systems; (2) process evaluation research activities to assess how well health information systems work with users, primarily through use of message testing, system usage analysis, and user feedback systems; as well as (3) summative evaluation research activities to assess the influences of the health information systems on important health outcomes, including costs and benefits. We will describe the use of multiple research methods as part of multimethodological designs for conducting health information system evaluation. These methods will include examinations of the applications of content analysis, interviews, focus groups, usability tests, cost-benefit analysis, user feedback systems, unobtrusive measures, the critical incidents method, and field experiments for evaluating health information systems.


Evaluation research Health information systems Formative evaluation Process evaluation Summative evaluation Multimethodological research Audience analysis Needs analysis Cost-benefit analysis 


  1. Albu M, Atack L, Srivastava I. Simulation and gaming to promote health education: results of a usability test. Health Educ J. 2015;74(2):244–54.CrossRefGoogle Scholar
  2. Alpert J, Desens L, Krist A, Kreps GL. Measuring health literacy levels of a patient portal using the CDC’s Clear Communication Index. Health Promot Pract. 2016a. Scholar
  3. Alpert JM, Krist AH, Aycock BA, Kreps GL. Applying multiple methods to comprehensively evaluate a patient portal’s effectiveness to convey information to patients. J Med Internet Res. 2016b;18(5):e112. Scholar
  4. Alpert JM, Krist AH, Aycock BA, Kreps GL. Designing user-centric patient portals: clinician and patients’ uses and gratifications. Telemed e-Health., Ahead of print. 2016c. Scholar
  5. Cho H, Salmon CT. Unintended effects of health communication campaigns. J Commun. 2007;57(2):293–317.CrossRefGoogle Scholar
  6. Chu LF, Chan BK. Evolution of web site design: implications for medical education on the internet. Comput Biol Med. 1998;28(5):459–72.CrossRefGoogle Scholar
  7. Finney Rutten L, Hesse B, Moser R, Kreps GL, editors. Building the evidence base in cancer communication. Cresskill: Hampton Press; 2011.Google Scholar
  8. Fonteyn ME, Kuipers B, Grobe SJ. A description of think aloud method and protocol analysis. Qual Health Res. 1993;3(4):430–41.CrossRefGoogle Scholar
  9. Green LW, Glasgow RE. Evaluating the relevance, generalization, and applicability of research: issues in external validation and translation methodology. Eval Health Prof. 2006;29(1):126–53.CrossRefGoogle Scholar
  10. Greenwood S, Perrin A, Duggan M (2016) Social media update 2016. Retrieved 16 March 2017, from
  11. Hesse BW, Nelson DE, Kreps GL, Croyle RT, Arora NK, Rimer BK, Viswanath K. Trust and sources of health information. The impact of the internet and its implications for health care providers: findings from the first Health Information National Trends Survey. Arch Intern Med. 2005;165(22):2618–24.CrossRefGoogle Scholar
  12. Hornik R, Jacobsohm L, Orwin R, Piesse A, Klton G. Effects of the National Youth Anti-Drug media campaign on youths. Am J Public Health. 2008;98(12):2229–36.CrossRefGoogle Scholar
  13. James KJ, Albrecht JA, Litchfield RE, Weishaar CA. A summative evaluation of a food safety social marketing campaign “4-day throw-away” using traditional and social media. J Food Sci Educ. 2013;12(3):48–55.CrossRefGoogle Scholar
  14. Kreps GL. Evaluating new health information technologies: expanding the frontiers of health care delivery and health promotion. Stud Health Technol Inform. 2002;80:205–12.Google Scholar
  15. Kreps GL. The information revolution and the changing face of health communication in modern society. J Health Psychol. 2011a;16:192–3.CrossRefGoogle Scholar
  16. Kreps GL. Methodological diversity and integration in health communication inquiry. Patient Educ Couns. 2011b;82:285–91.CrossRefGoogle Scholar
  17. Kreps GL. Consumer control over and access to health information. Ann Fam Med. 2012;10(5). Available at:
  18. Kreps GL. Evaluating health communication programs to enhance health care and health promotion. J Health Commun. 2014a;19(12):1449–59.CrossRefGoogle Scholar
  19. Kreps GL. Achieving the promise of digital health information systems. J Public Health Res. 2014b;3(471):128–9.Google Scholar
  20. Kreps GLR. Epilogue: lessons learned about evaluating health communication programs. J Health Commun. 2014c;19(12):1510–4.CrossRefGoogle Scholar
  21. Kreps GL. Communication technology and health: the advent of ehealth applications. In: Cantoni L, Danowski JA, editors. Communication and technology, Volume 5 of the Handbooks of communication science, p. 483–493, (Schulz PJ, Cobley P, General Editors). Berlin: De Gruyter Mouton Publications; 2015.Google Scholar
  22. Kreps GL, Neuhauser L. New directions in ehealth communication: opportunities and challenges. Patient Educ Couns. 2010;78:329–36.CrossRefGoogle Scholar
  23. Kreps GL, Neuhauser L. Artificial intelligence and immediacy: designing health communication to personally engage consumers and providers. Patient Educ Couns. 2013;92:205–10.CrossRefGoogle Scholar
  24. Krist AH, Nease DE, Kreps GL, Overholser L, McKenzie M. Engaging patients in primary and specialty care. In: Hesse BW, Ahern DK, Beckjord E, editors. Oncology informatics: using health information technology to improve processes and outcomes in cancer care. Amsterdam: Elsevier; 2016. p. 55–79.CrossRefGoogle Scholar
  25. Levine D, Madsen A, Wright E, Barar RE, Santelli J, Bull S. Formative research on MySpace: online methods to engage hard-to-reach populations. J Health Commun. 2011;16(4):448–54.CrossRefGoogle Scholar
  26. Moore G, Audrey S, Barker M, Bond L, Bonell C, Cooper C, Hardeman W, Moore L, O’Cathain A, Tinati T, Wight D, Baird J. Process evaluation in complex public health intervention studies: the need for guidance. J Epidemiol Community Health. 2014;68:101–2.CrossRefGoogle Scholar
  27. Murdough C. Social media measurement: It’s not impossible. J Interact Advert. 2009;10(1):94–9.CrossRefGoogle Scholar
  28. Neiger BL, Thackeray R, Van Wagenen SA, Hanson CL, West JH, Barnes MD, Fagen MC. Use of social media in health promotion purposes, key performance indicators, and evaluation metrics. Health Promot Pract. 2012;13(2):159–64.CrossRefGoogle Scholar
  29. Neuhauser L. Participatory design for better interactive health communication: a statewide model in the USA. Electron J Commun. 2001;11(3 and 4):43.Google Scholar
  30. Neuhauser L, Kreps G. Rethinking communication in the e-health era. J Health Psychol. 2003;8: 7–22.CrossRefGoogle Scholar
  31. Neuhauser L, Kreps GL. Online cancer communication interventions: meeting the literacy, linguistic, and cultural needs of diverse audiences. Patient Educ Couns. 2008;71(3):365–77.CrossRefGoogle Scholar
  32. Neuhauser L, Kreps G. Ehealth communication and behavior change: promise and performance. J Soc Semiot. 2010;20(1):9–27.CrossRefGoogle Scholar
  33. Neuhauser L, Kreps GL. Participatory design and artificial intelligence: strategies to improve health communication for diverse audiences. In: Green N, Rubinelli S, Scott D, editors. Artificial intelligence and health communication. Cambridge, MA: AAAI Press; 2011. p. 49–52.Google Scholar
  34. Neuhauser L, Kreps GL. Integrating design science theory and methods to improve the development and evaluation of health communication programs. J Health Commun. 2014;19(12): 1460–71.CrossRefGoogle Scholar
  35. Neuhauser L, Paul K. Readability, comprehension and usability. In: Communicating risks and benefits: an evidence-based user’s guide. Silver Spring/Bethesda: U.S. Department of Health and Human Services/Food and Drug Administration; 2011.Google Scholar
  36. Neuhauser L, Schwab M, Obarski SK, Syme SL, Bieber M. Community participation in health promotion: evaluation of the California wellness guide. Health Promot Int. 1998;13(3):211.CrossRefGoogle Scholar
  37. Neuhauser L, Constantine WL, Constantine NA, Sokal-Gutierrez K, Obarski SK, Clayton L, Desai M, Sumner G, Syme SL. Promoting prenatal and early childhood health: evaluation of a statewide materials-based intervention for parents. Am J Public Health. 2007;97(10):813–9.CrossRefGoogle Scholar
  38. Nguyen P, Gold J, Pedrana A, Chang S, Howard S, Ilic O, et al. Sexual health promotion on social networking sites: a process evaluation of the FaceSpace project. J Adolesc Health. 2013;53(1):98–104.CrossRefGoogle Scholar
  39. Nielsen J. Usability engineering: Amsterdam: Elsevier; 1994.Google Scholar
  40. Nielsen J. Designing web usability: the practice of simplicity. Indianapolis: New Riders Publishing; 1999.Google Scholar
  41. Nutbeam D. Evaluating health promotion – progress, problems, and solutions. Health Promot Int. 1998;13:27–44.CrossRefGoogle Scholar
  42. Rinegold DJ. Boomerang effects in response to public health interventions: some unintended consequences in the alcoholic beverage market. J Consum Policy. 2002;25:27–63.CrossRefGoogle Scholar
  43. Rootman I, Goodstadt M, McQueen D, Potvin L, Springett J, Ziglio E, editors. Evaluation in health promotion: principles and perspectives. Copenhagen: WHO; 2001.Google Scholar
  44. Rus HM, Cameron LD. Health communication in social media: message features predicting user engagement on diabetes-related Facebook pages. Ann Behav Med. 2016;50(5):678–89.CrossRefGoogle Scholar
  45. Sterne J. Social media metrics: how to measure and optimize your marketing investment. Hoboken: Wiley; 2010.Google Scholar
  46. Thackeray R, Neiger BL, Smith AK, Van Wagenen SB. Adoption and use of social media among public health departments. BMC Public Health. 2012;12(1):242.CrossRefGoogle Scholar
  47. van Wijngaarden JDH, Scholten GRM, van Wijk KP. Strategic analysis for health care organizations: the suitability of the SWOT-analysis. International journal of health planning and management. 2010. Available at:
  48. Webb EJ, Campbell DT, Schwartz RD, Sechrist L. Unobtrusive measures: nonreactive research in the social sciences. New York: Rand McNally & Company; 1972.Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  1. 1.Department of CommunicationGeorge Mason UniversityFairfaxUSA
  2. 2.Department of AdvertisingUniversity of FloridaGainesvilleUSA

Personalised recommendations