• Gondy Leroy
Part of the Health Informatics book series (HI)


This chapter provides an overview of different types of evaluation studies and their advantages and disadvantages. The studies that are reviewed range from naturalistic observations to controlled experiments. They differ in the amount of intervention by researchers and the degree to which the environment is controlled. For each, examples from the literature are included as illustrations.

Since this book focuses on the controlled experiment, the main section of the chapter provides an overview of the essential elements in such controlled experiments: the research hypotheses, the different types of variables that need to be defined, the sampling units which can be people or artifacts in informatics, random assignment and, finally, statistical analyses that allow generalization of a conclusion from a sample to a population. Although the examples in this and subsequent ­chapters are taken from the medical field, the study design methods are generic. As such, the content is applicable to the evaluation of information systems in other domains.


Correlation Study Coronary Angioplasty User Study Technology Acceptance Model Intended User 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Murphy TE (2009) Requirements form the foundation of software quality. Research report, G00165755Google Scholar
  2. 2.
    Gonzales CK (2010) Eliciting user requirements using appreciative inquiry. Ph.D., Claremont Graduate UniversityGoogle Scholar
  3. 3.
    Baroudi JJ, Olson MH, Ives B (1986) An empirical study of the impact of user involvement on system usage and information satisfaction. Commun ACM 29(3):232–238CrossRefGoogle Scholar
  4. 4.
    Schneider GM, Martin J, Tsai WT (1992) An experimental study of fault detection in user requirements documents. ACM Trans Software Eng Methodol (TOSEM) 1(2):188–204. doi: 10.1145/128894.128897 CrossRefGoogle Scholar
  5. 5.
    Westland JC (2002) The Cost of Errors in Software Development: Evidence from Industry. The Hournal of Systems and Software, 62, 1–9CrossRefGoogle Scholar
  6. 6.
    Friedman CP, Wyatt JC (2000) Evaluation methods in medical informatics. Springer-Verlag, New YorkGoogle Scholar
  7. 7.
    Brender J (2006) Handbook of evaluation methods for health informatics (trans: Carlander L). Elsevier Inc, San DiegoGoogle Scholar
  8. 8.
    Kaplan B (2001) Evaluating informatics applications — clinical decision support systems literature review. Int J Med Inform 64:15–37PubMedCrossRefGoogle Scholar
  9. 9.
    Tesh EM, Holditch-Davis D (1997) HOME inventory and NCATS: relation to mother and child behaviors during naturalistic observations. Home observation for measurement of the environment. Nursing Child Assessment Teaching Scale. Res Nurs Health 20(4):295–307PubMedCrossRefGoogle Scholar
  10. 10.
    Lambrechts G, Noortgate WVD, Eeman L, Maes B (2010) Staff reactions to challenging behaviour: an observation study. Res Dev Disabil 31:525–535PubMedCrossRefGoogle Scholar
  11. 11.
    Campos B, Graesch AP, Repetti R, Bradbury T, Ochs E (2009) Opportunity for interaction? A naturalistic observation study of dual-earner families after work and school. J Fam Psychol 23(6):798–807PubMedCrossRefGoogle Scholar
  12. 12.
    Kaplan B (2001) Evaluating informatics applications – some alternative approaches: theory, social interactionism, and call for methodological pluralism. Int J Med Inform 64:39–56PubMedCrossRefGoogle Scholar
  13. 13.
    Rosson MB, Carroll JM (2002) Usability engineering: scenario-based development of human-computer interaction. Interactive technologies. Morgan Kaufman Publishers, San FranciscoGoogle Scholar
  14. 14.
    Purkis ME (1999) Embracing technology: an exploration of the effects of writing nursing. Nurs Inq 6(3):147–156PubMedCrossRefGoogle Scholar
  15. 15.
    Stringer ET (1999) Action research. SAGE Publications Inc, Thousand OaksGoogle Scholar
  16. 16.
    Boursnell M, Prosser S (2010) Increasing identification of domestic violence in emergency departments: a collaborative contribution to increasing the quality of practice of emergency nurses. Contemp Nurse 35(12):7Google Scholar
  17. 17.
    Borg M, Karlsson B, Kim HS (2010) Double helix of research and practice - developing a practice model for crisis resolution and home treatment through participatory action research. Int J Qualitative Stud Health Well-being 5. doi: 10.3402/qhw.v5i1.4647Google Scholar
  18. 18.
    Butera-Prinzi F, Charles N, Heine K, Rutherford B, Lattin D (2010) Family-to-family link up program: a community-based initiative supporting families caring for someone with an acquired brain injury. NeuroRehabilitation 27:31–47PubMedGoogle Scholar
  19. 19.
    Davison R, Vogel D (2007) Group support systems in Hong Kong: an action research project. In: Galliers RD, Markus ML, Newell S (eds) Exploring information systems research approaches: readings and reflections. Routledge, New York, pp 33–46Google Scholar
  20. 20.
    Delobelle P, Onya H, Langa C, Mashamba J, Depoorter AM (2010) Advances in health promotion in Africa: promoting health through hospitals. Global Health Promot 17:33–36. doi: 10.1177/1757975910363929 CrossRefGoogle Scholar
  21. 21.
    Rust J, Golombok S (2009) Modern psychometrics: the science of psychological assessment, 3rd edn. Routledge, LondonGoogle Scholar
  22. 22.
    Magnusson D (1966) Test theory. Addison-Wesley Publishing Company, ReadingGoogle Scholar
  23. 23.
    Drenth PJD, Sijtsma K (1990) Testtheorie: Inleiding in the Theorie van de Psychologische Test en zijn Toepassingen. Bohn Stafleu Van Loghum, Houten/AntwerpenGoogle Scholar
  24. 24.
    de Ayala RJ (2009) The theory and practice of item response theory. Methodology in the social sciences. The Guilford Press, New YorkGoogle Scholar
  25. 25.
    Rushton P, Miller W, Lee KR, Eng J, Yip J (2011) Development and Content Validation of the Wheelchair Use Confidence Scale: A Mixed-Methods Study. Disability and Rehabilitation Assistive Technology, 6 (1), 57–66PubMedCrossRefGoogle Scholar
  26. 26.
    Fischer B, Nakamura N, Ialomiteanu A, Boak A, Rehm J (2010) Assessing the prevalence of nonmedical prescription opioid use in the general Canadian population: methodological issues and questions. La Revue Canadienne De Psychiatrie 55(9):606–609Google Scholar
  27. 27.
    Jaussent S, Labarère J, Boyer J, François P (2004) Psychometric characteristics of questionnaires designed to assess the knowledge, perceptions and practices of health care professionals with regards to alcoholic patients [translation: article in French]. Encephale 30(5):437–446PubMedCrossRefGoogle Scholar
  28. 28.
    Palmirotta R, Savonarola A, Ludovici G, Donati P, Cavaliere F, De Marchis M, Ferroni P, Guadagni F (2010) Association between Birt Hogg Dube syndrome and cancer predisposition. Anticancer Res 30(3):751–757PubMedGoogle Scholar
  29. 29.
    Ajzen I (1991) The theory of planned behavior. Organ Behav Hum Decis Process 50:179–211CrossRefGoogle Scholar
  30. 30.
    Davis FD (1989) Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q 13(3):319–339CrossRefGoogle Scholar
  31. 31.
    Hu PJ-H, Sheng ORL, Chau PY, Tam K-Y, Fung H (1999) Investigation physician acceptance of telemedicine technology: a survey study in Hong Kong. In: 32nd Hawaii International Conference on System Sciences, 1999Google Scholar
  32. 32.
    Hu PJ-H, Chau PYK, Sheng ORL (2000) Investigation of factors affecting healthcare organization’s adoption of telemedicine technology. In: 33rd Hawaii international conference on system sciences, 2000Google Scholar
  33. 33.
    Rawstorne P, Jayasuriya R, Caputi P (2000) Issues in predicting and explaining usage behaviors with the technology acceptance model and the theory of planned behavior when usage is mandatory. In: International conference on information systems, Brisbane, Australia, 2000, pp 35–44Google Scholar
  34. 34.
    Honeth L, Bexelius C, Eriksson M, Sandin S, Litton J, Rosenhall U, Nyrén O, Bagger-Sjöbäck D (2010) An Internet-based hearing test for simple audiometry in nonclinical settings: preliminary validation and proof of principle. Otol Neurotol 31(5):708–714PubMedCrossRefGoogle Scholar
  35. 35.
    Ives B, Olson MH, Baroudi JJ (1983) The measurement of user information satisfaction. Commun ACM 26(10):785–793CrossRefGoogle Scholar
  36. 36.
    Chinman M, Young AS, Schell T, Hassell J, Mintz J (2004) Computer-assisted self-assessment in persons with severe mental illness. J Clin Psychiatry 65(10):1343–1351PubMedCrossRefGoogle Scholar
  37. 37.
    Sunaert P, Bastiaens H, Nobels F, Feyen L, Verbeke G, Vermeire E, Maeseneer JD, Willems S, Sutter AD (2010) Effectiveness of the introduction of a chronic care model-based ­program for type 2 diabetes in Belgium. BMC Health Serv Res. doi: 10.1186/1472-6963-10-207 PubMedGoogle Scholar
  38. 38.
    Sunaert P, Bastiaens H, Feyen L, Snauwaert B, Nobels F, Wens J, Vermeire E, Royen PV, Maeseneer JD, Sutter AD, Willems S (2009) Implementation of a program for type 2 diabetes based on the chronic care model in a hospital-centered health care system: “the Belgian experience”. BMC Health Services Research 9 (152). doi:10.1186/1472-6963-9-152Google Scholar
  39. 39.
    Herz EJ, Goldberg WA, Reis JS (1984) Family life education for young adolescents: a quasi-experiment. J Youth Adolesc 13(4):309–327PubMedCrossRefGoogle Scholar
  40. 40.
    Glynn S, Randolph E, Garrick T, Lui A (2010) A proof of concept trial of an online psychoeducational program for relatives of both veterans and civilians living with schizophrenia. Psychiatr Rehabil J 33(4):278–287PubMedCrossRefGoogle Scholar
  41. 41.
    Kröncke K (2010) Computer-based learning versus practical course in pre-clinical education: acceptance and knowledge retention. Med Teach 32(5):408–413PubMedCrossRefGoogle Scholar
  42. 42.
    Kirk RE (1995) Experimental design: procedures for the behavioral sciences, 3rd edn. Brooks/Cole Publishing Company, Pacific GroveGoogle Scholar
  43. 43.
    Timm NH (2002) Applied multivariate analysis. Springer, New YorkGoogle Scholar
  44. 44.
    Everitt BS, Pickles A (2004) Statistical aspects of the design and analysis of clinical trials, Revisedth edn. Imperial College Press, LondonCrossRefGoogle Scholar
  45. 45.
    Ammenwertha E, Gräber S, Herrmann G, Bürkle T, König J (2003) Evaluation of health information systems—problems and challenges. Int J Med Inform 71:125–135CrossRefGoogle Scholar
  46. 46.
    Anderson JR, Schonfeld TL (2009) Data-sharing dilemmas: allowing pharmaceutical company access to research data. IRB: Ethics Hum Res 31(3):17–19Google Scholar

Copyright information

© Springer-Verlag London Limited 2011

Authors and Affiliations

  1. 1.School of Information Systems and TechnologyClaremont Graduate UniversityClaremontUSA

Personalised recommendations