Skip to main content
SpringerLink
Log in

Ecological Validity for Patient Reported Outcomes

  • Chapter
  • First Online:
Handbook of Behavioral Medicine

Abstract

Self-reports often serve as outcomes in all manner of behavioral science and medical research; as such, it is essential that they be measured as accurately as possible. In this article, we discuss the concept of ecological validity as it relates to methodologies used to collect self-report data. When phenomena are responsive to the environment and memory of them is questionable, we argue that ecological validity is necessary to adequately construct self-report measures and that global and retrospective assessment techniques are often not up to the task. Repeated momentary data collection incorporating careful sampling protocols can be achieved with electronically based data capture systems. These technologies are described and strengths and limitations are reviewed. It is now possible to collect self-report data to be truly representative of the participants’ everyday environments.

AAS is the associate chair of the Scientific Advisory Board of invivodata, inc., a company that supplies electronic data capture services for clinical research and is a senior scientist at the Gallup Organization. SS is a founder of invivodata, inc. and the chair of its Scientific Advisory Board.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 429.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 549.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 799.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    In fact, retrieval is not a simple process in that what is retrieved may be influenced by the individual’s psychological state at the time of retrieval. For example, unpleasant memories are more accessible when an individual is in a negative affective state than when in positive affective state (Kihlstrom, et al 2000).

  2. 2.

    A respondent’s affective or pain state at the time of retrospection also influences the accessibility of certain memories and the heuristic processes used to summarize retrieved memories.

References

  • Affleck, G., Tennen, H., Keefe, F. J., Lefebvre, J. C., Kashikar-Zuck, S. et al (1999). Everyday life with osteoarthritis or rheumatoid arthritis: independent effects of disease and gender on daily pain, mood and coping. Pain, 83, 601–609.

    Article  PubMed  CAS  Google Scholar 

  • Blanton, H., and Jaccard, J. (2006). Arbitrary metric in psychology. Am Psychol, 61, 27–41.

    Article  PubMed  Google Scholar 

  • Bolger, N., Davis, A., and Rafaeli, E. (2003). Dairy methods: capturing life as it is lived. Ann Rev Psychol, 54, 579–616.

    Article  Google Scholar 

  • Bradburn, N., Rips, L., and Shevell, S. (1987). Answering autobiographical questions: the impact of memory and inference on surveys. Science, 236, 151–167.

    Article  Google Scholar 

  • Broderick, J., Schwartz, J., and Stone, A. (2006, 3–6 May). Context (pain and affect) influences recall pain ratings [Poster presented at the Annual Meeting of the American Pain Society]. San Antonio, TX.

    Google Scholar 

  • Broderick, J., Schwartz, J., Vikingstad, G., Pribbernow, M., Grossman, S., and Stone, A. (2008). The accuracy of pain and fatigue items across different reporting periods. Pain, 139, 146–157.

    Article  PubMed  Google Scholar 

  • Brunswik, E. (1944). Distal focussing of perception: size constancy in a representative sample of situations. Psychol Monogr, 56, 1–49.

    Article  Google Scholar 

  • Brunswik, E. (1949). Systematic and Representative Design of Psychological Experiments. Berkeley and Los Angeles: University of California Press.

    Google Scholar 

  • Busey, T., Tunnicliff, J., Loftus, G., and Loftus, E. (2000). Accounts of the confidence-accuracy relation in recognition memory. Psychon Bull Rev, 7, 26–48.

    Article  PubMed  CAS  Google Scholar 

  • Byerly, M., Fisher, R., Whatley, K., Holland, R., Varghese, F. et al (2005). A comparison of electronic monitoring vs clinician rating of antipsychotic adherence in outpatients with schizophrenia. Psychiat Res, 133, 129–133.

    Article  Google Scholar 

  • Clark, D., and Teasdale, J. (1982). Diurnal variation in clinical depression and accessibility of memories of positive and negative experiences. J Abnorm Psychol, 91, 87–95.

    Article  PubMed  CAS  Google Scholar 

  • Cruise, C., Porter, L., Broderick, J., Kaell, A., and Stone, A. (1996). Reactive effects of diary self-assessment in chronic pain patients. Pain, 67, 253–258.

    Article  PubMed  CAS  Google Scholar 

  • Csikszentmihalyi, M., and Larsen, R. E. (1987). Validity and reliability of the experience sampling method. J Nerv Med Dis, 175, 526–536.

    Article  CAS  Google Scholar 

  • Delespaul, P. (1995). Assessing Schizophrenia in Daily Life -- The Experience Sampling Method. Maastricht: Maastricht University Press.

    Google Scholar 

  • DeVries, M. (1987). Investigating mental disorders in their natural settings: introduction to the special issue. J Nerv Men Dis, 175, 509–513.

    Article  CAS  Google Scholar 

  • Eich, E., Reeves, J., Jaeger, B., and Graff-Radford, S. (1985). Memory for pain: relation between past and present pain intensity. Pain, 223, 375–379.

    Article  Google Scholar 

  • Fredrickson, B. (2000). Extracting meaning from past affective experiences: the importance of peaks, ends, and specific emotions. Cogn Emot, 14, 577–606.

    Article  Google Scholar 

  • Freedman, M., Lester, K., McNamara, C., Milby, J., and Schumacher, J. (2006). Cell phones for Ecological Momentary Assessment with cocaine-addicted homeless patients in treatment. J Subst Abuse Treat, 30, 105–111.

    Article  PubMed  Google Scholar 

  • Green, A., Rafaeli, E., Bolger, N., Shrout, P., and Reis, H. (2006). Paper or plastic? Data equivalence in paper and electronic diaries. Psychol Methods, 11, 87–105.

    Article  PubMed  Google Scholar 

  • Gwaltney, C., Shields, A., and Shiffman, S. (2008). Equivalence of electronic and paper-and-pencil administration of patient reported outcome measures. Val Health , 11, 322–333.

    Article  Google Scholar 

  • Hammond, K., and Stewart, T. (2001). The Essential Brunswik: Beginnings, Explications, Applications. New York, NY: Oxford University Press.

    Google Scholar 

  • Hufford, M. (2007). Special methodological challenges and opportunities in Ecological Momentary Assessment. In A. Stone, S. Shiffman, A. Atienza, & L. Nebling (Eds.), The Science of Real-Time Data Capture: Self-Reports in Health Research (pp. 54–75). New York, NY: Oxford University Press.

    Google Scholar 

  • Hufford, M., and Shields, A. (2002). Electronic diaries: an examination of applications and what works in the field. Appl Clin Trials, 11, 46–56.

    Google Scholar 

  • Hufford, M., Shields, A., Shiffman, S., Paty, J., and Balabanis, M. (2002). Reactivity to ecological momentary assessment: an example using undergraduate problem drinkers. Psychol Addict Behav, 16, 205–211.

    Article  PubMed  Google Scholar 

  • Kahneman, D., Diener, E., and Schwarz, N. (1999). Well-Being: The Foundations of Hedonic Psychology. New York: Russell Sage Foundation.

    Google Scholar 

  • Kamarck, T., Shiffman, S., Smithline, L., Goodie, J., Paty, J. et al (1998). The effects of task strain, social conflict, and emotional activation on ambulatory cardiovascular activity: daily life consequences of “recurring stress” in a multiethnic sample. Health Psychol, 17, 17–29.

    Article  PubMed  CAS  Google Scholar 

  • Kamarck, T., Shiffman, S., Muldoon, M., Sutton-Tyrell, K., Gwaltney, C. et al (2007). Ecological Momentary Assessment as a resource for social epidemiology. In A. Stone, S. Shiffman, A. Atienza, & L. Nebling (Eds.), The Science of Real-Time Data Capture: Self-Reports in Health Research (pp. 268–285). New York: Oxford University Press.

    Google Scholar 

  • Kihlstrom, J., Eich, E., Sandbrand, D., and Tobias, B. (2000). Emotion and memory: implications for self-report. In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implication for Research and Practice (pp. 81–99). Mahwah, NJ: Erlbaum.

    Google Scholar 

  • Korotitsch, W., and Nelson-Grey, R. (1999). An overview of self-monitoring research assessment and treatment. Psychol Assess, 2, 415–425.

    Article  Google Scholar 

  • Linton, S., and Gotestam, K. (1983). A clinical comparison of two pain scales: correlation, remembering chronic pain, and a measure of compliance. Pain, 17, 53–65.

    Article  Google Scholar 

  • Linton, S., and Melin, L. (1982). The accuracy of remembering chronic pain. Pain, 13, 281–285.

    Article  PubMed  CAS  Google Scholar 

  • McFall, R. (2005). Theory and utility -- key themes in evidence-based assessment: comment on special section. Am Psychol, 17, 312–323.

    Google Scholar 

  • McFarland, C., Ross, M., and DeCourville, N. (1989). Women’s theories of menstruation and biases in recall of menstrual symptoms. J Pers Soc Psychol, 57, 522–531.

    Article  PubMed  CAS  Google Scholar 

  • Menon, G., and Yorkston, E. (2000). The use of memory and contextual cues in the formation of behavioral frequency judgements. In A. Stone, J. Turkkan, C. Bachrach, J. Jobe, H. Kurtzman, & V. Cain (Eds.), The Science of Self-Report: Implications for Research and Practice (pp. 63–79). Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Muraven, M., Collins, R., Shiffman, S., and Paty, J. (2005). Daily fluctuations in self-control demands and alcohol intake. Psychol Addict Behav, 19, 140–147.

    Article  PubMed  Google Scholar 

  • Redelmeier, D., and Kahneman, D. (1996). Patients’ memories of pain medical treatments: real-time and retrospective evaluations of two minimally invasive procedures. Pain, 66, 3–8.

    Article  PubMed  CAS  Google Scholar 

  • Redelmeier, D., Katz, J., and Kahneman, D. (2003). Memories of colonoscopy: a randomized trial. Pain, 104, 187–194.

    Article  PubMed  Google Scholar 

  • Robinson, M., and Clore, G. (2002). Belief and feeling: evidence for an accessibility model of emotional self-report. Psychol Bull, 128, 934–960.

    Article  PubMed  Google Scholar 

  • Rock, E., Scott, J., Kennedy, D., Sridhara, R., Pazdur, R., and Burke, L. (2007). Challenges to use of health-related quality of life for Food and Drug Administration Approval of anticancer products. J Natl Cancer Inst Monogr, 25, 27–30.

    Google Scholar 

  • Ross, M. (1989). Relation of implicit theories to the construction of personal histories. Psychol Rev, 96, 341–357.

    Article  Google Scholar 

  • Saito, M., Kumano, H., Yoshiuchi, K., Kokubo, N., Ohashi, K., Yamamoto, Y. et al (2005). Symptom profile of multiple chemical sensitivity in actual life. Psychosom Med, 67, 318–325.

    Article  PubMed  CAS  Google Scholar 

  • Salovey, P., Sieber, W., Jobe, J., and Willis, G. (1993). The recall of physical pain. In N. Schwarz & S. Sudman (Eds.), Autobiographical Memory and the Validity of Retrospective Reports (pp. 89–106). New York: Springer-Verlag.

    Google Scholar 

  • Schwarz, N. (1996). Cognition and Communication: Judgmental Biases, Research Methods, and the Logic of Conversation. Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Schwarz, N. (2007). Retrospective and concurrent self-report: the rationale for real-time data capture. In A. Stone, S. Shiffman, A. Atienza, & L. Nebling (Eds.), The Science of Real-Time Data Capture: Self-Reports in Health Research (pp. 11–26). New York: Oxford University Press.

    Google Scholar 

  • Schwarz, N., and Oyserman, D. (2001). Asking questions about behavior: cognition, communication and questionnaire construction. Am J Eval, 22, 127–160.

    Google Scholar 

  • Schwarz, N., Wanke, M., and Bless, H. (1994). Subjective assessments and evaluations of change: some lessons learned from social cognitive research. Eeuro Rev Soc Psychol, 5, 181–210.

    Article  Google Scholar 

  • Schwarz, N., and Strack, F. (1991). Evaluating one’s life: a judgment model of subjective well-being. In F. Strack, M. Argyle, & N. Schwarz (Eds.), Subjective Well-Being: An Interdisciplinary Approach (pp. 27–47). Oxford: Pergamon Press.

    Google Scholar 

  • Shiffman, S. (2006). Designing protocols for Ecological Momentary Assessment. In A. Stone, S. Shiffman, A. Atienza, & L. Nebling (Eds.), The Science of Real-Time Data Capture: Self-Reports in Health Research. New York: Oxford University Press.

    Google Scholar 

  • Shiffman, S. (1993). Assessing smoking patterns and motives. J Consult Clin Psychol, 61, 732–742.

    Article  PubMed  CAS  Google Scholar 

  • Shiffman, S., Balabanis, M., Gwaltney, C., Paty, J., Gnys, M. et al (2007). Prediction of lapse from associations between smoking and situational antecedents assessed by ecological momentary assessment. Drug Alc Depend, 91, 159–168.

    Article  Google Scholar 

  • Shiffman, S., Hufford, M., Hickcox, M., Paty, J. A., Gnys, M., and Kassel, J. D. (1997). Remember that? A comparison of real-time vs. retrospective recall of smoking lapses. J Consult Clin Psychol, 65, 292–300.

    Article  PubMed  CAS  Google Scholar 

  • Shiffman, S., Hufford, M., and Stone, A. (2008). Ecological momentary assessment in clinical psychology. Annu Rev Clin Psychol, 4, 1–32.

    Article  PubMed  Google Scholar 

  • Shiffman, S., and Waters, A. (2004). Negative affect and smoking lapses: a prospective analysis. J Consult Clin Psychol, 72, 1192–201.

    Google Scholar 

  • Smith, W., and Safer, M. (1993). Effects of present pain level on recall of chronic pain and medication use. Pain, 55, 355–361.

    Article  PubMed  CAS  Google Scholar 

  • Stone, A., Schwartz, J., Broderick, J., and Shiffman, S. (2005). Variability of momentary pain predicts recall of weekly pain: a consequence of the peak (or salience) memory heuristic. Person Soc Psychol Bull 31, 1340–1346.

    Article  Google Scholar 

  • Stone, A., Schwartz, J., Neale, J., Shiffman, S., Marco, C., Hickcox, M. et al (1998). How accurate are current coping assessments? A comparison of momentary versus end-of-day reports of coping efforts. J Person Soc Psychol, 74, 1670–1680.

    Article  CAS  Google Scholar 

  • Stone, A., Shiffman, S., Atienza, A., and Nebling, L. (2007). The Science of Real-Time Data Capture: Self-Reports in Health Research. New York: Oxford University.

    Google Scholar 

  • Stone, A., Shiffman, S., Schwartz, J., Broderick, J., and Hufford, M. (2002). Patient non-compliance with paper diaries. Br Med J, 324, 1193–1194.

    Article  Google Scholar 

  • Stone, A., Turkkan, J., Jobe, J., Bachrach, C., Kurtzman, H., and Cain, V. (2000). The science of self report. Mahwah, NJ: Erlbaum.

    Google Scholar 

  • Stone, A., Broderick, J., Shiffman, S., and Schwartz, J. (2004). Understanding recall of weekly pain from a momentary assessment perspective: absolute accuracy, between- and within-person consistency, and judged change in weekly pain. Pain, 107, 61–69.

    Article  PubMed  Google Scholar 

  • Stone, A. A., and Shiffman, S. (1994). Ecological Momentary Assessment (EMA) in behavioral medicine. Annals of Behavioral Medicine, 16, 199–202.

    Google Scholar 

  • Thompson, C., Skowronski, J., Larsen, S., and Betz, A. (1996). Autobiographical Memory: Remembering What and Remembering When. Mahwah, NJ: Erlbaum.

    Google Scholar 

  • vandenBrink, M., Bandell-Hoekstra, F., and Abu-Saad, H. (2001). The occurrence of recall bias in pediatric headache: a comparison of questionnaire and diary data. Headache, 41, 11–20.

    Article  CAS  Google Scholar 

  • Wells, G., and Bradfield, A. (1998). “Good, you identified the suspect”: feedback to eyewitnesses distorts their reports of the witnessing experience. J Apply Psychol, 83, 360–376.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Psychiatry and Behavioral Science, Stony Brook University, Stony Brook, NY, 11994-8790, USA

    Arthur A. Stone (Distinguished Professor and Vice Chairman)

  2. Department of Psychology, University of Pittsburgh, Sennott Square, 210 S. Bouquet Street, Pittsburgh, PA, 15260, USA

    Saul S. Shiffman (Professor of Psychology)

Authors
  1. Arthur A. Stone
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Saul S. Shiffman
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Arthur A. Stone .

Editor information

Editors and Affiliations

  1. St Georges Hospital Medical School, Dept. Psychology, University of London, Cranmer Terrace, London, SW17 0RE, United Kingdom

    A. Steptoe

Rights and permissions

Reprints and permissions

Copyright information

© 2010 Springer Science+Business Media, LLC

About this chapter

Cite this chapter

Stone, A.A., Shiffman, S.S. (2010). Ecological Validity for Patient Reported Outcomes. In: Steptoe, A. (eds) Handbook of Behavioral Medicine. Springer, New York, NY. https://doi.org/10.1007/978-0-387-09488-5_8

Download citation

Publish with us

Policies and ethics

Search

Navigation