Skip to main content
SpringerLink
Log in

Assessment of meaningful change in routine outcome measurement (ROM) with a combination of a longitudinal and a ‘classify and count’ approach

  • Published:
Quality & Quantity Aims and scope Submit manuscript

Abstract

To assess significant changes of health status in people receiving health care, distribution-based and anchor-based methods have been proposed. However, there is no real consensus on what method is the best for evaluating clinically meaningful change. To maximize the internal and external validity of outcome assessment, we propose combining two approaches as recommended by recent practical guidelines on this field. Specifically, we suggest applying longitudinal hierarchical linear models on subgroups of patients showing reliable change and reliable and clinically significant change. This combined approach improved the model’s ability (1) to quantify the magnitude of changes to be reliable and clinically meaningful and (2) to select significant predictors of changes. An empirical application on a prevalence sample of Italian outpatients attending four community mental health services was done. A cross-sectional model and three longitudinal models were applied on the entire study sample and reliable and clinically meaningful change subsamples to investigate the magnitude of change and the predictive effect on outcomes of clinical, socio-demographic and process variables on different patients’ subgroups. Differences were found suggesting that both the statistical method and the sample used to calculate individual changes affect the estimates. The main conclusion is that ignoring the longitudinal data structure or including patients with unreliable change at the follow-up might result in misleading inferences that can alter the real magnitude of changes and the contributions of predictors. The approach proposed provides robust feedback to clinicians on clinically significant change and can be recommended in outcome studies and research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Bauer, S., Lambert, M.J., Nielsen, S.L.: Clinical significance methods: a comparison of statistical techniques. J. Pers. Assess. 82, 60–70 (2003)

    Article  Google Scholar 

  • Cella, D., Bullinger, M., Scott, C., Barofsky, I.: The Clinical Significance Consensus Meeting Group: group vs individual approaches to understanding the clinical significance of differences or changes in quality of life. Mayo Clin. Proc. 77, 384–392 (2002)

    Article  Google Scholar 

  • Christensen, L., Mendoza, J.L.: A method of assessing change in a single subject: an alteration of the RC index. Behav. Res. Ther. 17, 305–308 (1986)

    Article  Google Scholar 

  • Clarke, P.S.: Causal analysis of individual change using the difference score. Epidemiology 15, 414–421 (2004)

    Article  Google Scholar 

  • Cohen, J.: Statistical Power Analysis for the Behavioural Sciences. Academic Press, London (1969)

    Google Scholar 

  • Commonwealth Department of Health and Ageing: National Outcome and Casemix Collection: Overview of Clinical Measures and Data Items. Commonwealth Department of Health and Ageing, Canberra (2002)

  • Crosby, R.D., Kolotkin, R.L., Williams, G.R.: Defining clinically meaningful change in health-related quality of life. J. Clin. Epidemiol. 56, 395–407 (2003)

    Article  Google Scholar 

  • Department of Health: Mental Health Information Strategy. Department of Health, London (2001)

  • Diggle, P.J., Heagerty, P., Liang, K.Y., Zeger, S.L.: Analysis of Longitudinal Data. Oxford University Press, Oxford (2002)

    Google Scholar 

  • Eisen, S.V., Ranganathan, G., Seal, P., Spiro, A.: Measuring clinically meaningful change following mental health treatment. J. Behav. Health Serv. Res. 34, 272–389 (2007)

    Article  Google Scholar 

  • Evans, C., Margison, F., Barkham, M.: The contribution of reliable and clinically significant change methods to evidence-based mental health. Evid. Based Ment. Health 1, 70–72 (1998)

    Article  Google Scholar 

  • Food and Drug Administration: Guidance for industry on patient-reported outcome measures: use in medical product development to support labeling claims. Fed. Regist. 74(235), 65132–65133 (2009)

  • Goldstein, H.: Multilevel Statistical Models. Edward Arnold, London (1995)

    Google Scholar 

  • Grundy, C.T., Lambert, M.J., Grundy, E.M.: Assessing clinical significance: application to the Hamilton Rating Scale for depression. J. Ment. Health 5, 25–33 (1996)

    Article  Google Scholar 

  • Hafkenscheid, A.: Psychometric measures of individual change: an empirical comparison with the Brief Psychiatric Rating Scale (BPRS). Acta Psychiatr. Scand. 101, 235–242 (2000)

    Article  Google Scholar 

  • Hageman, W.J., Arrindell, W.A.: Establishing clinically significant change: increment of precision between individual and group level of analysis. Behav. Res. Ther. 37, 1169–1193 (1999)

    Article  Google Scholar 

  • Jacobson, N.S., Truax, P.: Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. J. Consult. Clin. Psychol. 59, 12–19 (1991)

    Article  Google Scholar 

  • Jacobson, N.S., Follette, W.C., Revenstorf, D., Baucom, D.H., Hahlweg, K., Margolin, G.: Variability in outcome and clinical significance of behavioral marital therapy: a reanalysis of outcome data. J. Consult. Clin. Psychol. 52, 497–504 (1984)

    Article  Google Scholar 

  • Kazis, L.E., Anderson, J.J., Meenan, R.F.: Effect sizes for interpreting changes in health status. Med. Care 27, 178–189 (1989)

    Article  Google Scholar 

  • Liang, M.H., Larson, M.G., Cullen, K.E., Schwartz, J.A.: Comparative measurement efficiency and sensitivity of five health status instruments for arthritis research. Arthritis Rheum. 28, 542–547 (1985)

    Article  Google Scholar 

  • Lovaglio, P.G., Monzani, E.: Evaluation of the Health of the Nation Outcome Scales in a community setting population. Qual. Life Res. 21, 1643–1653 (2012)

    Article  Google Scholar 

  • Lydick, E., Epstein, R.S.: Interpretation of quality of life changes. Qual. Life Res. 2, 221–226 (1993)

    Article  Google Scholar 

  • Matthey, S.: Calculating clinically significant change in postnatal depression studies using the Edinburgh Postnatal Depression Scale. J. Affect. Disord. 78, 269–272 (2004)

    Article  Google Scholar 

  • McGlinchey, J.B., Atkins, D.C., Jacobson, N.S.: Clinical significance methods: which one to use and how useful are they? Behav. Ther. 33, 529–550 (2002)

    Article  Google Scholar 

  • Monzani, E., Erlicher, A., Lora, A., Lovaglio, P., Vittadini, G.: Does community care work? A model to evaluate the effectiveness of mental health services. Int. J. Ment. Health Syst. 2, 10 (2008)

    Article  Google Scholar 

  • Nich, C., Carroll, K.: Now you see it, now you don’t: a comparison of traditional versus random effects regression models in the analysis of longitudinal follow-up data from a clinical trial. J. Consult. Clin. Psychol. 65, 252–261 (1997)

    Article  Google Scholar 

  • Norman, G.R., Sridhar, F.G., Guyatt, G.H.: Relation of distribution- and anchor-based approaches in interpretation of changes in health-related quality of life. Med. Care 39, 1039–1047 (2001)

    Article  Google Scholar 

  • Parabiaghi, A., Rapisarda, F., D’Avanzo, B., Erlicher, A., Lora, A., Barbato, A.: Measuring clinical change in routine mental health care: differences between first time and longer term service users. Aust. N.Z. J. Psychiatry 45, 558–568 (2011)

    Article  Google Scholar 

  • Pirkis, J.E., Burgess, P.M., Kirk, P.K., Dodson, S., Coombs, T.J., Williamson, M.K.: A review of the psychometric properties of the Health of the Nation Outcome Scales (HoNOS) family of measures. Health Qual. Life Outcomes 3, 76 (2005)

    Article  Google Scholar 

  • Raudenbush, S.W., Chan, W.S.: Application of a hierarchical linear model to the study of adolescent deviance in an overlapping cohort design. J. Consult. Clin. Psychol. 61, 941–951 (1993)

    Article  Google Scholar 

  • Rogosa, D.R., Willett, J.B.: Demonstrating the reliability of the difference score in the measurement of change. J. Educ. Meas. 20, 335–353 (1983)

    Article  Google Scholar 

  • Rogosa, D.R., Willett, J.B.: Understanding correlates of change by modelling individual differences in growth. Psychometrika 50, 203–228 (1985)

    Article  Google Scholar 

  • Rogosa, D., Brandt, D., Zimowski, M.: A growth curve approach to the measurement of change. Psychol. Bull. 92, 726–748 (1982)

    Article  Google Scholar 

  • Rossi, A., Amaddeo, F., Bisoffi, G., Ruggeri, M., Thornicroft, G., Tansella, M.: Dropping out of care: inappropriate terminations of contact with community-based psychiatric services. Br. J. Psychiatry 181, 331–338 (2002)

    Article  Google Scholar 

  • Sherman, M., Le Cessie, S.: A comparison between bootstrap methods and generalized estimating equations for correlated outcomes in generalized linear models. Commun. Stat. 26, 901–925 (1997)

    Article  Google Scholar 

  • Speer, D.C.: Clinically significant change: Jacobson & Truax (1991) revisited. J. Consult. Clin. Psychol. 60, 402–408 (1992)

    Article  Google Scholar 

  • Speer, D.C.: What is the role of two-wave designs in clinical research? Comment on Hageman and Arrindell. Behav. Res. Ther. 37, 1203–1210 (1999)

    Article  Google Scholar 

  • Speer, D.C., Greenbaum, P.E.: Five methods for computing significant individual client change and improvement rates: support for an individual growth curve approach. J. Consult. Clin. Psychol. 63, 1044–1048 (1995)

    Article  Google Scholar 

  • Stefanski, L.A., Cook, J.R.: Simulation-extrapolation: the measurement error jackknife. J. Am. Stat. Assoc. 90, 1247–1256 (1995)

    Article  Google Scholar 

  • Trauer, T.: Assessment of change in outcome measurement. In: Trauer, T. (ed.) Outcome Measurement in Mental Health, pp. 206–218. Cambridge University Press, Cambridge (2010)

    Google Scholar 

  • White, H.: Maximum likelihood estimation of misspecified models. Econometrica 50, 1–25 (1982)

    Article  Google Scholar 

  • Willet, J.B.: Questions and answers in the measurement of change. In: Rothkopf, E. (ed.) Review of Research in Education (1988–89), pp. 345–422. American Education Research Association, Washington, DC (1988)

    Google Scholar 

  • Wing, J.K., Beevor, A.S., Curtis, R.H., Park, S.B., Hadden, S., Burns, A.: Health of the Nation Outcome Scales (HoNOS). Research and development. Br. J. Psychiatry 172, 11–18 (1998)

    Article  Google Scholar 

  • Wyrwich, K., Nienaber, N., Tierney, W., et al.: Further evidence supporting an SEM-based criterion for identifying meaningful intra-individual changes in health-related quality of life. J. Clin. Epidemiol. 52, 861–873 (1999)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Statistics and Quantitative Methods, University Bicocca-Milan, Via Bicocca degli Arcimboldi 8, 20126 , Milan, Italy

    Pietro Giorgio Lovaglio

  2. Laboratory of Epidemiology and Social Psychiatry, IRCCS ‘Mario Negri’ Institute for Pharmacological Research, Via La Masa 19, 20156 , Milan, Italy

    Alberto Parabiaghi

Authors
  1. Pietro Giorgio Lovaglio
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alberto Parabiaghi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pietro Giorgio Lovaglio.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Lovaglio, P.G., Parabiaghi, A. Assessment of meaningful change in routine outcome measurement (ROM) with a combination of a longitudinal and a ‘classify and count’ approach. Qual Quant 48, 2479–2499 (2014). https://doi.org/10.1007/s11135-013-9902-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11135-013-9902-9

Keywords

Search

Navigation