Approaches to the Development and Use of PRO Measures: A New Roadmap

Abstract

Various recent developments are driving an evolution in the way PRO measures are developed. For example, drug regulatory agencies (e.g. the FDA) can now be more substantively involved or consulted on measure development work; the patients’ role is now seen as going beyond being a study subject to being involved as research partners. Use of modern test theory in development of scoring algorithms and overall measure development work is now the new ‘gold standard’.

The challenge

(anecdote)

In the measurement of glucose, the glucose assay and computational algorithm are embodied in the instrument that is used to provide glucose readings…The quality of life measurement process is essentially an assay that defines the construct, uses question and response formats to obtain answers, and has an algorithm that scores those responses to yield quality of life readings. However, because of the subjectivity and indirect measurement issues, it is harder to adopt a standard assay for quality of life.

Testa 2000

Appendix: Technical Notes

Practical Considerations in Study Design During PRO Development

Sample Size

Sample size considerations differ between qualitative and quantitative research. In the former, it is not possible to determine the needed sample size prior to data collection; rather sample adequacy is determined in the course of data collection. Data collection continues until ‘saturation’ has been reached, which reflects a situation where further data collection (e.g. interviews) is not yielding new data (Kerr et al. 2010). On the other hand, in quantitative research, sample size is dependent on the particular statistical analysis performed. Required sample size will reflect the intended power of analysis, the magnitude of effect size to be observed and the chosen level of significance and reliability of measurement (Lipsey 1990). Exploratory studies, where the magnitude of the effect size and reliability are unknown a priori, may present some challenges in this regard. A useful recommendation is to use a sample matrix based on key disease or treatment characteristics for a particular disease, where each subcategory (each cell) should have at least 15 subjects (Johnson et al. 2011). For initial estimates of reliability and validity, at least 200 subjects are recommended (Frost et al. 2007a). If a test-retest correlation of 0.85 is observed with a sample size of 100, the 95% confidence interval is 0.78–0.90, while a sample size of 150 would narrow this to 0.8–0.89 (Johnson et al. 2011).

Rules of thumb on sample size requirements for correlation analysis and factor analysis vary in their guidance, ranging from 5 to 20 observations per variable with more suggestion above and below this ratio (Costello and Osborne 2005). However, the minimum sample size required for accurate recovery of the population factor pattern matrix is influenced by many factors including the distribution and reliability of the variables, degree of association among variables, communalities and degree to which factors are overidentified (Reise et al. 2000; Schmitt 2011). Thus power and precision ought to be core considerations in parametric estimation-based factor methods (Schmitt 2011), while in non-parametric approaches when communalities are high, sample size of 100 may be adequate (Reise et al. 2000).

Assessment of adequacy of sample size for a given statistical test should be made along with other key considerations relating to the sample, for instance, ensuring that the target population is adequately represented along with all important disease characteristics. Otherwise, appropriate tools should be applied to indicate the uncertainty surrounding estimates, e.g. using confidence intervals in presenting results.

Missing Data

Situations where a question or an entire questionnaire has not been completed are common during data collection in QoL research. The reason behind the missing data has an influence on choice of tools for dealing with the consequent problems in data analysis, for example, whether an item is skipped by mistake or due to its irrelevance. There are three main classifications of patterns of missing data: missing completely at random (MCAR), missing at random (MAR) and missing not at random (MNAR) (Fayers and Machin 2007). MCAR arises where the probability of having a missing item (questionnaire) is independent of previous or unobserved current and future scores. MAR occurs where missingness is dependent on known covariates and scores of previous items, but not on the unobserved scores. The third case relates to where the unobserved HRQoL influences the missingness. The presence of MAR and MCAR is not worrisome, as their impact on accurate measurement of HRQoL is minimal (Leidy et al. 1999). MNAR causes the greatest concern as its presence may lead to an over- or underestimation of HRQoL, highlighting the need for transparent approaches in addressing its presence.

There are no clear guidelines on the number of missing items to warrant the exclusion of an entire respondent’s questionnaire from analyses although Streiner and Norman (2008) have mentioned a ceiling of 5% of items. However, it’s worth noting that where Rasch scoring is applied, a higher number of missing items may be tolerated without much bias in measurement (Fayers and Machin 2007). On the other hand, in some situations (e.g. during instrument development work), data imputation to replace the missing data offers a viable alternative. This is done in various ways including using the last observed value carried forward, by calculating a simple mean or using regression methods (Fairclough 2010). Other more sophisticated imputation approaches such as hot-deck and Markov chain are capable of preserving variability in the data.

Strategies for preventing problems of missing data from rising should be considered, e.g. inviting the patients to cross-check their questionnaires to make sure all items are completed (~assuming paper-and-pencil in-person administration). Here electronic PRO administration may have advantages.

Suggestions for Statistical Analysis in PROM Development

Data should initially be explored through descriptive analysis of each variable, calculating measures of central tendency (mean, median), variability (SD) and interquartile range for continuous variable. Frequency counts for ordinal and categorical variables. Further analyses will involve making inferences based on various hypotheses tests. In order to reject a null hypothesis, observed probability of a false positive, type I error, as reflected in P-value, needs to be less than the required level of significance (α) (Altman et al. 2013). Most studies will use a level of significance (α) of 5%. Where several hypotheses need to be simultaneously tested, Bonferroni adjustment should be applied to the level of significance, as (α/k), where k is the number of tests (Fayers and Machin 2007).

• Testing for differences between two means should use independent or paired t-test, depending on whether the two means are mutually exclusive or are related. The Mann-Whitney and Wilcoxon tests are the non-parametric alternatives, respectively, for situations where assumptions of the t-tests are not met. These latter tests are somewhat more conservative.

• Hypothesis tests involving differences among more than two groups should be carried out using the ANOVA test. Where the core assumptions of this test are not met, particularly, the assumption of homogenous variances across group, the Kruskal-Wallis test should be used alternatively.

• Testing of hypotheses relating to associations between means of variables should be carried out based on Pearson’s correlations. Where the data is not continuous, Spearman’s rank correlation should be used.

• Polychoric correlations can be estimated in order to assess multicollinearity among items. This type of correlation produces consistent and robust results in ordinal data. They are based on the assumption that the variable is linear and continuous but divided up in a series of categories (Holgado-Tello et al. 2010). Multicollinearity is identified when correlation coefficients are 0.8 or greater.

• Possible influences on the magnitude of observed inter-item correlations including range of score values, homogeneity of items, distribution of the data (particularly departures from normality) and existence of outliers in the data (Fayers and Machin 2007) should be explored. Normality assumption implies the absolute value of skewness not exceeding 3, while the absolute value of Kurtosis must not be greater than 7 (Ozer et al. 2009; Byrne 2011). While the former impacts on means, covariance tends to be vulnerable to Kurtosis values (Byrne 2011).

Further statistical analyses carried out during construct validation can use various forms of regression methods, modelling latent variables including exploratory factor analysis, confirmatory factor analysis and the Rasch model.

Performing Exploratory Factor Analysis

The aim is to identify the smallest number of interpretable factors explaining the covariation among items (Muthen and Muthén 2010). This involves first generating the variance-covariance matrix, followed by the estimation of the factors which entails putting together those items sharing the highest covariation. Subsequently, the initial factor solution is rotated in order to achieve a simple structure that is more interpretable, as the initial solution is not unique (DeVellis 2016).

To perform an EFA on the instrument, first a polychoric correlation matrix should be generated. This more appropriately takes into account the ordinality of the data and remains robust when data are skewed, in comparison to the conventional Pearson’s correlation coefficients (Byrne 2011). The initial factor estimation can be carried out using a robust diagonally least squares estimator (WLSMV) which yields robust test statistics, parameter estimates and standard errors when indicator variables are categorical and where normality assumptions are violated (Byrne 2011). Rotation can be performed using the Geomin routine (~available in Mplus software, equivalents might be known by other names in other software), which allows correlation among factors. This rotation is particularly suitable for psychosocial domains known to be highly related (Lackey et al. 2003). Where the factors are not related, Geomin still performs well yielding results comparable to orthogonal rotation routines. Choice of the appropriate number of factors to be extracted will be based on the parallel analysis and will be confirmed by statistical goodness of fit measures (Schmitt 2011). Kaiser’s rule, based on size of eigenvalues; scree plot, which is a graph of number of factors against eigenvalues; and parallel analysis, comparing actual against ones randomly generated, should also be reported. The following criteria can be applied:

• Kaiser’s rule: factors with eigenvalues greater than are included (Kaiser (1960) in (DeVellis 2016)).

• Scree plot: all factors to the left of the ‘ankle’ are extracted, where there is a change in the slope.

• Parallel analysis: the last factor to be retained must have an eigenvalue greater than the one that would be produced randomly (Williams et al. 2010).

An advantage of factor estimation using Likelihood methods is the possibility to generate goodness of fit indices to explore how well hypothesised models fit data. These can be classified into three groups:

• Chi-square-based indices compare a single factor model against a model with the chosen number of factors (k). For the ‘chi-goodness of fit test’, a non-significant chi-statistic represents good fit (Lackey et al. 2003).

• Practical fit indices evaluate the proportionate improvement in the model by comparing a hypothesised model against a less restricted baseline model (Byrne 2011). For comparative fit index and Tucker-Lewis Index, values of below 0.9 and 0.95 indicate acceptable and adequate fit, respectively (Schmitt 2011).

• Absolute fit indices are based on analysis of residuals after fitting the model to the data. For Root Mean Square Error of Approximation (RMSEA), a value below 0.05 shows good fit, 0.08–0.1 mediocre fit and above 0.1 poor fit (Brown 2014). For the Standardized Root Mean Square Residual (SRMR), values lower than 0.05 indicate ‘adequate fit’ although values below 0.8 are still acceptable. The Weighted Root Mean Square Residual uses a cut-off value of 0.95 for good fit.

Performing Rasch Analysis

Appropriate fit to the Rasch model ensures that an PRO measure is sufficiently unidimensional and that it complies with conjoint measurement principles, a precondition for converting the data from the instrument into interval scales (Bond 2004). The intention of Rasch analysis, therefore, is to evaluate whether data have sufficient fit to the model to warrant such claims.

Demonstrating conformity to the Rasch model may have several advantages for a PRO instrument. First, ordinal scores may be transformed into interval-level logit scores using the RM—a requisite property for the calculation of effect sizes and other statistics in clinical research that is usually taken for granted (Reise and Haviland 2005). Second, by conceptualising measurement error as an item-level property, high reliability can be attained even with a shorter questionnaire, making it possible to minimise patient burden without compromising precision (Reeve et al. 2007). In addition, much more complex comparisons such as ‘anchoring’ or ‘equating’ may be easily carried out between an instrument and other instruments.

When assessing conformity to the Rasch model, its assumptions and properties involve the following:

• 1. Assessing whether the response categories are functioning optimally. Average latent measure across observations in a response category and category thresholds should monotonically increase with the category; each response category should have a distinct peak on the category probability curve graph reflecting the space along the latent variable where it is most probable (Linacre 1999). Category characteristic curves define the most likely response category for a specific person location value on the latent variable. The category threshold indicates a location on the latent variable where the probability of selecting adjacent categories is equivalent (Linacre 1999).

• 2. Testing item and person fit to the model. This uses residuals obtained after fitting data to the model, calculating a fit residual statistic and an item-trait interaction chi-statistic.

  The residual statistic for items is calculated as the squared summation of the standardised residuals of the responses of all persons to an item (Andrich et al. 2012b). Fit residuals exceeding | ± 2.5| indicate poor fit (Andrich et al. 2012a). As the Rasch model does not distinguish between items and persons, the residual fit statistics for persons is calculated and interpreted in a similar way as the statistic for items (Bond and Fox 2015).

• 3. The item-trait interaction test of fit assesses the discrepancy between actual and model scores of class intervals (which group patients according to ability), visually reflected by discrepancies between the ICC and empirical counterpart. An item chi-value is generated by adding all standardised differences for class intervals (Andrich et al. 2012a).

• 4. Testing of overall model fit. Mean fit residual value of 0 and standard deviation of 1 reflect overall model fit (Shea et al. 2009). The item-trait interaction statistics for all items are summed up into total item-trait interaction statistic. Optimal fit is reflected in a non-significant statistic (chi-squared statistic, P-value >0.05). Good fit to the Rasch model implies that the hierarchical ordering of the items remains invariant across the different levels of disease severity assessed by the construct.

• 5. How well the instrument can differentiate persons according to disease severity should be assessed. This is reflected in the Person Separation Index (PSI) which reflects the proportion of variance explained by the model out of the total person variability (Wright and Masters 1982; Bond and Fox 2015). A PSI of 0.8 reflects capability to reliably distinguish patients into at least two groups of severity, e.g. high and low severity.

• 6. Assessing targeting of items. The item-person map is visually examined for adequacy in spread of the items along the breadth of the latent variable, and ideally there should not be large gaps between items (Wright and Masters 1982); mean location of persons should be close to 0 to match the item mean location centred at 0 logits (Gorecki et al. 2011).

• 7. Assessing unidimensionality. First, a principal component analysis should be carried out on the residuals after fitting the Rasch model. Unidimensionality is supported if the first component accounts for no more than 30% of the variance in the data and has an eigenvalue of 3 or less (Linacre 1998). A more stringent assessment of unidimensionality has been suggested by Smith (2002). Items are grouped according to their loading on the first residual factor, comprised of high positive and high negative loading items, respectively. Pairs of person estimates generated from the two item sets are compared using a series of t-tests. If the proportion of significant tests (or the lower bound of its confidence interval) exceeds 5%, unidimensionality is ruled out (Tennant and Pallant 2006).

• 8. The assumption of local independence can be assessed by examining the correlation matrix of the item residuals. Residual correlation exceeding 0.2–0.3 reflects a violation of this assumption. The magnitude of the response dependence is calculated as the shift in the latent variable range representing a given response choice on the dependent item, induced by a particular response choice on the independent item (Andrich et al. 2012b).

• 9. Assessing for invariance across demographic factors. Differential item functioning (DIF) can be assessed for key demographic factors using a two-way ANOVA test. A significant main effect (demographic variable) at a 0.05 level of significance, with Bonferroni adjustment, indicates presence of uniform DIF. On the other hand, a significant interaction effect (demographic variable X class interval representing ability groups along the latent trait), after Bonferroni adjustment, indicates non-uniform DIF (Andrich et al. 2012a). Identification of DIF requires a pure set of items, upon which the scale is anchored (Teresi and Fleishman 2007).

  Any action on DIF requires an understanding of its magnitude and impact. Magnitude indicates the difference between item difficulty estimates based on all patients and comparable estimates specific for each demographic group (Linacre 2009). The impact of the DIF on estimation of person estimates is assessed by comparing person estimates generated from the DIF-free items against estimates based on all items including those with DIF (Pallant and Tennant 2007). Using a t-test, significant results, at 0.05 level of significance, indicate that DIF has an impact. The Item Characteristic Curve (ICC) of the two series may also be useful in assessing whether the pairs of person ability estimates agree. Impact of DIF can also be explored by assessing whether the test characteristic curves (TCCs) from different demographic groups are comparable, i.e. whether there is a relationship between the raw score and the underlying latent variable varies across the demographic groups. Identical TCCs indicate the absence of impact of DIF on the total score (Edelen et al. 2006). The criterion for magnitude of DIF is also relevant for differential scale functioning.