An Introduction to Diagnostic Meta-analysis

Abstract

Systematic review, and its corresponding statistical analysis, is becoming popular in the literature to assess the diagnostic accuracy of a test. When correctly performed, this research methodology provides fundamental data to inform medical decision making. This chapter reviews key concepts of the meta-analysis of diagnostic test accuracy data, dealing with the particular case in which primary studies report a pair of estimates of sensitivity and specificity. We describe the potential sources of heterogeneity unique to diagnostic test evaluation and we illustrate how to explore this heterogeneity. We distinguish two situations according to the presence or absence of inter-study variability and propose two alternative approaches to the analysis. First, simple methods for statistical pooling are described when accuracy indices of individual studies show a reasonable level of homogeneity. Second, we describe more complex and robust statistical methods that take the paired nature of the accuracy indices and their correlation into account. We end with a description of the analysis of publication bias and enumerate some software tools available to perform the analyses discussed in the chapter.

Appendix

Example 1

For this example we selected the 17 studies included in Scheidler et al.’s meta-analysis (Table 8.1). In their meta-analysis, they evaluated the diagnostic accuracy of lymphangiography (LAG) to detect lymphatic metastasis in patients with cervical cancer.

Table 8.1 The studies included in Scheidler et al.’s meta-analysis

First, the indices of diagnostic accuracy, sensitivity and specificity (Fig. 8.1) or the positive and negative LRs (Fig. 8.2) of the reviewed studies are described for exploratory purposes using paired forest plots as obtained with Meta-DiSc.

Fig. 8.1

Forest plot of sensitivity and specificity. The box sizes are proportional to the weights assigned and the horizontal lines depict the confidence intervals

Fig. 8.2

Forest plot of positive and negative LRs. The box sizes are proportional to the weights assigned and the horizontal lines depict the confidence intervals

Second, still within the graphical data exploration, we can illustrate the TPR or sensitivity, the FPR (i.e. 1 − specificity), and the LRs (LR + and LR−) organized by one of these indices (Fig. 8.4) or illustrate the pairing indices on a ROC space (Fig. 8.3). At this exploratory phase, all graphical representations should not include pooled estimates of accuracy.

Fig. 8.3

The ROC plane: Plot of 1-specificity against sensitivity

Fig. 8.4

Forest plot with studies sorted by FPR: Heterogeneity is evident

To perform these exploratory analyses, we can use free software (Meta-DiSc, RevMan or the DiagMeta package in the R environment) or any other commercial software.

In this example, and looking at the forest plot generated, we cannot rule out the presence of heterogeneity across the studies included in the review; thus, the analysis should focus on fitting an sROC model.

Given the limitations of the Moses–Littenberg model, we fit a bivariate model using the DiagMeta package. The output is presented below:

> bivarROC(Scheidler)
	ML	MCMC	lower limit	upper limit
average TPR%	67.38561	67.59189	60.52091	74.75159
average FPR%	16.22516	16.05203	9.25013	25.49491
SD logit TPR	0.34943	0.31889	0.04271	0.87571
SD logit FPR	0.90087	1.06136	0.63934	1.84290
correlation	−0.23882	−0.53898	−0.99999	0.59240

Because the estimated correlation between logit (sensitivity) and logit (specificity) is small and it cannot be ruled out that it is not different from zero, the results estimated by the bivariate model do not significantly differ from those obtained through separate pooling of sensitivity and specificity. Based on the same example, the results using a simple pooling with a fixed or random effects model according to the variability of each of the indices are as follows:

> twouni(subset(Scheidler,GROUP=='LAG'))
TPR	TPR	lower limit	upper limit
Fixed effects	0.6711590	6.218139e-01	0.7169960
Random effects from ML	0.6763973	6.056993e-01	0.7398633
Random effects from MCMC	0.6729242	6.148178e-01	0.7327660
SD of REff	0.0692814	5.935713e-07	0.7516062
FPR	FPR	lower limit	upper limit
Fixed effects	0.1996143	0.1764147	0.2250311
Random effects from ML	0.1619847	0.1059149	0.2397768
Random effects from MCMC	0.1631190	0.1035210	0.2426649
SD of REff	0.9576528	0.5716529	1.5829222

Figure 8.5 shows the sROC curve fitted with a STATA bivariate model, together with the estimated summary point and confidence and prediction intervals.

Fig. 8.5

Fitted SROC curve: Study estimates are shown as circles sized according to the total number of individuals in each study. Summary sensitivity and specificity are depicted by the square marker and the 95 % confidence region for the summary operating point is depicted by the small oval in the centre. The larger oval is the 95 % prediction region (confidence region for a forecast of the true sensitivity and specificity in a future study). The summary curve is from the HSROC model

Example 2

For this illustration we used Fahey et al.’s data (Table 8.2). The goal of their study was to estimate the accuracy of the Papanicolaou (Pap) test for detection of cervical cancer and precancerous lesions.

Table 8.2 Data from Fahy et al.’s study

The sensitivity and specificity forest plots (data not shown) confirm the presence of substantial heterogeneity, in both indices, across the studies included in the review. Figure 8.6 shows the representation of the studies in the ROC space. The slight curvilinear pattern of their distribution suggests the presence of a correlation between sensitivity and specificity.

Fig. 8.6

ROC plane: Plot of 1-specificity versus sensitivity

Using Meta-DiSc we calculated the Spearman correlation coefficient between the TPR and FPR logits and obtained a positive and statistically significant correlation of 0.584 (p < 0.001) which confirms the results of the bivariate adjustment obtained using the package DiagMeta:

Estimates and 95 % confidence intervals from mcmc samples
	ML	MCMC median	lower limit	upper limit
average TPR%	65.56718	64.93881	57.58497	72.49102
average FPR%	25.38124	25.27866	18.74132	32.57494
SD logit TPR	1.21834	1.27374	1.04000	1.59237
SD logit FPR	1.22834	1.27623	1.02968	1.60834
correlation	0.77408	0.77709	0.61593	0.87730
Posterior probability that rho positive 1
Correlation positive - threshold model appropriate

With this information in hand, we conclude that the most appropriate method to summarize the results of the meta-analysis is using an sROC curve (Fig. 8.7). This curve was fitted using the bivariate model produced by the macro METANDI in STATA. Figure 8.8 shows the results of a comparable analysis with Meta-DiSc using the Moses–Littenberg model which, in this case, has generated a practically identical sROC curve to that in Fig. 8.7.

Fig. 8.7

Fitted SROC curve (bivariate model)

Fig. 8.8

Fitted SROC curve using the Moses–Littenberg model