Visualizing inconsistency in network meta-analysis by independent path decomposition
- 1.9k Downloads
In network meta-analysis, several alternative treatments can be compared by pooling the evidence of all randomised comparisons made in different studies. Incorporated indirect conclusions require a consistent network of treatment effects. An assessment of this assumption and of the influence of deviations is fundamental for the validity evaluation.
We show that network estimates for single pairwise treatment comparisons can be approximated by the evidence of a subnet that is decomposable into independent paths. Path-based estimates and the estimate of the residual evidence can be used with their contribution to the network estimate to set up a forest plot for the consistency assessment. Using a network meta-analysis of twelve antidepressants and controlled perturbations in the real and constructed consistent data, we discuss the consistency assessment by the independent path decomposition in contrast to an approach using a recently presented graphical tool, the net heat plot. In addition, we define influence functions that describe how changes in study effects are translated into network estimates.
While the consistency assessment by the net heat plot comprises all network estimates, an independent path decomposition and visualisation in a forest plot is tailored to one specific treatment comparison. It allows for the recognition as to whether inconsistencies between different paths of evidence and outlier effects do affect the considered treatment comparison.
The approximation of the network estimate for a single comparison by the evidence of a subnet and the visualisation of the decomposition into independent paths provide the applicability of a graphical validation instrument that is known from classical meta-analysis.
KeywordsNetwork meta-analysis Multiple treatments comparison meta-analysis Mixed treatment comparison meta-analysis Inconsistency Influence diagnostics Forest plot
In medical practice, several treatments are frequently suitable for a single indication, but often only two or three of them are directly compared in one study. Network meta-analysis is an approach that combines the information from all clinical trials on any of the treatments based on the assumption of consistent treatment effects and the inclusion of indirect comparisons (for an overview see e.g. ). The evaluation of the consistency and the influence of possible deviations of this assumption on network estimates plays an important role in the validation of results , especially as more complex models are fitted . In classical meta-analysis, a forest plot of study effects and the pooled estimate offers the visualisation of outlier effects and their contribution to the aggregated treatment effect . In network meta-analysis, the homogeneity between studies of each pairwise treatment comparison can also be analysed using forest plots. However, to assess the consistency assumption in the network, study-based forest plots are not directly applicable, since for different pairwise treatment comparisons various effects are expected . Therefore, a generalised forest plot approach is needed.
Salanti et al.  graphically analysed consistency using a forest plot of the differences between direct and indirect evidence in single network loops. In a forest plot, estimates based on direct, indirect (obtained by back-calculation or node-splitting ), and combined evidence for one treatment comparison can be compared [1, 2] without reflecting detailed sources of potential inconsistencies. Tools for regression diagnostics like the plot of posterior mean deviance of individual data points [6, 7], the plot of leverage against the residual deviance for each data point , or the assessment of PRESS residuals and studentised residuals  allow for the singling out of individual studies or a set of studies that compared the same treatments whose direct evidence is badly fitted and may be responsible for heterogeneity or inconsistency. For an influence analysis of potentially inconsistent direct evidence, the contribution of direct evidence to network estimates has to be taken into account. Krahn et al.  proposed a matrix visualisation, called net heat plot, that highlights hot spots of inconsistency between specific direct evidence in the whole network and renders possible drivers transparent.
None of these approaches can offer all the analytical capacities available in a classical meta-analysis forest plot to a consistency assessment in a network meta-analysis: namely the composition of the network effect estimate based on direct evidence, the consistency between different evidence sources, outlier observations and the influence on aggregated treatment effect estimates.
In the following, we show that a single network effect estimate (e.g. for the comparison between treatments A and B) can also be approximated by the evidence of a subnet that is decomposable into independent paths and can be visualised in a forest plot. For a consistency investigation, it allows the visualisation of the contribution of each independent path as well as that of the residual evidence in combination with their corresponding treatment effect estimate. Due to the additional display of the network-based treatment effect, an influence assessment of deviating direct evidence is possible. We discuss this tool for consistency and influence assessment in contrast to the net heat plot  by using an evidence network of twelve antidepressants. We assess controlled perturbations in a constructed, consistent dataset of the example and subsequently in the real data.
This article is structured as follows: We start with the data example. In the Methods section, we discuss the influence of direct evidence in network meta-analysis and derive an influence function as well as the concept of a decomposable subnet approximation and its visualisation. In the Results section, we apply our approach to the data and compare it with net heat plot results. The paper concludes with a discussion of our findings.
Twelve antidepressants examined in the network meta-analysis published by Cipriani et al. 
In the following, we briefly present a fixed effects model for network meta-analysis that has already been explained in more detail by Krahn et al.  and in this context analyse how changes in study effects are translated into network estimates. We introduce a simplification of meta-analysis networks into decomposable subnets that allows the application of a forest plot for a single network-based treatment effect.
Fixed effects model in network meta-analysis
The design matrix X comprises one column for each treatment A1,…,A T and one row for each design and each comparison with a design-specific reference (i.e. rows). If included in a design, this design-specific reference equals A0. The entries of X have value 1 in the column corresponding to the treatment compared with the reference, -1 in the column corresponding to a design-specific reference whenever it is not A0, and 0 else. The vector of all error terms ε is assumed to be independent across designs and normally distributed with covariance matrix V.
where Ht u,. denotes one row vector, and ht u,d denotes one entry (or in the case that d is a design of one or more N d -armed studies, a transposed partial row vector of length (N d −1)) of H.
Influence of single studies or designs in network meta-analysis
In classical meta-analysis, a study with a highly deviating treatment effect estimate may have a strong influence on the aggregated effect estimate depending on its weighting in the estimation process. Using generalised least squares estimation, the weight is proportional to the inverse variance of the treatment effect estimate.
In the case of multi-armed designs, influence functions for each pairwise comparison can be defined similarly.
Net heat plot
To assess consistency and influence in a classical meta-analysis, study weights and deviations between study effects and the aggregated treatment effect are, for example, visualised in a forest plot. For a similar analysis in network meta-analysis, a so-called net heat plot has been proposed . In a matrix visualisation (see examples in the Results section), the contribution of the aggregated direct evidence of each design (in a column) to each network estimate (in a row) is shown by the area of gray squares. The greater the area of a square, the greater the contribution of the respective direct evidence to the network estimate in the row. In combination with this evidence flow, a heat matrix for assessing the inconsistency in the network (quantified by a generalized Cochran’s Q statistic) is shown: The colours on the diagonal represent the inconsistency contribution, the summand of the Cochran’s Q statistic, of the corresponding design. The colours on the off-diagonal are associated with the change in inconsistency between direct and indirect evidence in a network estimate in the row after relaxing the consistency assumption for the effect of one design in the column. Blue colours indicate an increase and warm colours a decrease. In the case that the colour vector of a column is equal to the colour vector on the diagonal, the detachment of the respective design resolves the inconsistency in the whole network.
The detaching of single designs is similar to the node splitting technique of Dias et al. , but the net heat plot approach additionally tracks the influence of each design on the fit on all other designs and visualises the successive detaching of each designs in one plot. In contrast, the approach to the consistency assessment presented in the following allows for the application of a forest plot for one specific treatment comparison.
For the assessment of consistency and influence in one network-based comparison, we use a simplification of the network into a decomposable subnet. Firstly, we will describe our approach for networks formed only by two-armed studies and we will refer to the case of multi-armed studies in Section ‘Multi-armed designs’.
A treatment effect estimate between treatments t and u is most easily assessable if the underlying network is entirely composed of independent paths between the treatment nodes u and t (see also ). Two paths are defined as independent if they do not share any edges. Each path-based estimate can be formulated by the sum over the intermediate effect estimates. For example, the indirect estimate for treatment effect u versus t via treatment v can be calculated by . Path based estimates of independent paths are uncorrelated because they are based on independent evidence. The variance is obtained by summing up the direct estimate variances as described by Bucher et al. . All path-based estimates are combined as a weighted sum with weights proportional to their inverse variances. For assessing the consistency, we display this aggregation step in a forest plot where each row represents a path-based effect estimate and each path is identified by its unique list of intermediate treatments.
If the given network is not decomposable as described above, we consider the set of all decomposable subnetworks and choose one with the most precise resulting effect estimate. Although challenging in general, it is often easy to find a well approximating decomposable subnet just by selecting all paths containing no or only one intermediate treatment. Note that in a network represented by a complete graph with direct effect estimates of constant precision, the best approximating decomposable subnet (i.e. with the most precise resulting effect estimate) is uniquely defined by the direct path and all two-step indirect paths . The resulting estimate is identical to the network-based estimate.
which can be used to straightforwardly define a Z statistic to test for consistency between the approximation and the network-based estimate (by analogy with e.g. ).
We use the path-based effect estimates and the pseudo estimate together with their precisions to hierarchically set up a forest plot that captures how the subnetwork is composed of independent path-based estimates and, at a higher level, how the subnet contributes to the whole network-based estimate (see examples in the Results section). The resulting overall Cochran‘s Q statistic has degrees of freedom (df) equal to the number of independent paths. It is similar but not identical to the inconsistency Q. It captures solely that aspects of inconsistency that have consequences for the uncertainty about the effect estimate The weights in the forest plot can be interpreted as the proportion at which a perturbation in any direct estimate contributing to that path is translated into a bias of the network estimate If the weight of the residual evidence is small, no perturbation outside the subnet at any plausible scale is able to substantially affect the network estimate.
Iterated shortest path algorithm
Start selecting the shortest path between nodes t and u and eliminate all its edges .
Iterate until the nodes t and u are no longer connected.
The set of all eventually selected paths make up the approximating sub-network and its independent path decomposition.
Different strategies can be used for multi-armed designs: Firstly, they can be kept out of all paths of a subnetwork of independent paths. Secondly, one convenient treatment comparison for each multi-armed design, for example, the comparison between A and B for the multi-armed design ABC, can be allowed to potentially contribute in combination with all two-armed studies of design AB. This implicitly assumes that direct evidence for the relative effect θ tu does not depend on the remaining treatments investigated in the same study. A further strategy is to use at most one comparison (e.g. AB) of each multi-armed design separated from all two-armed studies (of design AB) and to build extra paths with these comparisons if possible. For the examples discussed in the following, we use the first strategy.
Construction of validation datasets
We demonstrate the influence analysis as well as the approach of the independent path decomposition and its visualisation by applying the methods to the antidepressants example. The analysis capability by the independent path decomposition is discussed in comparison to the net heat plot approach using controlled perturbations in the consistent validation dataset as well as in the real data. To produce the forest plots and the net heat plots, we used functions of the R packages meta and netmeta. Software instructions and an R function for the iterated shortest path algorithm are available on the website http://www.unimedizin-mainz.de/imbei/biometrie/software.html.
The contribution of all design-specific direct evidence to the network estimates in the antidepressants example is visualised in Figure 2a. The areas of the gray squares are determined by the elements of the projection matrix H of the underlying model (as well as in ). As seen from the column of design fluo:venl for example, the corresponding direct estimate represents a large source of evidence as it drives the network estimates for many treatment contrasts. In particular, (as visualised by the area of the gray square in the diagonal element fluo:venl) the network estimate for fluo:venl is based on 49% direct evidence of 11 studies with inverse variance weights between 2.1% and 16.3%. The contribution of single studies to the network estimate is therefore beween 1% and 8%.
Figure 2b shows the influence of the direct evidence of design fluo:venl on three exemplarily chosen network estimates that could be important if the evidence of design fluo:venl deviates from the assumption of consistency. Here, the relative change in the network estimate against the change in the direct estimate is plotted. The functions are linear on a log scale with slope determined by the H matrix entries. For instance, doubling the OR of the direct estimate with design fluo:venl (, with δ= log(2)) changes the three network estimates shown by 8%, 26%, and 40%. In the constructed, consistent data example (see Section ‘Construction of validation datasets’), this means that a directly estimated OR in design fluo:venl of 2.00 [1.67;2.39] instead of 1.00 [0.84;1.2] shifts the network estimates for comparisons fluo:paro, sert:venl, fluo:venl from 1.00 [0.89;1.12], 1.00 [0.86;1.17], and 1.00 [0.88;1.13] to 1.08 [0.96;1.21], 1.26 [1.08;1.47], and 1.41 [1.24;1.59]. Such a deviation of the direct evidence of design fluo:venl from the assumption of consistency would bias the network estimates of comparisons sert:venl and fluo:venl to at 5% level significant treatment effects.
Approximation by independent path decomposition
This display shows fairly consistent independent path-based estimates with exception of the indirectly estimated treatment effect via fluv with OR 2.8 [1.01;7.77]. Since the contribution of this estimate is 2.3%, the network estimate is only influenced a little. The pseudo estimate of the residual evidence is in accordance with the approximation, and the Q statistic of 10.46 (df=9, p=0.31) between the different sources of evidence indicates only slight inconsistency.
We generated approximations for all 66 pairwise treatment comparisons in the network. The median approximating evidence proportion was 85% (with range of 55-97%).
Validation of the sensitivity to perturbations in contrast to the net heat plot approach
On the right side of Figure 4a, the forest plot for the network estimate of comparison sert:venl based on the independent path decomposition of Figure 3a is shown. Since we constructed the dataset by setting the OR of all studies to one (with exception of the OR of studies with design fluo:venl in this perturbed case), the network estimate for comparison sert:venl, shown by the diamond on the bottom of the forest plot, should be equal to one. But it can be seen that the indirect estimate via fluo (OR: ) as well as the pseudo effect estimate of the residual evidence is affected by the perturbed direct treatment effect estimate of design fluo:venl. This, in turn, influences the network estimate of comparison sert:venl, since the indirect estimate via fluo comprises 26% of the estimate.In perturbation setting two, when we perturbed not only the direct treatment effect of design fluo:venl, but also of design dulo:esci to the same extent, we can observe the largest inconsistency contributions in the net heat plot in Figure 4b for these designs as well as for design dulo:paro. The blue-coloured elements in the upper left corner indicate that the direct effect of design fluo:venl and that of dulo:esci (or alternatively dulo:paro) support each other. This means the detachment of one of both designs from the network estimation increases the residual of the other design and the inconsistency in the network can be eliminated in neither of the detachments. Since the adjacent edges corresponding to designs dulo:esci and dulo:paro are part of an essentially non-branching path, the residuals resulting from the detachment of one of both designs are highly correlated, and the two corresponding columns contains very similarly coloured elements. Because the direct evidence of design dulo:esci drives only a little of the network estimate of comparison sert:venl, which can be seen by the little gray square in column dulo:esci and row sert:venl, the additional perturbation of the effect of design dulo:esci hardly changes the network estimate of comparison sert:venl.
This is duly reflected in the accompanying forest plot. Due to the small contribution of the direct evidence of design dulo:esci to the network estimate of comparison sert:venl, the edge corresponding to design dulo:esci is not part of the subnet that approximates the network estimate of comparison sert:venl, and the perturbation is only slightly recognisable in the forest plot by the small change of the residual evidence.
In perturbation setting three, in addition to the direct treatment effect of design fluo:venl, we perturbed the effect of design fluo:sert by inflating the OR by a factor of two. An indirect effect estimate of comparison sert:venl via fluo therefore also results in an OR of 1 as in the unperturbed case (). The net heat plot in Figure 4c indicates that both perturbations take effect in the same direction (the direct effects of both designs support each other, as shown by the blue-coloured elements), whereas in the forest plot only a small change in the pseudo effect estimate of the residual evidence is recognisable, and the indirect estimate via fluo is not affected at all. Thus, the forest plot clearly reveals that both perturbations taken together are not relevant for the comparison sert:venl.
Discussion and conclusions
In network meta-analysis, evidence from different designs contributes to a comparison between two treatments. For the consistency assessment between the effects of different evidence sources, we have provided a visualisation of an approximating independent path decomposition by forest plots and have investigated its performance in comparison to that of the net heat plot. We have shown for the example of the highly interconnected network of twelve antidepressants that most network-based treatment comparisons are approximated well by the evidence of independent paths. The proposed forest plot discloses and summarises the essentials of a given network-based treatment comparison: the weight given to each path, the consistency between all paths, the comparison between the estimate based on the approximation and on the whole network, and the residual evidence that is not included in the approximation is condensed into a pseudo estimate. The heterogeneity reported in this forest plot captures these aspects of the inconsistency, which are crucial for the considered comparison.
By perturbing the antidepressants network, after making it artificially perfectly consistent, we have shown how different kinds of perturbation show up as drivers of inconsistency in the net heat plot or as outlying path-specific evidence in the forest plot. While the net heat plot is sensitive to all kinds of perturbation that sufficiently inflate Cochran’s Q statistic for inconsistency, the forest plot indeed revealed to be selectively sensitive to perturbations that are influential to the considered comparison.
Independent path approximations should be considered only if they capture, say, more than 80% of the network evidence. If this is not the case, the complete network should be inspected. The flow of evidence can be displayed as a graph as outlined in , and influential designs can be sought for in the net heat plot. A forest plot that exhibits consistent and balanced evidence from several independent paths and that is additionally in accordance with the residual evidence may indeed be more convincing than a simple meta-analysis of direct comparisons (as similarly argued in ).
Paths independence can be defined in two ways, edge independence and vertex independence. In both definitions, the path-based effect estimates are uncorrelated. We have applied edge independence. Note that in an edge independent path decomposition, the estimate based on the subnet may be different from the estimate based on the independent paths if some paths share a common intermediate vertex. Two paths may contain some common intermediate vertices, and the forest plot implicitly splits these nodes. As a consequence, the subnet has more inconsistency degrees of freedom  than the corresponding forest plot. For the antidepressants network, on average only 2% of precision is lost by the approximation based on independent paths instead of the subnet.
We focused on the fixed effects model in our exposition, because inconsistency is most easily detectable then , but note that the proposed forest plot can be adapted to random effects models by assuming that the heterogeneity variance is known and fixed and by adding appropriate elements to the covariance matrix of direct effect estimates. The heterogeneity variance parameters would have to be assessed using the study level data (see e.g.  for various options to model the heterogeneity covariance structure). Then a forest plot is set up for the resulting independent paths formally using the fixed effects option but based on increased variances of the direct effect estimates. Random effects parameters can be estimated e.g. via the method of moments, by restricted maximum likelihood or by Bayesian methods. If the resulting point estimates of heterogeneity variance parameters are plugged in, our diagnostic tools are valuable for all these approaches.
In the literature, various methods have been discussed for the assessment of outliers and influence. In principle, all methods introduced for linear regression [22, 23] can be used [2, 9]. For network meta-analysis, plots of deviance residuals  have been discussed  as well plots of squared Pearson residuals . At an aggregate level, regression diagnostics amount to analysis of inconsistency, with index plots of leverages and residuals , concepts like node-splitting , and design-by-treatment interaction [12, 13]. None of these approaches focuses on the meaning of inconsistency-generating evidence for a specific treatment comparison and none offers all the analytical capacities known from the forest plot in classical meta-analysis for visualising the composition of evidence and identifying potential discrepancies.
Our proposed methods are confined to comparisons that are well-approximated by an estimate based on an independent path decomposition. In complex networks, comparisons that do not fit into this scheme may well be assessed by defining an approximating subnet that is more complex than a decomposable subnet, but much less complex than the whole network. In the antidepressants example more than half of all edges have weights less than 1/44, which is one fourth of the weights seen in a complete and balanced network of 12 treatments. Only a few percentages loss in precision should result from omitting these edges. Resorting to a subnet estimate if it captures more than 95% of the evidence for one comparison could be a remedy to the model uncertainty, e.g. with regard to the network size that was discussed in .
In conclusion, we have introduced forest plots of independent path decompositions for the assessment of consistency in complex meta-analytic networks as well. We have seen that this graphical presentation known from classical meta-analysis captures the essentials of a network-based treatment comparison and discloses both the composition of evidence and sources of potential inconsistency relevant for the considered comparison.
We thank Katherine Taylor for proofreading.
- 11.Cipriani A, Furukawa TA, Salanti G, Geddes JR, Higgins JP, Churchill R, Watanabe N, Nakagawa A, Omori IM, McGuire H, Tansella M, Barbui C: Comparative efficacy and acceptability of 12 new-generation antidepressants: a multiple-treatments meta-analysis. Lancet. 2009, 373 (9665): 746-758. 10.1016/S0140-6736(09)60046-5.CrossRefPubMedGoogle Scholar
- 22.Cook RD, Weisberg S: Residuals and Influence in Regression. 1982, New York: Chapman and Hall,Google Scholar
- 23.Belsley DA, Kuh E, Welsch RE: Regression Diagnostics: Identifying Influential Data and Sources of Collinearity. 2004, New Jersey: John Wiley & Sons,Google Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/14/131/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.