1 Claims and Counter-Claims

A growing amount of literature claims that multicriteria decision analysis (MCDA) is superior to economic evaluation in health technology assessment (HTA). The main arguments are that (1) MCDA includes a comprehensive and explicit list of value criteria not captured by the traditional methods of economic evaluation; (2) since it allocates quantitative weights to the different evaluation criteria, their relative importance is incorporated explicitly in the evaluation, thus making values and elicited preferences more consistent and transparent; and (3) the participation of all agents involved in assessing the value of alternatives increases the legitimacy of the process [1,2,3,4,5,6,7,8,9,10,11,12,13,14]. Arguments against MCDA include the inadequate treatment of opportunity cost, its vulnerability to double counting, and the fallacious attribution of deficiencies to methods of economic evaluation (EA) like cost-effectiveness analysis (CEA) in HTA [13,14,15,16,17,18,19,20,21,22,23,24]. We do not consider further the third argument for MCDA, because such participation seems to be a standard element of good practice in CEA [25,26,27,28,29,30,31,32].

2 Cost and Scarcity

When health budgets are limited, the costs of providing additional health services are the benefits forgone to other patients as other services are displaced. Inclusion of opportunity cost requires that all benefits forgone (or potentially forgone when a new intervention is being assessed) be valued and included in the estimated incremental cost-effectiveness ratio (ICER) of the additional service. An efficient allocation of limited resources between alternative interventions cannot neglect opportunity cost. The multidimensionality of “value” provides no protection from this since opportunity cost includes multidimensional benefits forgone. For consistency, therefore, MCDA requires full consideration of the multidimensional attributes of all potentially displaced services as well as those of the additional intervention. This is an implication generally understood by CEA practitioners [33]. CEA analysts also recognise that efficiency is not the sole criterion for making healthcare investments, that there can be conflict between criteria, including cost-effectiveness, and that double counting can be a major distortion [8].

3 Double Counting in MCDA

Much MDCA considers cost as a criterion additional to cost-effectiveness. This amounts to a serious case of double counting and usually fails anyway to distinguish accounting cost from opportunity cost. For a procedure to be cost-effective, whatever other features it may have, it must offer a benefit larger than the sacrifice of benefit that its resource use entails in comparison with a comparator intervention or a threshold ICER. Whether it is affordable, given current budgets, thresholds and accounting conventions; whether it is fair; whether now is the right time; whether it has other relevant attributes: these are not themselves questions of cost-effectiveness.

4 Cost and Non-costs

Confusion between prices, expenditures, harms, accounting cost and opportunity cost arises amongst the plethora of components in MCDA. Opportunity cost represents the most valued alternative use of a resource (by assumption, one that is scarce). It is dependent on the context of a decision. For example, an ambulance service provider’s decision to buy a vehicle entails a transfer of assets (the price of the vehicle). This may be recorded variously as an expenditure or as an accounting cost. The opportunity cost of the decision to purchase is, however, the loss of value (depreciation) upon the change of ownership. By contrast a decision to buy and to operate the vehicle entails the depreciation over the relevant planned period of operation together with the present value of the recurrent labour and other variable opportunity costs. This labour also has an opportunity cost in the form of its most valued alternative. It may also be tedious (“blood, sweat and tears”). These “harms” are not the opportunity cost of any resource. They are negative impacts on the worker’s utility and should accordingly be deducted from the expected benefit—not added to its cost. Opportunity costs of resources are reflected in, or can be inferred from, prices in well-functioning markets, but are also sometimes entirely subjective—such as when a decision is made to adjust resources within an organization [34, 35]. Similarly, to determine the cost of the decision to buy and to operate, the intended period over which the ambulance service is to be provided must be planned, and if the opportunity cost is to be made explicit, this period too must be made explicit in any CEA (or MCDA) calculation. Opportunity cost cannot be simply read from accounts.

5 Further Double Counting in EVIDEM and Elsewhere

The widely used Evidence and Value: Impact of Decision Making (EVIDEM) Value Matrix [9] for MCDA incorporates the net expenditure on the intervention (i.e. the expenditure differential between it and a comparator), together with other medical and non-medical expenditures [10]. These expenditures are loosely referred to as “costs”. However, EVIDEM explicitly also considers that “cost-effectiveness is a composite measure of data considered in other criteria and does not comply with the non-redundancy design requirement of MCDA” [10]. It then adds two further criteria: “opportunity costs and financial feasibility through a budgeting exercise”. While opportunity cost (correctly understood) is indeed a relevant criterion of cost-effectiveness, to add to it accounting costs (expenditures) is not correct. Benefits and costs are not equivalent to “pros and cons”. Predicted expenditures against predicted budgets are, of course, important managerial considerations which may have impact on the size of the budget or the threshold ICER, but they are not relevant in calculating an ICER, and their inclusion entails both double counting and the insertion of an attribute that is irrelevant to the question of efficiency.

Double counting may also occur on the benefit side. EVIDEM includes four criteria representing value (“need for intervention”, “comparative outcomes of intervention”, “type of benefit of intervention” and “knowledge about intervention”). The imprecision of these categories invites overlaps. EVIDEM recommends a test criterion of “mutual independence”, but despite this, still includes clearly interdependent criteria like comparative outcomes, type of benefit and knowledge of intervention. Though few in number, applications of MCDA to drugs provide more examples of blurred definition, such as manufacturing complexity and public health interest, which overlap and lack mutual independence, as with level of research undertaken, knowledge of the intervention, level of uncertainty of effectiveness, survival and life-saving [9, 14, 36,37,38,39,40,41,42].

Angelis and Kanavos [3] propose the Advanced Value Framework, in which they add socio-economic impact to burden of disease, therapeutic impact, safety and innovation. Socio-economic impact is further built upon three intermediate criteria: public health (risk reduction and prevention), “direct” incremental costs (medical and non-medical) and “indirect” incremental costs (absenteeism, presentism, premature abandonment, premature mortality, and caregivers) [3]. There are clear risks of double (or repeated) counting and an evident lack of independence of criteria like social impact, reduced burden, therapeutic effects and safety effects.

MCDA proposes an adaptation of the ICER: the incremental cost-value ratio (ICVR), in which value is a multidimensional (net) benefit measure [3]. The ICVR, however, is afflicted with even more methodological problems than the conventional ICER. Angelis and Kanavos and others claim that MCDA is “holistic”, by which they seem to mean “more completely representative of patients’ preferences” [3, 9, 15]. This raises a major question as to the purposes of decision-making aids such as HTA. Healthcare is notorious for being a territory in which virtually none of the usual conditions for efficient resource allocation apply: the socio-economic gradient linking health and wealth (willingness to pay being correlated inversely with ill-health and disability); principal-agent imperfections (supplier-induced demand); asymmetrical information; ignorant and prejudiced medical judgments taken without regard to any evidence of benefit to the patient; irrational behaviour (whether by the professional or the patient); patient incompetence to decide through ignorance, youth or old age; externalities (physical and psychic) and publicness. In CEA, the source of value is a matter for decision makers to determine. Angelis and Kanavos go on to suggest that a multidimensional index of benefit based on patient preferences should then be compared with “purchasing costs”. It is hard to see what kind of welfare or extra-welfare theory could lead to such a conclusion.

MCDA advocates commonly assume that CEA analysts hold that cost-effectiveness (relative to some threshold value) is a sufficient condition for judging the desirability of investments. This is a straw man, for we know of none holding such a view. Cost-effectiveness is a necessary condition, but not a sufficient one. More pertinent would be the more sophisticated claim that, in order to inform decision makers better, CEA must resort to ad hoc additional criteria related to dimensions of benefit other than “health”, like feasibility, political acceptability, or severity of disease. Modern reference cases for CEA are not entirely ad hoc, however, since they seek both to provide information on additional criteria like distributive justice and protection from catastrophic financial out-of-pocket private expenditures (topics that do not appear in most MCDA listings) and to create decision-making processes that provide opportunities to test the construct validity of many decision variables, their adequacy and completeness, as well as practical matters of implementability and socio-political credibility and acceptability.

6 Other Criteria for Choice than Cost-Effectiveness

These are issues for both CEA and MCDA. For example, CEA researchers search for methods of bringing distributional concerns into cost-effectiveness [43,44,45,46,47]. It seems to us to be a substantial empirical question whether it is more helpful for decision makers to have considerations such as these embodied in a simple numerical decision rule (after some form of quantification and weighting) than it is for them to be laid out explicitly for consideration, with whatever qualitative and quantitative evidence that may be available, for a committee to consider and come to a view. Thus, in many low-income countries, the main issues regarding equity relate to protection from financial catastrophe and the relative treatment of people who live in high-cost remote regions of the country. Complex and sophisticated quality-adjusted life year (QALY) weights do not seem to be the most obvious way of helping decision makers to make a judgment on such matters. We conjecture that much apparent statistical precision is spurious and that the really important information can be lost in such statistical summaries—but that is something to be tested. Our conjecture is no more than that.

7 Conclusion

MCDA’s strong (procedural) points are already perfectly well adoptable (and indeed adopted) in CEA/HTA [19, 27, 28] and have been since before MCDA started to become fashionable, but the risks of double counting in MCDA; its advocates’ contempt for qualitative evidence; the way they confuse expenditure, opportunity cost and harm; and its lack of ready accessibility/transparency for the public and other non-participating stakeholder all make it an unsatisfactory vehicle for good decision making. But we are not the ultimate judges of that—the ultimate judges are accountable decision makers. We can only adduce evidence, provide logically consistent ways of thinking about major healthcare investments and what they are intended to achieve, and suggest reasonable procedures for the appropriate involvement of stakeholders.