1 Claims and Counter-Claims
A growing amount of literature claims that multicriteria decision analysis (MCDA) is superior to economic evaluation in health technology assessment (HTA). The main arguments are that (1) MCDA includes a comprehensive and explicit list of value criteria not captured by the traditional methods of economic evaluation; (2) since it allocates quantitative weights to the different evaluation criteria, their relative importance is incorporated explicitly in the evaluation, thus making values and elicited preferences more consistent and transparent; and (3) the participation of all agents involved in assessing the value of alternatives increases the legitimacy of the process [1,2,3,4,5,6,7,8,9,10,11,12,13,14]. Arguments against MCDA include the inadequate treatment of opportunity cost, its vulnerability to double counting, and the fallacious attribution of deficiencies to methods of economic evaluation (EA) like cost-effectiveness analysis (CEA) in HTA [13,14,15,16,17,18,19,20,21,22,23,24]. We do not consider further the third argument for MCDA, because such participation seems to be a standard element of good practice in CEA [25,26,27,28,29,30,31,32].
2 Cost and Scarcity
When health budgets are limited, the costs of providing additional health services are the benefits forgone to other patients as other services are displaced. Inclusion of opportunity cost requires that all benefits forgone (or potentially forgone when a new intervention is being assessed) be valued and included in the estimated incremental cost-effectiveness ratio (ICER) of the additional service. An efficient allocation of limited resources between alternative interventions cannot neglect opportunity cost. The multidimensionality of “value” provides no protection from this since opportunity cost includes multidimensional benefits forgone. For consistency, therefore, MCDA requires full consideration of the multidimensional attributes of all potentially displaced services as well as those of the additional intervention. This is an implication generally understood by CEA practitioners [33]. CEA analysts also recognise that efficiency is not the sole criterion for making healthcare investments, that there can be conflict between criteria, including cost-effectiveness, and that double counting can be a major distortion [8].
3 Double Counting in MCDA
Much MDCA considers cost as a criterion additional to cost-effectiveness. This amounts to a serious case of double counting and usually fails anyway to distinguish accounting cost from opportunity cost. For a procedure to be cost-effective, whatever other features it may have, it must offer a benefit larger than the sacrifice of benefit that its resource use entails in comparison with a comparator intervention or a threshold ICER. Whether it is affordable, given current budgets, thresholds and accounting conventions; whether it is fair; whether now is the right time; whether it has other relevant attributes: these are not themselves questions of cost-effectiveness.
4 Cost and Non-costs
Confusion between prices, expenditures, harms, accounting cost and opportunity cost arises amongst the plethora of components in MCDA. Opportunity cost represents the most valued alternative use of a resource (by assumption, one that is scarce). It is dependent on the context of a decision. For example, an ambulance service provider’s decision to buy a vehicle entails a transfer of assets (the price of the vehicle). This may be recorded variously as an expenditure or as an accounting cost. The opportunity cost of the decision to purchase is, however, the loss of value (depreciation) upon the change of ownership. By contrast a decision to buy and to operate the vehicle entails the depreciation over the relevant planned period of operation together with the present value of the recurrent labour and other variable opportunity costs. This labour also has an opportunity cost in the form of its most valued alternative. It may also be tedious (“blood, sweat and tears”). These “harms” are not the opportunity cost of any resource. They are negative impacts on the worker’s utility and should accordingly be deducted from the expected benefit—not added to its cost. Opportunity costs of resources are reflected in, or can be inferred from, prices in well-functioning markets, but are also sometimes entirely subjective—such as when a decision is made to adjust resources within an organization [34, 35]. Similarly, to determine the cost of the decision to buy and to operate, the intended period over which the ambulance service is to be provided must be planned, and if the opportunity cost is to be made explicit, this period too must be made explicit in any CEA (or MCDA) calculation. Opportunity cost cannot be simply read from accounts.
5 Further Double Counting in EVIDEM and Elsewhere
The widely used Evidence and Value: Impact of Decision Making (EVIDEM) Value Matrix [9] for MCDA incorporates the net expenditure on the intervention (i.e. the expenditure differential between it and a comparator), together with other medical and non-medical expenditures [10]. These expenditures are loosely referred to as “costs”. However, EVIDEM explicitly also considers that “cost-effectiveness is a composite measure of data considered in other criteria and does not comply with the non-redundancy design requirement of MCDA” [10]. It then adds two further criteria: “opportunity costs and financial feasibility through a budgeting exercise”. While opportunity cost (correctly understood) is indeed a relevant criterion of cost-effectiveness, to add to it accounting costs (expenditures) is not correct. Benefits and costs are not equivalent to “pros and cons”. Predicted expenditures against predicted budgets are, of course, important managerial considerations which may have impact on the size of the budget or the threshold ICER, but they are not relevant in calculating an ICER, and their inclusion entails both double counting and the insertion of an attribute that is irrelevant to the question of efficiency.
Double counting may also occur on the benefit side. EVIDEM includes four criteria representing value (“need for intervention”, “comparative outcomes of intervention”, “type of benefit of intervention” and “knowledge about intervention”). The imprecision of these categories invites overlaps. EVIDEM recommends a test criterion of “mutual independence”, but despite this, still includes clearly interdependent criteria like comparative outcomes, type of benefit and knowledge of intervention. Though few in number, applications of MCDA to drugs provide more examples of blurred definition, such as manufacturing complexity and public health interest, which overlap and lack mutual independence, as with level of research undertaken, knowledge of the intervention, level of uncertainty of effectiveness, survival and life-saving [9, 14, 36,37,38,39,40,41,42].
Angelis and Kanavos [3] propose the Advanced Value Framework, in which they add socio-economic impact to burden of disease, therapeutic impact, safety and innovation. Socio-economic impact is further built upon three intermediate criteria: public health (risk reduction and prevention), “direct” incremental costs (medical and non-medical) and “indirect” incremental costs (absenteeism, presentism, premature abandonment, premature mortality, and caregivers) [3]. There are clear risks of double (or repeated) counting and an evident lack of independence of criteria like social impact, reduced burden, therapeutic effects and safety effects.
MCDA proposes an adaptation of the ICER: the incremental cost-value ratio (ICVR), in which value is a multidimensional (net) benefit measure [3]. The ICVR, however, is afflicted with even more methodological problems than the conventional ICER. Angelis and Kanavos and others claim that MCDA is “holistic”, by which they seem to mean “more completely representative of patients’ preferences” [3, 9, 15]. This raises a major question as to the purposes of decision-making aids such as HTA. Healthcare is notorious for being a territory in which virtually none of the usual conditions for efficient resource allocation apply: the socio-economic gradient linking health and wealth (willingness to pay being correlated inversely with ill-health and disability); principal-agent imperfections (supplier-induced demand); asymmetrical information; ignorant and prejudiced medical judgments taken without regard to any evidence of benefit to the patient; irrational behaviour (whether by the professional or the patient); patient incompetence to decide through ignorance, youth or old age; externalities (physical and psychic) and publicness. In CEA, the source of value is a matter for decision makers to determine. Angelis and Kanavos go on to suggest that a multidimensional index of benefit based on patient preferences should then be compared with “purchasing costs”. It is hard to see what kind of welfare or extra-welfare theory could lead to such a conclusion.
MCDA advocates commonly assume that CEA analysts hold that cost-effectiveness (relative to some threshold value) is a sufficient condition for judging the desirability of investments. This is a straw man, for we know of none holding such a view. Cost-effectiveness is a necessary condition, but not a sufficient one. More pertinent would be the more sophisticated claim that, in order to inform decision makers better, CEA must resort to ad hoc additional criteria related to dimensions of benefit other than “health”, like feasibility, political acceptability, or severity of disease. Modern reference cases for CEA are not entirely ad hoc, however, since they seek both to provide information on additional criteria like distributive justice and protection from catastrophic financial out-of-pocket private expenditures (topics that do not appear in most MCDA listings) and to create decision-making processes that provide opportunities to test the construct validity of many decision variables, their adequacy and completeness, as well as practical matters of implementability and socio-political credibility and acceptability.
6 Other Criteria for Choice than Cost-Effectiveness
These are issues for both CEA and MCDA. For example, CEA researchers search for methods of bringing distributional concerns into cost-effectiveness [43,44,45,46,47]. It seems to us to be a substantial empirical question whether it is more helpful for decision makers to have considerations such as these embodied in a simple numerical decision rule (after some form of quantification and weighting) than it is for them to be laid out explicitly for consideration, with whatever qualitative and quantitative evidence that may be available, for a committee to consider and come to a view. Thus, in many low-income countries, the main issues regarding equity relate to protection from financial catastrophe and the relative treatment of people who live in high-cost remote regions of the country. Complex and sophisticated quality-adjusted life year (QALY) weights do not seem to be the most obvious way of helping decision makers to make a judgment on such matters. We conjecture that much apparent statistical precision is spurious and that the really important information can be lost in such statistical summaries—but that is something to be tested. Our conjecture is no more than that.
7 Conclusion
MCDA’s strong (procedural) points are already perfectly well adoptable (and indeed adopted) in CEA/HTA [19, 27, 28] and have been since before MCDA started to become fashionable, but the risks of double counting in MCDA; its advocates’ contempt for qualitative evidence; the way they confuse expenditure, opportunity cost and harm; and its lack of ready accessibility/transparency for the public and other non-participating stakeholder all make it an unsatisfactory vehicle for good decision making. But we are not the ultimate judges of that—the ultimate judges are accountable decision makers. We can only adduce evidence, provide logically consistent ways of thinking about major healthcare investments and what they are intended to achieve, and suggest reasonable procedures for the appropriate involvement of stakeholders.
References
Garatini L, Padula A. Multiple criteria decision analysis in health technology assessment for drugs: just another illusion? Appl Health Econ Health Policy. 2018;16:1–4.
Mühlbacher AC, Kaczynski A. Making good decisions in healthcare with multi-criteria decision analysis: the use, current research and future development of MCDA. Appl Health Econ Health Policy. 2016;14:29–40.
Angelis A, Kanavos P. Multiple criteria decision analysis (MCDA) for evaluating new medicines in health technology assessment and beyond: the advance value framework. Soc Sci Med. 2017;188:137–56.
Angelis A, Kanavos P. Value-based assessment of new medical technologies: towards a robust methodological framework for the application of multiple criteria decision analysis in the context of health technology assessment. Pharmacoeconomics. 2016;34:435–46.
Drummond MF, Wilson DA, Kanavos P, Ubel P, Rovira J. Assessing the economic challenges posed by orphan drugs. Int J Technol Assess Health Care. 2007;23:26–42.
Drummond M, Evans B, Lelorier J, Karakiewicz P, Martin D, Tugwell P, et al. Evidence and values: requirements for public reimbursement of drugs for rare diseases—a case study in oncology. Can J Clin Pharmacol. 2009;16:e273–81.
Drummond M, Tarricone R, Torbica A. Assessing the added value of health technologies: reconciling different perspectives. Value Health. 2013;16:S7–13.
Garau M, Devlin NJ. Using MCDA as a decision aid in health technology appraisal for coverage decisions: opportunities, challenges and unresolved questions. In: Marsh K, Goetghebeur M, Thokala P, Baltussen R, editors. Multi-criteria decision analysis to support healthcare decisions. Cham: Springer International Publishing; 2017.
Goetghebeur MM, Wagner M, Khoury H, Levitt RJ, Erickson LJ, Rindress D. Evidence and value: impact on decision making—the EVIDEM framework and potential applications. BMC Health Serv Res. 2008;8:270.
EVIDEM Collaboration. EVIDEM. The framework, 10th edn. Concept and definitions. 2017. https://www.evidem.org/evidem-framework.
Schey C, Krabbe PFM, Postma MJ, Connolly MP. Multi-criteria decision analysis (MCDA): testing a proposed MCDA framework for orphan drugs. Orphanet J Rare Dis. 2017;12:10.
Schlander M, Garattini S, Holm S, Kolominsky-Rabas P, Nord E, Persson U, et al. Incremental cost per quality-adjusted life year gained? The need for alternative methods to evaluate interventions for ultra-rare disorders. J Comp Eff Res. 2014;3:399–422.
Sussex J, Towse A, Devlin N. Operationalising value based pricing of medicines: a taxonomy of approaches. Pharmacoeconomics. 2013;31:1–10.
Sussex J, Rollet P, Garau M, Schmitt C, Kent A, Hutchings A. A pilot study of multicriteria decision analysis for valuing orphan medicines. Value Health. 2013;16:1163–9.
Thokala P, Duenas A. Multiple criteria decision analysis for health technology assessment. Value Health. 2012;15:1172–81.
Laupacis A. Evidence and values: requirements for public reimbursement of drugs for rare diseases—a case study in oncology. Can J Clin Pharmacol. 2009;16:2.
McCabe C, Tsuchiya A, Claxton K, Raftery J, et al. Assessing the economic challenges posed by orphan drugs: a comment on Drummond. Int J Technol Assess Health Care. 2007;23:397–404.
Claxton K, Culyer AJ. Wickedness of folly? The ethics of NICE’s decisions. J Med Ethics. 2006;32:373–7.
Culyer AJ, Lomas J. Deliberative processes and evidence-informed decision making in healthcare: do they work and how might we know? Evid Policy. 2006;2:357–71.
Culyer AJ. The humble economist. York: University of York, Office of Health Economics; 2012.
Culyer AJ. Where are the limits of cost-effectiveness analysis and health technology assessment? J Med Assoc Thai. 2014;97(supl 5):S1–2.
Culyer J. Cost-effectiveness thresholds in health care: a bookshelf guide to their meaning and use. Health Econ Policy Law. 2016;11:415–32.
Culyer AJ. Ethics, priorities and cancer. J Cancer Policy. 2017;11:6–11.
Campillo-Artero C, Peiró S. Rare diseases, orphan drugs: the value of orphanhood. Gest Clin Sanit. 2009;4:119–26.
Puig-Junoy J. Multicriteria decision analysis: what is it and what is it for? In: Hidalgo A, Oliva J, editors. Multicriteria decision analysis. Barcelona: Chiesi; 2018 (in press).
Campillo-Artero C. The application of MCDA to orphan drugs. In: Hidalgo A, Oliva J, editors. Multicriteria decision analysis. Barcelona: Chiesi; 2018 (in press).
Lomas J, Culyer AJ, McCutcheon C, McAuley L, Law S. Conceptualizing and combining evidence for health system guidance. Ottawa: Canadian Health Services Research Foundation; 2005. p. 38.
Culyer AJ. NICE’s use of cost-effectiveness as an exemplar of a deliberative process. Health Econ Policy Law. 2006;1:299–318.
Culyer AJ, Lomas J. Deliberative processes and evidence-informed decision-making in health care—do they work and how might we know? Evid Policy. 2006;2:357–71.
Culyer AJ. Deliberative Processes in decisions about health care technologies: combining different types of evidence, values, algorithms and people. London: Office of Health Economics; 2009. p. 1–20.
Culyer AJ. “HTA – algorithm or process? Comment on expanded HTA: enhancing fairness and legitimacy. Int J Health Pol Manag. 2016;5:1–5.
Wilkinson T, Claxton KP, Sculpher MJ, Revill P, Briggs AH, Teerawattananon Y, et al. The international decision support initiative reference case for economic evaluation: an aid to thought. Value Health. 2016;19:921–8.
Claxton K, Palmer S, Longworth L, Bojke L, Griffin S, McKenna C, et al. Informing a decision framework for when NICE should recommend the use of health technologies only in the context of an appropriately designed programme of evidence development. Health Technol Assess. 2012;16:1–323.
Culyer AJ. Opportunity cost. The dictionary of health economics. 3rd ed. Cheltenham: Edward Elgar; 2014.
Alchian AA. Costs and outputs. In: Baran PA, Scitovsky T, Shaw ES, editors. The allocation of resources. Stanford: Stanford University Press; 1959. p. 23–40.
Hughes-Wilson W, Palma A, Shuurman A, Simoens S. Paying for the orphan drug system: break or bend? It is time for a new evaluation system for payers in Europe to take account of new rare disease treatments? Orphanet J Rare Dis. 2012;7:74.
Hutchings A, Ethgen O, Schmitt C, Rollet P. Defining elements of value for rare disease treatments. Value Health. 2012;15:A1–256.
Iskrov G, Miteva-Katrandzhieva T, Stefanov R. Multi-criteria decision analysis for assessment and appraisal of orphan drugs. Front Public Health. 2016;4:214.
Garau M, Marsden G, Devlin N, Mazzanti NA, Profico A. Applying a multi-criteria decision analysis (MCDA) approach to elicit stakeholders’ preferences in Italy. The case of obinutuzumab for rituximab-refractory indolent non-hodgkin lymphoma (iNHL). London: Office of Health Economics; 2016.
Wagner M, Khoury H, Bennetts L, Berto P, Ehreth J, Badia X, et al. Appraising the holistic value of lenvatinib for radio-iodine refractory differentiated thyroid cancer: a multi-country study applying pragmatic MCDA. BMC Cancer. 2017;12:272.
Kolasa K, Zwolinski KM, Hermanowski T. Potential impact of the implementation of multiple-criteria decision analysis (MCDA) on the Polish pricing and reimbursement process of orphan drugs. Orphanet J Rare Dis. 2016;11:23.
Gilabert-Perramon A, Torrent-Farnell J, Catalan A, Prat A, Fontanet M, Puig-Peiró R, et al. Drug evaluation and decision making in Catalonia: development and validation of a methodological framework based on multi-criteria decision analysis (MCDA) for orphan drugs. Int J Technol Assess Health Care. 2017;33:111–20.
Asaria M, Griffin S, Kookson R. Distributional cost-effectiveness analysis: a tutorial. Med Decis Mak. 2016;36:8–19.
Cookson R, Drummond M, Weatherly H, et al. Explicit incorporation of equity considerations into economic evaluation of public health interventions. Health Econ Policy Law. 2009;4(Pt 2):231–45.
Johri M, Norheim O. Can cost-effectiveness analysis integrate concerns for equity? Systematic review. Int J Technol Assess Health Care. 2012;28:125–32.
Asaria M, Griffin S, Cookson R, Whyte S, Tappenden P. Distributional cost-effectiveness analysis of health care programmes–a methodological case study of the UK Bowel Cancer Screening Programme. Health Econ. 2015;24:742–54.
Rowen D, Brazier J, Mukuria C, Keetharuth A, Risa Hole A, Tsuchiya A, et al. Eliciting societal preferences for weighting QALYs for burden of illness and end of life. Med Decis Mak. 2016;36:210–22.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
Carlos Campillo-Artero, Jaume Puig-Junoy and Anthony J. Culyer have no conflict of interest and have not received any funding for this paper.
Funding
None declared.
Ethical approval
Not applicable.
Rights and permissions
About this article
Cite this article
Campillo-Artero, C., Puig-Junoy, J. & Culyer, A.J. Does MCDA Trump CEA?. Appl Health Econ Health Policy 16, 147–151 (2018). https://doi.org/10.1007/s40258-018-0373-y
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40258-018-0373-y