FormalPara Key Points for Decision Makers

Despite increased attention towards patient preferences (PP), the use of PP in medical product decision making remains limited and unstructured.

In this qualitative research, 143 individual interviews were conducted to characterize stakeholders’ attitudes, needs, and concerns towards PP in decision making along the medical product lifecycle.

To increase the use of PP, efforts are needed on three levels: (1) the cultural and educational level, via increasing acceptance and understanding among stakeholders about PP; (2) the methodological level, via increasing understanding of quality criteria of PP studies; and (3) the procedural level, via increasing understanding on how to integrate PP in current medical product decision making.

1 Introduction

The role of patient preferences (PP) is being increasingly explored in several decision-making contexts throughout the life cycle of drugs and medical devices (i.e., the medical product life cycle [MPLC]) [1,2,3]. Different definitions, conceptualizations, and categorizations for PP exist. The US Food and Drug Administration (FDA) adopted the following definition for ‘patient preference information’: “qualitative or quantitative assessments of the relative desirability or acceptability to patients of specified alternatives or choices among outcomes or other attributes that differ among alternative health interventions” [3]. Others have defined PP and methods to investigate them in a narrower manner by solely referring to ‘utilities’ and quantitative ‘elicitation’ methods, respectively [4,5,6,7,8,9]. In this study, the terms ‘PP’ and ‘preference methods’ are used to refer to both qualitative and quantitative PP and preference methods, according to the FDA definition [3].

Decision-making stakeholders interested in PP include regulatory agencies responsible for the marketing authorization of medical products, such as the European Medicines Agency (EMA) and the FDA, and health technology assessment (HTA) bodies and reimbursement agencies responsible for HTA and the reimbursement of medical products [2, 10,11,12]. In the regulatory context, PP have been argued to provide insights into what benefit-risk trade-offs patients accept [3, 5, 13,14,15,16,17,18,19,20,21,22,23]. On the HTA/reimbursement level, PP could provide information about patients’ preferred medical products and clinical outcomes [6, 24,25,26,27,28,29]. Outside the regulatory and reimbursement context, industry stakeholders are exploring how PP can inform priority setting, clinical trial design and analysis, and post-marketing risk assessments [3, 7, 15, 19, 21, 30].

Reflecting these interests, decision-making bodies have started assessing the value of preference methods; the EMA investigated patient benefit-risk trade-offs for treatment outcomes [31], and the FDA quantified PP for the treatment attributes of weight-loss devices. The FDA issued two guidance documents about PP: recommendations for collecting PP in the context of medical devices and draft guidance for collecting patient experience data, including PP, in the context of drugs [3, 17, 32,33,34,35]. Examples of efforts at the reimbursement level include the conduct of PP studies by the Institute for Quality and Efficiency in Health CareFootnote 1 and the first scientific advice on a PP study design from the National Institute for Health and Care ExcellenceFootnote 2 [29, 36, 37].

Despite demonstrated interest, PP studies are currently not submitted and evaluated systematically during marketing authorization [38]. Similarly, the use of PP in HTA has been described to be in its infancy, and few HTA organizations have guidelines on how PP should be measured [35, 39]. In addition, no evidence-based guidance on the design and conduct of PP studies is available for industry. As the adoption of PP depends on the stakeholders involved in the MPLC (academics, HTA bodies, payers, industry, patients, physicians, and regulators), knowledge of their perceptions of PP is a critical step towards increasing the use of PP. Moreover, as these stakeholders’ decisions depend on each other, implementing PP in the MPLC would impact each stakeholder involved. This study therefore aimed to characterize stakeholders’ perceptions of PP and, more specifically, their attitudes, needs, and concerns related to PP in decision making along the MPLC.

2 Materials and Methods

2.1 General Design

Because this study aimed to gain deeper insights into stakeholders’ attitudes, needs, and concerns, a qualitative study design was chosen. In view of the geographical dispersion, diversity, and number of interviewees within each stakeholder group, interviews were chosen as the data collection method [40]. Interviews were conducted as part of PREFERFootnote 3 between April and August 2017.

2.2 Interviewee Selection

Interviewees were recruited from six different stakeholder groups to account for the diversity of stakeholders involved in the MPLC. Interviewees were recruited from eight countries to obtain insights from stakeholders familiar with different healthcare systems. Patients, caregivers, physicians, and patient organization representatives were recruited from four disease areasFootnote 4 to increase the heterogeneity of the sample in terms of disease types. To further ensure a heterogeneous sample, inclusion criteria and quota were set (Electronic Supplementary Material [ESM] 1). An initial sample was established through suggestions from PREFER consortium members. From these suggestions, 22 persons met the inclusion criteria and were invited to participate via e-mail. Twelve persons (55%) accepted; the others did not answer, did not consider themselves knowledgeable enough, or did not have time. Subsequent interviewees were suggested by interviewed persons (snowballing). Recruitment was slowest in Romania and Italy and in the HTA and regulatory stakeholder group.Footnote 5 No relationship between interviewers and interviewees was established prior to invitation.

2.3 Interview Guide

The interview guide was developed based upon the research aims, 16 exploratory interviews, and a literature review (Table 1, ESM 2). Exploratory interviews were conducted with HTA/payer representatives (n = 4), regulators (n = 2), industry representatives (n = 3), patient organization representatives (n = 4), physicians (n = 2), and one academic (n = 1). The literature review identified the opportunities and challenges associated with PP (Janssens et al., submitted, 2019). One version of the interview guide was developed for HTA/payer representatives, regulators, industry representatives, and academics and one version for patients, patient organization representatives, caregivers, and physicians. The latter version was shorter, used simpler terminology, included more comprehensive information parts, and did not address the more methodological topics (Table 1). Both versions were reviewed and pilot tested with five interviewees of the targeted stakeholder groups.

Table 1 Topics addressed in the interview
Full size table

2.4 Data Collection

An interview protocol was developed and discussed with eight interviewers (RJ, SR, EvO, CW, KSB, AC, ME, RD)Footnote 6 as a form of training for conducting the interviews and to minimize variability across interviews (ESM 3). A glossary was sent to patients, patient organization representatives, caregivers, and physicians prior to the interview (ESM 4). At the start of the semi-structuredFootnote 7 interview, interviewers presented themselves and explained the aim of the PREFER project and interview. Interviews with patients, patient organization representatives, caregivers, and physicians were conducted in their native language. Other interviews were conducted in English unless preferred otherwise by the interviewee. Interviews took approximately 1 hour and were conducted via telephone or face to face, the latter at a location preferred by the interviewee. In addition to the interviewer and interviewee, no one was present during the interview. Interviews were audio-recorded. One interview was conducted per interviewee.Footnote 8

2.5 Data Analysis

Interviews were analyzed thematically by two researchers (RJ, SR). The framework method was chosen for analysis, as it enables multiple researchers to independently analyse one large dataset [41, 42]. Table 2 explains the implementation of the seven stages of the framework method: transcription, familiarization, coding, developing an analytical framework, applying the analytical framework, charting, and interpreting. Open answers to the question asking about interviewees’ familiarity with PP were categorized into “not familiar”, “moderately familiar”, or “very familiar” and were analyzed descriptively in Excel.

Table 2 Stages of the framework method
Full size table

3 Results

The results section is structured according to the themes identified during analysis. Key findings per stakeholder group are visualized in Fig. 1.

Fig. 1
figure 1

Stakeholders’ perceptions about patient preferences (PP). Left: interviewees’ self-reported familiarity with PP studies and methods and how they defined PP. Right: stakeholders’ key needs and concerns related to PP in decision making along the medical product life cycle. The patient group includes patients, caregivers, and patient organization representatives

Full size image

3.1 Interviewees’ Characteristics

One hundred and forty-three persons participated. Interviewees reported being unfamiliar (43%), moderately familiar (42%), or very familiar (15%) with preference methods and studies (Table 3). Self-reported familiarity was highest in academics, among whom 50% reported being very familiar, and it was the lowest in patients and physicians, among whom 86% and 75%, respectively, reported being unfamiliar with PP (ESM 7).

Table 3 Interviewees’ characteristics
Full size table

3.2 Definition of Patient Preferences (PP) and Awareness

3.2.1 Defining ‘PP’

When interviewees were asked about their personal definition for PP, interviewees from all stakeholder groups underlined the challenge of providing such a definition. Whereas academics more often used a focused definition (e.g., trade-offs between treatment outcomes), interviewees from the other stakeholder groups most frequently adopted a broad definition (e.g., patients’ perspectives) (Fig. 1). An academic and regulator explained how the lack of an agreed-upon definition causes confusion: “Well, I think it is a term that is being used in a number of different ways, and I think it would be beneficial if a definition can be agreed upon, just to reduce confusion” (RE_US_6).Footnote 9

3.2.2 Awareness and Acceptance

Interviewees from all stakeholders mentioned a need for higher understanding of the concept of PP and preference methods. In this context, some academics and regulators called for training opportunities for decision makers and researchers. Several patient organization representatives, academics, and industry representatives mentioned that more acceptance of the concept of PP is needed: “A kind of general alignment of what patient preferences are, why they are important and why they need to be included in the drug development pathway” (AC_IT_24).

3.3 Attitudes Towards PP, Their Value and Impact on Decision Making

3.3.1 Interviewees’ Attitudes Towards PP

The majority of interviewees across stakeholder groups were positive towards using PP in the MPLC, as patients are final healthcare users, have disease experience, and the use of PP could improve decision making (see Sect. 3.3.3). Whereas industry interviewees seemed most positive towards PP, physicians most frequently were hesitant towards using PP or against using PP. Interviewees who doubted the value and use of PP made the following arguments:

  • Patients lack scientific knowledge (AC, IN, PA, CA, PH, RE; see Sect. 3.4.2);

  • Patients have unrealistic preferences and are not objective enough (PH);

  • Preference methods are of low quality (PY, RE, AC, see Sect. 3.4.5);

  • PP do not fit in current decision-making processes (PH, AC, PY, see Sect. 3.5.1);

  • Using PP would not alter the decision (PH, AC, see Sect. 3.3.2); and

  • In the context of reimbursement, societal preferences rather than patient preferences should be used (PY, see Sect. 3.4.3) (Fig. 1).

3.3.2 Value of PP is Context Dependent

Interviewees from all stakeholder groups, particularly regulators, underlined that the value of PP in decision making depends on factors such as the type of medical product and disease area (Fig. 1). Furthermore, some academics and regulators stated that PP studies only need to be conducted when they are relevant for decision making.

3.3.3 Impact of PP on Decision Making

There were diverging views regarding the impact of PP on decisions. Interviewees from different stakeholder groups (AC, PY, PA, IN) underlined that it would take a long time before PP would become part of the regular decision-making system because of the complexity of doing so (see Sect. 3.5.1).

Regarding development, interviewees from different stakeholder groups described the impact as follows:

  • Better industry investments (PA, PY, RE);

  • Enhanced patient recruitment into clinical trials (PA);

  • Increased development of medical products that meet patient needs (AC, PY, IN, PH, RE, PO);

  • Easier or faster market access (RE, IN); and

  • Higher acceptance of the drug among patients (IN, PH, PO, RE, AC).

Regarding the impact on marketing authorization and HTA/reimbursement, some HTA/payers, industry representatives, and regulators predicted the following:

  • A higher external acceptance of decisions (IN, RE);

  • A positive change in the decision makers’ approach to benefit-risk assessments (RE): “A dramatic change in the concern of regulators […], it would become normal when discussing the benefits and risks” (RE_UK_11);

  • A price increase (PY): “If medical products have attributes that patients value, then presumably manufacturers will charge more for those products” (PY_SE_14);

  • Higher quality decisions (RE, PY, IN) by increasing certainty for decision makers when there are uncertainties regarding the clinical evidence; and

  • More decisions are made (PY): “Now some decisions […] just can’t be made simply because they don’t have the proper data” (PY_DE_8).

Regarding the impact of PP on decision outcomes, different opinions were expressed. With respect to marketing authorization, several interviewees (IN, AC, PY, RE) foresaw a marginal effect: “If you already approve 90% of all treatments, then you cannot really make that much of an impact” (AC_NL_12). One regulator explained that specifically in cases in which there is a clear benefit-risk balance, the decision outcome would not necessarily change if PP were considered. On the HTA/reimbursement level, some HTA/payers and industry representatives stated that the use of PP would lead to different decisions being made “due to the system we have here currently (…) it says that only patient relevant endpoints are considered. However, patients were never asked what patient-relevant end-points are” (IN_DE_10).

3.4 Concerns and Needs Regarding Measuring PP for Decision Making

3.4.1 Guiding Framework for Measuring PP

Several academics and industry representatives expressed concerns about a lack of consensus on how to measure PP and, more specifically, how to select the following: (1) a sample (see Sect. 3.4.3); (2) a preference method; (3) attributes and levels; and (4) the stage in the MPLC to conduct a preference study. Several of them elaborated on how more standardization, guidance, and best practices would demonstrate the manner in which PP studies should be undertaken. Some of them argued that standardization could be achieved through regulatory guidance, a change in legislation, or a clearer regulatory framework: “Some rules, clearly defined, telling you in what process you involve the patients, how representative they should be (…) and so on” (AC_RO_22). Some industry interviewees explained that a regulatory framework would provide them with more certainty to invest in a preference study: “Every study needs investment when the clinical development of a drug is a costly process, where we have to make choices” (IN_IT_18). Some academics underlined, however, the difficulty of formalizing preference research and argued that developing strict guidance would stop preference research: “It is very risky to have to sit down and write rules about how to do things involving research” (AC_US_12). Similarly, there were mixed views on the level of detail this guidance should have; whereas one academic highlighted that existing guidelines “are probably too basic” (AC_UK_21), another academic and industry representative elaborated that this guidance should be flexible so that “according to the specific context, you are free to choose which one (method) is most suitable” (AC_IT_21). Several academics and HTA/payer interviewees particularly described how a checklist of quality criteria would be useful, and some academics mentioned that more research on how to rate the quality of PP studies and their outcomes would help in the construction of such a checklist.

3.4.2 Patient Knowledge and Education

Interviewees from all stakeholder groups described the current level of patient knowledge as a concern for the measurement and use of PP in decision making. One patient attributed the lack of knowledge to the lack of information given to patients about the preference study. In addition to knowledge about the preference study, interviewees from all stakeholder groups also referred to patients’ educational level and knowledge about the MPLC and about the medical product. To increase patient understanding, some patients advocated for the inclusion of an educational component in PP studies. Interviewees from all stakeholder groups were concerned about the complexity of questions in PP studies. Some regulators and academics underlined how a lack of understanding among participants decreases the reliability of the preference study results and advocated for more research verifying whether patients actually understood the questions.

3.4.3 Sample

Interviewees from all stakeholder groups, except patients and physicians, raised a need for clarifying what would be the “ideal” sample in terms of the size, heterogeneity, and type of participants that would ensure the usefulness of the study results for decision making: “Should it be European, Nordic, American, Asian? (…) I mean, there are a number of questions like these that you have to sit down and decide” (IN_SE_17). Some academics and HTA/payer representatives specifically raised the question of using societal vs. patient preferences to guide reimbursement decisions: “I think I personally would have a preference for doing public so - I feel like preferences are important, it’s just whose preferences that I’m worried about” (UK_AC_21). The concept of ‘heterogeneity’, both of the patient sample and of the measured preferences, triggered diverging views across academics, regulators, HTA/reimbursement, and industry representatives. Some of them considered heterogeneity to be a desirable element. Others were concerned about preference heterogeneity being present but not being measured or reported transparently. Finally, some considered heterogeneity to be both negative and positive: “It’s true; we have to do a better job, we have to explain heterogeneity, I think it will be positive in the future because that could actually lead to a more personalised approach to treatments” (AC_DE_11).

3.4.4 Factors Influencing PP Study Results

Several academics, physicians, and regulators described factors that may influence PP: sociocultural status, emotions, time, media, psychological factors, and disease status. They reasoned that the impact of these factors on PP study results is often unknown to researchers. One regulator was concerned about how the sole act of measuring PP might impact preferences of participants: “If I obtain the patient’s preference, the problem is understanding what I have really obtained (…) I have also directed this preference involuntary” (RE_IT_11). Interviewees across all stakeholder groups were apprehensive about methodological choices in the design of PP studies (e.g., question framing, selecting certain patient groups), leading to biased results, and several mentioned the risk that pharmaceutical companies would want to use these results to “persuade the regulator of something that is actually quite potentially dangerous” (RE_US_6).

3.4.5 Scientifically Robust Preference Methods

Interviewees from all stakeholder groups except patients, caregivers, and patient organization representatives indicated a current lack of methodological understanding related to the available methods for measuring PP. Some of them considered the preference research field to be “not a very well-developed area of research” (UK_PY_15). More specifically, interviewees across these stakeholder groups expressed doubts about the validity, reliability, reproducibility, and generalizability of preference methods and results. Some of these interviewees reasoned that not knowing how preference methods perform on these criteria lowers their usefulness for decision making: “We know little about the reliability, the reproducibility, etc. (…) It (preference research) has little weight on policy decision making because of those reasons” (AC_FR_11).

Several academics, regulators, and HTA/payer and industry representatives stressed the need for more research that validates different preference methods. However, only a few suggested specific study designs to better characterize preference methods: “Different types of methods in the same patient population and this type of information I think we need” (RE_NL_6). Furthermore, while methods need to be “scientifically credible” (IN_DE_9), some industry and academic interviewees raised a need for simpler methods because (1) “many of these questions come up on a very time critical path” and (2) complex methods are “cognitively burdensome” (AC_IT_21).

3.4.6 Multi-Stakeholder Approach

Some patient organization representatives and regulators underlined how relevant stakeholder groups (e.g., sponsors, patients, caregivers, regulatory agencies) need to be involved in the design and conduct of PP studies to increase their value for decision making. For example, collaboration between researchers and regulators was described as useful because scientists have preference research expertise and regulators can “determine whether PP information would be useful to answer certain regulatory decision questions” (US_RE_5).

3.5 Concerns and Needs Regarding the Use of PP in Decision Making

3.5.1 Integrating PP in Decision Making

Interviewees from all stakeholder groups except patients, caregivers, and patient organization representatives questioned the feasibility and lack of a clear strategy for including PP in current development, marketing authorization, and reimbursement decisions. Several HTA/payers, industry representatives, and academics questioned how PP should be weighted in decisions. These stakeholder groups but also particularly regulators underlined that PP should be considered together with other evidence. Similarly, regulators and physicians stated that PP should not completely guide the decision, as they perceived other “traditional” criteria (e.g., efficacy, safety) to be more important. To be useful for marketing authorization, one US regulator underlined that regulators need to be able to link the preference study results to the clinical trial outcomes of the medical product under review. Similarly, one HTA/payer representative raised the need for a “consistent approach of measuring patient preferences in different indications so that they can be very directly translatable to decision making” (HT_NL_8).

3.5.2 Framework for Using PP

There were diverging views regarding what specifically is needed to integrate PP into decision making. While several academics, regulators, payers, and patient organization interviewees felt that PP should be integrated based on a framework that allows for the transparent use of PP in decision making, another academic argued to “just use this as a source of information, and they will come up with decisions” (AC_DE_11).

3.5.3 Handling Preference Study Results

Several regulators, academics, and physicians emphasized that the submission of PP studies to decision makers needs to be transparent. More specifically, they argued how all participant variables as well as all the preference study results need to be reported to avoid the selective reporting of results. One industry representative raised the question of dealing with unexpected results: “But what if (…) it doesn’t really fit in our advantage; how will we deal with it?” (IN_NL_10). Some patients and patient organization representatives raised the issue of confidentiality of patient information: “Getting access to your name and identity, and that whole thing, or someone sending a bulk email” (PA_SE_1). One caregiver mentioned the need to disclose preference study outcomes to patients.

3.5.4 Misuse of PP in Decision Making and Creating Disparities

Several regulators, academics, and HTA/payer representatives expressed concerns about the risk of misusing PP in decision making “to overcome other important aspects of the technology” (AC_IT_24) or “to make more profit” (PY_IT_13). Two HTA/payer representatives further proposed that the risk of misusing PP to force decision makers towards a certain decision outcome should be mitigated when embarking on preference research: “I am concerned that there will be (…) a huge amount of so-called preference studies that will be used to say ‘Oh, you say, the patient said that so you have to act’” (PY_IT_13). In the context of marketing authorization, some regulators described the risk that pharmaceutical companies could misuse PP to “circumvent the normal regulatory process” (RE_UK_9). Two HTA/payer representatives were concerned about disparities generated by PP being central for regulatory and HTA decision making as more influential advocacy groups may have a more central political role: “It favours that group of patients for the sole reason that they prefer it that way and they’ve been able to influence because of they have a very strong patient organization” (PY_SE_14).

4 Discussion

This study provides insights into stakeholders’ attitudes, needs, and concerns related to PP in decision making along the MPLC. Interviewees from all stakeholder groups stressed the importance of increasing stakeholders’ understanding of the concept of PP and preference methods and ensuring patients’ understanding of the questions asked in PP studies. Key concerns and needs per stakeholder group included the following (Fig. 1):

  • Academics: researching the validity, reliability, reproducibility, and generalizability of preference methods;

  • HTA/payer representatives: developing quality criteria for evaluating PP studies and gaining insights into how to weigh PP study results in decision making;

  • Industry representatives: obtaining guidance on the design and conduct of PP studies and recognition on the importance of PP from decision makers;

  • Patients, caregivers, and patient representatives: providing incentive and adequate information towards patients when participating in PP studies;

  • Physicians: avoiding bias due to commercial agendas in PP studies and clarifying how to address subjective and emotional elements when measuring PP; and

  • Regulators: avoiding the misuse of PP study results to overrule the traditional efficacy and safety criteria for marketing authorization and obtaining robust PP study results.

This study underpins the importance of educating stakeholders about the concept of PP and preference methods; 43% of interviewees reported not being familiar with PP, and interviewees from all stakeholder groups raised the need for higher awareness among stakeholders (decision makers, clinicians, academics, and patients) on PP studies. Examples of efforts to increase familiarity among stakeholders are accumulating, e.g., webinars by the FDA and congresses about health preference research [43, 44]. This study also points towards a critical need for a framework to measure PP more structurally. Such efforts are also ongoing, e.g., the issuance of draft guidance by the FDA for collecting and submitting patient experience data (including PP) [33].

Interviewees raised several methodological questions (e.g., about the sample and the validity of PP study results), some of which are touched upon in regulatory guidance [3] already available from before the initiation of this study. This finding could imply that interviewees were unaware of this guidance and/or that more research on these topics is needed because the existing guidance is not sufficiently detailed. The former implication might be explained by a lack of guidance in the European context; thus far, there is no guidance from the EMA on how to measure PP for regulatory purposes. The second implication has also been raised by Johnson et al. [45], stating that available FDA guidance “provides little help for readers in assessing the relevance of the listed methods for FDA regulatory reviews”. Regarding method selection, the present study reveals tension related to the desired level of robustness of preference methods; while interviewees from several stakeholder groups found it crucial that preference methods are scientifically robust, some interviewees argued that “simpler” methods are necessary to reduce the cognitive burden of the method and to reduce the cost and time needed to conduct the preference study. The FDA guidance [3] states that “newer methods may also be acceptable” but does not explicitly mention what their quality criteria would be.

Both on the HTA/reimbursement and marketing authorization level, some interviewees argued that using PP could improve the quality of decision making by increasing certainty for decision makers when they need to evaluate uncertain clinical evidence. Examples that reflect the interest of HTA bodies are the Institute for Quality and Efficiency in Health Care PP studies and the report by the Belgian Health Care Knowledge CentreFootnote 10 on how to involve patients and the public in reimbursement and resource allocation decisions [2, 29, 36]. However, no formal guidance has been issued by HTA bodies or reimbursement decision makers, which might explain the uncertainty expressed by interviewees on how to conduct PP studies for HTA and reimbursement.

As suggested by some regulators in this study and reflected in a recent example in which the National Institute for Health and Care Excellence provided scientific advice [37], gathering input from decision makers on specific PP study designs may be an important step forward. Based on their experience, the National Institute for Health and Care Excellence recommends that companies seek their advice on PP study designs to allow for the PP study results to be used in clinical development programs for new treatments. Such advice may be both valuable from the company’s perspective to ensure that PP study results will be useful for decision makers as well as from the decision makers’ perspective to ensure that potential methodological issues are addressed prior to conducting the PP study.

The main strength of this study is its inclusiveness; interviewees were equally recruited from six different stakeholder groups and eight countries, resulting in a heterogeneous and large sample. However, the magnitude of this study can also be viewed as a limitation because it precluded gaining in-depth insights into the motivations and background of each interviewee. We captured the country, stakeholder group, and familiarity of interviewees but did not differentiate results according to other interviewee characteristics. For example, their opinions may also be shaped by the type of preference method they were most familiar with.

It is important to interpret the results in light of interviewees’ knowledge of the topic; interviewees reported being unfamiliar (43%), moderately familiar (42%), or very familiar (15%) with preference methods and studies. Self-reported familiarity was highest among academics and lowest among patients and physicians. We tried to account for a lower familiarity among these latter stakeholder groups by developing a simpler version of the interview guide for these stakeholder groups. Nonetheless, interviewees’ opinions were most likely influenced by their familiarity regarding the topic. For example, the stakeholder groups least familiar with PP studies may also be the individuals least positive about using PP study results in decisions. Second, interviewees were recruited upon suggestion by PREFER members and by interviewed persons. This might have resulted in a biased sample because these persons may be more positive about PP. The opinions presented in this paper should therefore not be interpreted as representative of a larger population than the interviewed sample; there may very well be persons within each stakeholder group with views other than those presented in the paper.

Additionally, four interviewees per stakeholder group from each country do not enable comparison across all possible permutations of relevant interviewees’ features (country, stakeholder group, disease area). There were eight interviewers with likely different interviewing styles. However, having eight interviewers was necessary to conduct interviews in some interviewees’ native language. We tried to minimize differences in interviewing styles by using an interviewing protocol and organizing an explanatory protocol meeting. All but one interviewer (AC) were part of the PREFER project. Although interviewers had no personal interest in the study outcome, their knowledge and assumptions about the research topic may have shaped the direction of the interviews. A final limitation relates to terminology; after interviewees’ personal definitions of PP were elicited, a definition of PP was provided at the beginning of the interview. Nevertheless, interviewees might have referred to their own pre-existing definitions during the interview.

5 Conclusions

Despite interest from interviewees within each of the stakeholder groups, an effective use of PP is hampered by a lack of standardization and consensus on how to measure PP and use PP in decision making. To advance usage of PP, efforts are needed on three levels that mirror stakeholders’ needs and concerns: (1) the educational and cultural level, via increased awareness and understanding among stakeholders about concept of PP, preference methods and the potential value of using PP in decision making; (2) the methodological level, via increased understanding of quality criteria of PP studies; and (3) the procedural level, via increased understanding on how to integrate PP into decision making.