Pharmaceutical Medicine

, Volume 33, Issue 2, pp 121–133 | Cite as

Evaluation of the Effectiveness of Additional Risk Minimization Measures for Voriconazole in the EU: Findings and Lessons Learned from a Healthcare Professional Survey

  • Joanna LemEmail author
  • Muhammad Younus
  • Jalal A. Aram
  • Shahrzad Moosavi
  • Klaus Freivogel
  • Anne Lewis
  • Rachel E. Sobel
Open Access
Original Research Article



Voriconazole is an extended-spectrum antifungal agent approved for the treatment and prophylaxis of invasive aspergillosis and other serious fungal infections. In 2014, additional risk minimization measures (aRMM) consisting of a Healthcare Professional (HCP) Question and Answer (Q&A) Brochure, HCP Checklist, and Patient Alert Card were implemented on a rolling basis across the European Union (EU) to mitigate three key risks with voriconazole: phototoxicity, squamous cell carcinoma (SCC) of the skin, and hepatotoxicity. The risks of phototoxicity and hepatotoxicity have been documented in the Summary of Product Characteristics (SmPC) since voriconazole was first approved in the EU in 2002. However, the risk of SCC of the skin was a more recent addition to the SmPC (added in 2010).


We evaluated the effectiveness of the aRMM, as per EU Good Pharmacovigilance Practices Module XVI, via a survey of HCPs.


An online survey was conducted among specialty care HCPs in 10 EU countries who had received by mail aRMM tools 12 months previously. Survey questions evaluated HCPs’ receipt and utilization of aRMM tools, and knowledge of the three risks.


Of 27,396 HCPs invited to participate, 332 eligible respondents completed the survey (response rate: 447/26,735; 1.7%). In total, 19.6% of respondents recalled receiving the HCP Q&A Brochure, 22.6% the HCP Checklist, and 25.9% the Patient Alert Card. HCPs had a high level of knowledge of phototoxicity and hepatotoxicity; however, knowledge of SCC was lower. Knowledge of the three risks and self-reported risk minimization behavior was slightly improved in those who had read the HCP Q&A Brochure compared with those who had not.


The effectiveness of the voriconazole aRMM cannot be meaningfully inferred from the results due to the low survey response rate. The assessment indirectly points to the SmPC or other resources being the main source of risk information for HCPs. Engaging HCPs before designing and implementing an aRMM program is crucial to ensure an effective and focused program. (EU PAS registration number: EUPAS12624).

Key Points for Decision Makers

The 332 (of 27,396 invited) healthcare professionals (HCPs) who completed the survey were generally well informed about the risks of phototoxicity and hepatotoxicity associated with voriconazole treatment, but the awareness of the risk of squamous cell carcinoma was lower than the prespecified threshold of 80%.

Due to the low response rate and small sample sizes of the sub-groups, the educational value of the voriconazole additional risk minimization measure (aRMM) tools cannot be inferred from the results, which indirectly point to the Summary of Product Characteristics or other resources being the main source of risk information for HCPs.

Engaging HCPs before designing and implementing an aRMM program is a crucial step in ensuring a useful and HCP-focused aRMM program.

Further research and alternative methods are needed to overcome the limitations of survey research, thereby helping risk minimization evaluations evolve.

1 Introduction

Risk minimization measures are interventions intended to reduce or prevent the occurrence of adverse reactions, or their sequelae, associated with exposure to medical treatments. These are recommended as part of good pharmacovigilance practices (GVP), for example, as detailed in the European Medicines Agency (EMA) GVP Modules V and XVI [1, 2]. Routine risk minimization measures are applicable to all medicinal products and can take the form of the Summary of Product Characteristics (SmPC), appropriate medicine labeling, safety information in the package leaflet, and pack size. The majority of safety concerns are addressed by these routine measures. However, for certain medicinal products, when routine measures are deemed to be insufficient for the safe and effective use of a medicine, additional risk minimization measures (aRMM) are also necessary [1, 2, 3]. A number of aRMM tools are currently utilized (e.g. educational programs [2], controlled access programs, controlled distribution programs, etc.), and this field is continuously evolving with technological advances, such as interactive web- and application-based tools for healthcare professionals (HCPs) [4].

Voriconazole (VFEND®) is an extended-spectrum triazole antifungal agent indicated for the treatment of invasive aspergillosis, candidemia in non-neutropenic patients, and other serious fungal infections, in adults in both the USA and the EU since 2002. It is also approved in children over the age of 12 years in the USA [5] and in children aged over 2 years in the EU [6]. Voriconazole is also indicated in the EU for the prophylaxis of invasive fungal infections in high-risk hematopoietic stem cell transplant (HSCT) recipients [6]. Voriconazole is effective for the primary and secondary prevention of invasive fungal infections including invasive aspergillosis in patients with hematologic diseases and HSCT recipients with a favorable risk-benefit balance [7].

In light of a number of spontaneous reports, clinical trial findings, and other publications about the risks of phototoxicity [8, 9, 10, 11], squamous cell carcinoma (SCC) of the skin [12, 13, 14, 15, 16], and hepatotoxicity [17, 18] with voriconazole, an aRMM program was implemented beginning in April 2014 based on rolling local health authority approvals in 33 EU countries (with implementation dates from April 2014 through December 2014). The goals of the aRMM tools were to increase HCPs’ awareness of these three risks and their management. Specialty care physicians who generally prescribe voriconazole, including infectious disease physicians, oncologists, hematologists, and solid organ transplant physicians, were mailed the hard copies of the aRMM tools, which included an 8-page (with 5 pages of content) HCP Question and Answer (Q&A) Brochure, a 2-page HCP Checklist, and a double-sided, wallet-sized Patient Alert Card (Table 1).
Table 1

Description of the aRMM tools

aRMM tool


The HCP Q&A Brochure

Provides detailed information about the risks and risk minimization of phototoxicity, SCC of the skin, and hepatotoxicity, and recommendations about how to manage them

Highlights the topics that should be discussed with patients during patient counselling

Offers guidance about the signs and symptoms that should warrant patients/caregivers to immediately contact the doctor

Patient Alert Card

Provided to each HCP to fill in their contact details on the card and to distribute to patients prescribed voriconazole

Reminds patients of the need:

 for dermatological evaluations on a regular basis (if phototoxic reactions occur)

 to avoid exposure to sunlight

 to use protective clothing and sunscreen with high sun protective factor

 to inform their physician if they develop sunburn, severe skin reactions, or skin abnormalities

The HCP Checklist

HCPs are instructed to complete the HCP checklist for all new patients receiving voriconazole and retain it in the patient’s medical record

This is the core aRMM tool that reminds HCPs about:

 the risks of phototoxicity, SCC of the skin, and hepatotoxicity with the use of voriconazole

 the appropriate use of voriconazole, and management of patients with underlying hepatic impairment and those developing hepatic injury during voriconazole therapy

 the requirements for dermatological examination and liver function monitoring

 discussing the importance of monitoring these risks with patients

 providing the Patient Alert Card to each patient prescribed voriconazole

aRMM additional risk minimization measures, HCP healthcare professional, Q&A question and answer, SCC squamous cell carcinoma

Module XVI of EMA GVP stipulates that the effectiveness of aRMM should be evaluated for all medicinal products that utilize such tools [2]. A systematic evaluation of the effectiveness of aRMM helps to understand whether the program objectives have been met (i.e. whether an intervention has been effective) or provide evidence that further amendments to the program are needed [1, 2]. These evaluation studies of aRMM are considered post-authorization safety studies (PASS) in the EU and are regulatory commitments. We evaluated the effectiveness of the aRMM via a survey of HCPs who were prescribers of voriconazole (EUPAS12624). The overall objective of this PASS was to evaluate the effectiveness of the voriconazole aRMM tools implemented across Europe in mitigating the risks of phototoxicity, SCC of the skin, and hepatotoxicity in patients receiving voriconazole; specifically, to assess HCPs’ receipt and utilization of the aRMM tools, knowledge of the risks of phototoxicity, SCC, and hepatotoxicity, and knowledge of risk minimization practices, as well as self-reported practices, with respect to minimizing these risks in accordance with the prescribing information.

The results of the evaluation are presented and discussed in the context of the strengths and limitations of the evaluation methodology, factors influencing target population behavior, and knowledge and lessons learned from the study. As recommended by the Council for International Organizations of Medical Sciences (CIOMS) Working Group in their report [19], this manuscript was developed to add to the limited body of published evidence, and to help facilitate advances in the risk management and evaluation field.

2 Methods

2.1 Study Design

This was a voluntary cross-sectional survey of HCPs conducted 12–17 months after the distribution of aRMM tools, depending on country. A structured, self-administered questionnaire was used to collect the survey data. The HCPs were invited to take the online survey using a secure uniform resource locator requiring a unique identifier. Eligible HCPs to whom the aRMM tools were mailed, received a letter by postal mail inviting them to participate in the survey. HCPs also received an email in addition to postal mail in three countries where email addresses were available (France, UK, and Italy).

2.2 Survey Population

The survey population constituted HCPs from 10 EU countries who were mailed the aRMM tools: Austria, Denmark, France, Germany, Hungary, Ireland, Italy, the Netherlands, Spain, and the UK. The inclusion criteria included those HCPs who self-reported prescribing voriconazole at least once within 12 months preceding the survey and consented to participate in the survey. Exclusion criteria included non-practicing physicians, those who had participated in pilot testing of the aRMM tools or draft survey, and employees of Pfizer or of a research organization/vendor contracted by Pfizer to administer the survey.

2.3 Survey

Using established survey methodology [20], an online survey was developed to assess the HCPs’: (1) receipt, awareness, and utilization of the aRMM tools; (2) knowledge of the risks of phototoxicity, SCC of the skin, and hepatotoxicity with voriconazole; (3) knowledge of the practices recommended to minimize these three risks in accordance with the aRMM tools; and (4) self-reported risk minimization behaviors for these three risks (Electronic Supplementary Material).

Prior to launching the survey, moderator-guided user testing of the survey instrument was conducted to test clarity and comprehension of the draft questions, with a sample of potential prescribers of voriconazole in the UK. Overall, most physicians involved in the user testing found the voriconazole aRMM effectiveness evaluation survey instrument to be clear and comprehensive and did not propose significant revisions to the instrument. Instead, their feedback tended to focus on their clinical experiences using voriconazole. Physicians also suggested minor formatting, language choice, and editorial changes to improve the clarity of the survey. These were incorporated as appropriate prior to the implementation.

In addition, an operational feasibility assessment was completed, and relevant Competent Authority, Ethics Committee, and/or Institutional Review Board reviews were conducted according to each country’s local regulations. The ability to provide a nominal honorarium to reimburse HCPs for their time to complete the survey was also assessed. However, due to variations in local processes [21] and/or significant administrative burdens for a brief one-time survey, which would preclude survey implementation in a timely manner, it was only deemed feasible to provide an honorarium in Spain.

The final survey comprised 21 questions, which evaluated each of the 4 objectives. Other survey questions included screening/eligibility questions, demographic characteristics, and HCPs’ opinion about the usefulness of aRMM tools in clinical practice.

2.4 Statistical Analysis

Descriptive statistics were used to analyze the survey data including counts and proportion of correct/desired responses with exact two-sided 95% confidence intervals by the method of Clopper–Pearson [22]. An a priori threshold of 80% correct responses per risk questions was used to define the success of the program. The selection of this threshold for success was regarded as being subjective and not based on prior knowledge, experience, or established scientific criteria, as per previous publications [23, 24, 25].

3 Results

3.1 Survey Response

Of 27,396 specialty HCPs invited to participate, 26,735 invitations were successfully delivered, 447 respondents (1.7%) completed the online screening (visited the survey website and provided their unique ID from the invitation letter), 354 (1.3%) were eligible for participation, and 332 (1.2%) completed the survey (Table 2). The most frequent primary medical specialties of the completers were hematology (46.1%) and oncology (26.8%; Table 2). Almost half of the prescribers who completed the survey had been practicing medicine for 15 or more years (48.8%) and an additional 38.3% had been practicing for between 6–15 years.
Table 2

Summary of country of practice and medical specialty of survey participants

n (%)

Invited HCPs

N = 27,396

Survey completers

N = 332

Response rate (%) (screened respondents)a

Country of practice


4823 (17.6)

191 (57.5)



6229 (22.7)

42 (12.7)


 The Netherlands

3615 (13.2)

21 (6.3)


 United Kingdom

4779 (17.4)

21 (6.3)



3136 (11.4)

16 (4.8)



2375 (8.7)

14 (4.2)



888 (3.2)

13 (3.9)



308 (1.1)

7 (2.1)



870 (3.2)

5 (1.5)



373 (1.4)

2 (0.6)


Primary medical specialty


10,116 (36.9)

153 (46.1)

Not available


8723 (31.8)

89 (26.8)

Not available

 Infectious diseases

2980 (10.9)

56 (16.9)

Not available

 Intensive care

275 (1.0)

3 (0.9)

Not available

 Solid organ transplant

93 (0.3)

3 (0.9)

Not available

 Critical care

13 (0.0)

2 (0.6)

Not available


44 (0.2)

1 (0.3)

Not available


4973 (18.2)

25 (7.5)

Not available

HCP healthcare professional

aResponse rate calculated as the percentage of those screened over the invited prescribers excluding the number of invitations returned as undeliverable

b‘Other’ was a category that could be selected by respondents whose self-reported specialties were not in any of the above categories

3.2 Receipt and Utilization of Voriconazole aRMM Tools

The average proportion of undeliverable aRMM material packages across all ten study countries was 1.4% (range 0.3−4.1). However, of 332 survey completers, 19.6% (n = 65) recalled receiving the HCP Q&A Brochure, 22.6% (n = 75) recalled receiving the HCP Checklist, and 25.9% (n = 86) recalled receiving the Patient Alert Card (Fig. 1), with the majority (47.0–56.9%) of respondents stating that they did not remember receiving the aRMM tools. Of those who reported receipt of the HCP Q&A brochure, 33.8% (n = 22) reported reading the entire brochure with 53.8% (n = 35) reading some of the brochure; whereas 17.3% (n = 13) of HCPs who reported receipt of the checklist also stated that they always completed the HCP Checklist when prescribing voriconazole and 48.0% (n = 36) stated that they sometimes used it. Of HCPs who reported receipt of the Patient Alert Card, 25.6% (n = 22) always and 41.9% (n = 36) sometimes filled in and distributed the card to patients. The proportion of reported aRMM receipt and utilization was slightly higher among those HCPs who treated the highest number of patients (n > 20) with voriconazole in the past 12 months, compared with those who reported treating fewer patients (n ≤ 20) with voriconazole (Fig. 2).
Fig. 1

Proportion of HCP respondents reporting receipt of the three aRMM tools (Question: “Did you or your hospital receive the HCP Q&A Brochure/HCP Checklist/Patient Alert Card?”). aRMM additional risk minimization measures, CI confidence interval, HCP healthcare professional, Q&A question and answer

Fig. 2

Proportion of correct responses to questions on receipt and utilization of the aRMM tools by HCPs, stratified by the number of patients they had treated with voriconazole in the past 12 months. AE adverse event, aRMM additional risk minimization measures, CI confidence interval, HCP healthcare professional, Q&A question and answer

3.3 HCPs’ Knowledge of Risks

Of the 332 survey completers, the following proportions correctly identified phototoxicity [88.6% (n = 294)], SCC [44.3% (n = 147)], and hepatotoxicity [96.4% (n = 320)] as risks associated with voriconazole.

3.4 HCPs’ Knowledge of Risk Minimization Measures for Phototoxicity, SCC of the Skin, and Hepatotoxicity

Respondents were generally knowledgeable about practices recommended to mitigate the risks of phototoxicity and SCC of the skin (> 75.0% correct responses for 5/7 questions). However, some questions had lower proportions of correct responses. For example, 34.9% correctly answered that dermatologic evaluation should be performed on a systematic and regular basis, and 62.0% correctly answered that sun-exposed areas of the skin should be covered (Table 3).
Table 3

Proportion of correct responses to questions about the risks of phototoxicity and SCC of the skin and associated risk minimization practices


n (%)

95% CI

Question: Which precautionary measures should physicians communicate to their patients for whom they have prescribed voriconazole?

 Dermatologic evaluation should be performed on a systematic and regular basis

116 (34.9)


 Use of sufficient sunscreen with high sun protection factor

271 (81.6)


 Covering sun-exposed areas of skin

206 (62.0)


 Detecting signs and symptoms of phototoxicity

306 (92.2)


 Avoiding exposure to direct sunlight

297 (89.5)


Knowledge about voriconazole risk minimization practices

 Voriconazole should be discontinued if premalignant skin lesions or SCC of the skin are identified

251 (75.6)


 Multidisciplinary advice should be sought if phototoxic reactions occur, including dermatologist referral

287 (86.4)


 Long-term treatment (> 6 months) should only be considered if the benefits outweigh the potential risks

311 (93.7)


CI confidence interval, SCC squamous cell carcinoma

Similar to the knowledge of the risk of phototoxicity and SCC of the skin, respondents were well informed about mitigating the risk of hepatotoxicity (> 78.0% correct responses for all four questions; Table 4).
Table 4

Proportion of correct responses to questions about the risk of hepatotoxicity and associated risk minimization practices


n (%)

95% CI

Prescribers should communicate to their patients the clinical signs of liver damage that warrant contacting the doctor immediately

304 (91.6)


Laboratory tests of hepatic function are required at initiation of voriconazole and during the first month of treatment

317 (95.5)


Liver function should be monitored monthly, if there are no changes in liver function tests after 1 month of voriconazole treatment

260 (78.3)


If liver function tests are markedly elevated, voriconazole should be discontinued (unless risk-benefit balance suggests otherwise)

313 (94.3)


CI confidence interval

3.5 Effect of Reading the HCP Q&A Brochure on Risk Minimization Knowledge and Behavior

Although the study was not powered for statistical comparisons, descriptive data showed that reading the HCP Q&A Brochure was associated with slightly improved knowledge of risks and risk minimization questions (Fig. 3) compared with those who indicated that they did not read the brochure. Similarly, self-reported risk minimization behavior appeared slightly improved in those who indicated that they had read some or all of the brochure, compared to those who had not read it (Fig. 3).
Fig. 3

Proportion of correct responses to the three risks, knowledge of risk minimization questions, and associated risk minimization behavior by HCP Q&A Brochure reading status of the respondents. AEs adverse events, CI confidence interval, HCP healthcare professional, SCC squamous cell carcinoma, Q&A question and answer

Subgroup analyses were also conducted that looked at the effects of country, medical specialty, and the frequency of voriconazole prescribing on knowledge and risk minimization practice. However, the sample sizes of many of the subgroups were too small to permit comparisons and meaningfully assess the differences. The effect of frequency of prescribing on knowledge and risk minimization practices is shown in Supplementary Tables 1 and 2.

3.6 Perceived Usefulness of the aRMM Tools

The distribution of perceived usefulness of the aRMM tools, by HCPs who reported receipt of the materials, was similar across the different tools. A small number of HCPs found them either ‘extremely useful’ or ‘not useful’ (Supplementary Table 3).

4 Discussion

This study was conducted to evaluate the effectiveness of the aRMM implemented across the EU to mitigate the risks of phototoxicity, SCC of the skin, and hepatotoxicity with voriconazole. Among respondents, we found that, in general, knowledge of the risks of phototoxicity and hepatotoxicity was high (88.6% and 96.4%, respectively), but appeared to be lower for SCC of the skin (44.3%).

The study results reported here should be interpreted with caution due to the low survey response rate, and thus unknown generalizability, a common challenge of surveys among HCPs [26]. Participation in the survey was voluntary and, despite extensive follow-up efforts, a relatively small proportion responded (1.7%) and completed (1.2%) the survey.

As a result of this low response rate, there is a possibility that the respondents may differ from those who did not respond to the survey in terms of characteristics, motivations, awareness of the aRMM, and knowledge of voriconazole risks. Indeed, when survey completers were compared with all invited HCPs in terms of country of practice and medical specialty, the results showed that the distribution of these characteristics was different between the two groups [e.g. the majority (57.5%) of survey completers were from Spain; Table 2]. Thus, a potential selection bias cannot be ruled out and the generalizability of the study results to all prescribers of voriconazole is unknown.

Some interesting differences in response rates between countries were observed in this study. We note that the highest response rate of 4.2%, which was achieved in Spain, could have been in part related to the fact that Spain was the only country where we were able to offer a small honorarium for participation. In the EU, the requirements for offering honoraria payments are variable and often quite strict due to classification of such studies as PASS. This variability has been noted by others as a significant barrier to operational success and scientific interpretation [26].

Furthermore, the differential proportions of the physician specialties in the study population compared with the overall invited population may also reflect the underlying prescriber population for whom aRMM materials for voriconazole were relevant. It is one of the well-known limitations of conducting this type of research among HCPs in the EU, that due to strict privacy regulations [21] (such as General Data Protection Regulation (GDPR) [27]), it is not possible for sponsors to obtain access to details of those prescribing voriconazole, which could have been utilized to develop more targeted lists for both distribution of aRMM materials and evaluation, and to better contextualize non-responders.

In this evaluation, the response rate (1.7%) was much lower than expected (3–7%) based on the authors’ experience with similar studies in Europe (data on file). In the USA, the Food and Drug Administration (FDA) noted that it is not uncommon to observe response rates to surveys below 10% [28]. Previous recommendations suggest that survey response rates of at least 60% are preferred to ensure that non-response bias does not influence the validity of survey findings [29]. Efforts to improve the response rate during the data collection were taken, including conducting all communications in the local language and providing reminders to complete the survey (56,080 postal reminders and 4236 email reminders). Over recent years there has been a steady downward trend in clinicians’ response rates to surveys [24, 30, 31, 32], which could partially explain the low response rate observed in this survey.

Another limitation of this survey was the absence of comparative data from pre-aRMM implementation (baseline data). Being able to evaluate HCPs’ knowledge and risk minimization practices pre- and post-aRMM implementation would enable stronger conclusions about the effectiveness of this aRMM program. Whilst it would have been desirable to have baseline data from before aRMM implementation, it was not feasible due to the need to implement the tools quickly and efficiently following the EMA’s request. Furthermore, the threshold for program success (80%) was also not based on published literature, prior knowledge, or established scientific criteria, because pertinent data are currently lacking. A review of all completed aRMM evaluation surveys (classified as PASS) from the EU PAS Register found that only 2 of 11 cross-sectional surveys selected 80% as a predefined threshold of success [23]. Better understanding of the suitable thresholds for aRMM success [33] and whether these thresholds differ by stakeholder, specialty, risk, or other factors, would be valuable when making conclusions about the program’s effectiveness.

Surveys are a well-established standard to measure process indicators such as distribution and utilization of educational materials, as well as knowledge and self-reported behavior [26]. However, despite many strengths, surveys also have limitations [24]. The low response rates from surveys suggest that providing feedback on a voluntary basis does not always achieve the desired level of response [33]. Many limitations are illustrated by this survey. The utility of such surveys in evaluating the effectiveness of the aRMMs and drawing meaningful conclusions is questionable considering the lack of comparative data, the extremely low response rate, unknown generalizability, and potential selection bias (e.g. HCPs who took the time to complete to this survey were likely to be better informed about or have greater interest in pharmacovigilance than those who did not respond; the majority of respondents were from one of the many countries surveyed). Other study designs and frameworks to assess effectiveness of aRMMs should be considered in future evaluation work [34, 35, 36].

An additional notable limitation of this effectiveness evaluation is the sole focus on process indicators (i.e. receipt and utilization of aRMM materials, knowledge, and self-reported practice pertaining to risk minimization). We were unable to measure outcome indicators (i.e. incidence of safety risks) pre- and post-implementation of the program via either primary or secondary data collection studies due to the long latency of outcomes such as SCC [37], combined with the relatively low frequency of voriconazole usage; under these circumstances, meaningful changes in risks would be nearly impossible to observe. Thus, even though assessment of both process and outcome indicators is strongly encouraged, in some situations, such as this, assessment of outcomes is not feasible and conclusions about effectiveness can only be drawn based on an incomplete set of process indicators.

The subgroup analyses indicated that those who have read some or all of the HCP Q&A brochure may have somewhat increased knowledge of the risks, compared with those who did not receive or read it. However, as the sample size of subgroups was small, caution must be applied in the interpretation of these data. This finding was not replicated when looking at the impact of reading of the brochure on risk minimization practices. In addition, no strong relationship between voriconazole prescribing frequency and HCPs’ knowledge was found, but again, the small subgroup sample size warrants cautious interpretation.

4.1 Potential Reasons Behind Low Utilization of aRMM Tools

A large proportion of survey respondents (80.4%) either did not receive or did not recall receiving the HCP Q&A Brochure.

The low recollection of receipt of the aRMM tools (HCP Q&A Brochure: 19.6%; HCP Checklist: 22.6%; Patient Alert Card: 25.9%) may have several explanations. It may be that, although delivered, the plain white envelopes were not given to the intended recipient due to administrative staff filtering or HCPs working at multiple clinics. Alternatively, the timeframe for conducting the survey (i.e. within the recommended 12–18 months after implementation of aRMMs [2]) may be too long an interval for HCPs to recall receipt of the tools. Both of these issues would be areas for future research to further inform best aRMM evaluation practice.

Of those who indicated receiving the brochure, 33.8% reported reading the entire document and 53.8% reported only reading some parts. Thus, the survey results point to a selective utilization of the aRMM tools, which could in part explain the lower knowledge of the risk of SCC compared with the risks of hepatotoxicity and phototoxicity among survey responders. One of the potential reasons behind low utilization of aRMM tools include HCPs’ own perception of familiarity with the risks. Almost half of the responding prescribers had been practicing medicine for 15 or more years (48.8%) and an additional 38.3% had been practicing for between 6–15 years. Among prescribers who treated the highest numbers of patients with voriconazole in the past 12 months (i.e. > 20 patients), slightly higher proportions reported receipt and utilization of the aRMM tools compared with prescribers who treated ≤ 20 patients with voriconazole. This observation is consistent with the finding reported by Landsberg and colleagues [38]. We note that prescriber’s general medical experience may affect aRMM awareness and utilization differently than specific experience of treating patients with voriconazole. Given the extent of the respondents’ experience in medicine and in managing patients who may require voriconazole, and that voriconazole was originally approved in 2002 in the EU, perhaps many believed they were already well aware of the risks and how to manage them. If, after a cursory review of the voriconazole HCP Q&A Brochure, HCPs assess that they are familiar with some or all of the risks that are being presented, they may not go on to read the full content and subsequently may discard the material.

Another potential reason for only partial review of the aRMM tools is that the risks presented may be considered low priority in relation to treating patients with complex underlying conditions. Patients receiving voriconazole typically experience life-threatening underlying conditions such as allogeneic HSCT, solid organ transplant, or inherited or acquired immunodeficiencies, who have also developed or are at risk for life-threatening fungal infections (e.g. invasive aspergillosis) [10, 39]. Physicians managing these critically ill patients are primarily focused on saving their lives by treating or preventing fungal infections. Within this context, the risks presented in the educational materials may be considered either already routinely monitored in the context of the underlying disease or secondary with respect to the management of the acute, life-threatening illness. Thus, reading the aRMM tools may not be a high priority for these HCPs. A similar theme was discussed in a National Comprehensive Cancer Network (NCCN) white paper, which described how oncologists, another highly specialized and trained group of physicians, perceived risk evaluation and mitigation strategies (REMS) in the USA as an unnecessary imposition on their daily workflow, which entails managing patients with serious conditions and prescribing drugs that have serious toxicities [40].

Additionally, given that a pharmaceutical company is the source of the aRMM tools, physicians may possibly view it as promotional in nature even though it was unbranded (i.e. there was no Pfizer logo on it). Some physicians may consider educational material from a pharmaceutical company as inconsequential or promotional by default, and thus may deprioritize or discard it. This explanation is supported by results from Topic 2 of the Strengthening Collaboration for Operating Pharmacovigilance in Europe (SCOPE), which indicated that of 3625 HCPs surveyed across the EU, the level of familiarity with educational materials, although still quite high (~ 66%), was considerably less than familiarity with Direct Healthcare Professional Communication (~ 90%) or National Competent Authorities (NCAs) communications (90%) [41]. The study also found that the most positively perceived senders of safety information were NCAs. This was followed by professional bodies and the EMA. Pharmaceutical companies and the press were negatively perceived when sending safety information. Electronic communications were reported by HCPs as the preferred method of communication. This finding also helps explain the low recall and utilization of the printed voriconazole aRMM tools.

4.2 What do the Survey Results Tell Us About aRMM Effectiveness?

Among the HCPs who responded to the survey, knowledge of the risks of phototoxicity and hepatotoxicity, and the associated risk minimization practices, were high, despite the low reported receipt of the aRMM tools; however, knowledge of the risk of SCC of the skin was low.

Phototoxicity and SCC of the skin are related risks and clinically linked by the role of ultraviolet exposure in both and use of the same preventive measures to minimize the risk of the two conditions. Hence, in the aRMM tools, risks of phototoxicity and SCC of the skin were presented together in the same section under one sub-heading. However, HCPs’ awareness of the risk of phototoxicity with voriconazole use was higher than for SCC of the skin. A potential reason for this lower level of knowledge about the risk of SCC of the skin may be the rarity and long latency of this risk compared with phototoxicity [42]. Another explanation could be, in part, due to a cursory review of the aRMM tools. Lastly, it is also possible that relatively lower levels of knowledge about the risk of SCC could be in part due to the fact that the risks of phototoxicity and hepatotoxicity have been documented in the SmPC for a longer period (since initial approval in 2002) than those for SCC (added in 2010).

The source of HCPs’ knowledge of the risks and associated mitigation practices for voriconazole is not clearly understood. However, the low reported rate of receipt of the aRMM tools and the relatively high knowledge of phototoxicity and hepatotoxicity compared with SCC of the skin indirectly suggests that the SmPC and other resources [e.g. the media, scientific conferences, continuing medical education (CME)] may be the sources of knowledge for safety information for these specialized HCPs. Similar to that highlighted by the US FDA in their 2013 commentary on patient education through medication guides [43], we note that it is challenging to isolate and quantify the precise impact of a specific communication medium via a knowledge survey, and especially one without baseline data. Thus, we acknowledge that the actual source of knowledge is unknown and probably multi-factorial. One action taken in agreement with the EMA to enhance HCPs’ awareness of the risk of SCC of the skin was to update the SmPC (i.e. Section 4.4 ‘Special Warnings and Precautions for Use’ was revised by reordering the adverse events to provide SCC of the skin with greater prominence) [6]. Also supporting the theory that sources of medical knowledge are multi-factorial, a study evaluating the knowledge of physicians who were sent educational materials on the thromboembolism risk of cyproterone acetate/ethinylestradiol, concluded that knowledge of the risk did not vary by receipt of the materials [25].

Of note, risk minimization measures have previously been shown to be effective. They were introduced to reduce the off-label use of tigecycline, and a retrospective study of their effectiveness concluded that this did decrease following implementation [44].

4.3 Lessons Learned

The findings from this study, in the context of the results from the SCOPE survey of HCPs, suggest that for future programs, optimization at the stage of aRMM program design and implementation may improve the uptake of aRMM tools. In retrospect, the design of the aRMM tools could have been more targeted if more extensive formative research with the key stakeholders (i.e. research with the target population to gain insights into their views of the aRMM tools planned, information presented, and incorporation of those tools into practice) had been executed prior to aRMM implementation. Therefore, for future aRMM programs, we would recommend first conducting a formative research study to investigate the optimal aRMM program design which would allow a more targeted approach to risk communication, as detailed by a relevant publication (2014) [45]. How the materials can be incorporated into existing clinical practice should be considered during this formative research, as this will greatly influence the type(s) of aRMM designed. Ideally, they should be compatible with existing patterns of care and workflows to facilitate adoption. Additionally, a multi-faceted approach to design (i.e. having multiple, integrated intervention elements delivered in unison or sequentially) could increase effectiveness [19, 45]. Considering the implementation of the aRMM program, the optimal method of distribution should be further investigated to maximize uptake, whether that involves changing the envelope in which it is posted or tailoring the number of copies of aRMM tools provided per envelope, based on stakeholder’s workflow or by using electronic modes of dissemination. Best practices recommend designing aRMM programs with dissemination in mind, to maximize external validity [45], and consider data-driven development, to avoid common pitfalls of aRMM programs [46]. In theory, these steps should take place prior to program implementation and sponsors should be encouraged to apply these approaches whenever feasible [45]. However, the timelines and urgency, which typically accompany decisions to implement an aRMM program, the need for regulatory endorsement from central health authorities, and country-specific approval requirements often make it extremely challenging to follow best practices from implementation science.

The experience with this study has also demonstrated that the addition of explanatory questions to the survey instrument (e.g. “What is your primary source of obtaining information about product safety?”, “Which aRMM tools do you find unhelpful?”, etc.) is an important element which would have helped to provide further useful context about the value of the aRMM tools in clinical practice.

We faced a number of operational challenges while implementing this evaluation survey. The most important ones stemmed from the varying applicable country-specific regulations and requirements with respect to local regulatory and ethics committee submissions. These need to be followed because of the classification of aRMM effectiveness evaluation studies as non-interventional PASS in accordance with GVP Module XVI [2]. A survey study design typically used in ‘market research’ is relatively straightforward to implement and usually does not require any additional regulatory notifications/submissions/approvals. As reviewed in a white paper by the International Society for Pharmacoepidemiology (ISPE) Special Interest Group (SIG): Benefit Risk Assessment, Communication and Evaluation (BRACE) [26], there are substantial implications of the effectiveness evaluation surveys being classified as PASS. In many EU countries, PASS are required to follow more stringent regulatory notification, submission, and approval procedures than non-PASS or market research surveys. These additional steps significantly increased the operational complexity of this PASS, delayed the time of survey launch, and reduced generalizability because at least one country was unable to be surveyed because the lengthy local ethics process prevented timely participation.

Offering financial incentives to reimburse participants for taking part has been found to increase response rates in similar surveys [47]. In our survey, the relatively greater response rate in Spain, the only country where payment was offered, corroborates this finding. However, our experience also showed this can be difficult to implement in practice, due to local regulations requiring contracts with HCPs and public disclosures stemming from the PASS classification and others which may render survey participation no longer anonymous. Further discussion of these issues can also be found in the ISPE white paper that highlights the challenges of conducting these evaluations in the EU [26].

A further operational challenge stemmed from the strict local privacy protection legislation, which resulted in difficulty in identifying a comprehensive list of HCPs who were mailed aRMM tools and thus a list to target with a survey invitation. We note that the new data protection regulations (GDPR) in the EU [27] will likely pose a further challenge in this regard.

Finally, the origin and mode of survey distribution is another factor in the success of aRMM evaluation. A recent EMA-sponsored survey of HCPs [41] demonstrated that the most preferred safety information channels were emails, point-of-care alerts (i.e. alerts through prescription systems) and, irrespective of electronic or paper-based format, reference books and national clinical guidelines. Thus, it is conceivable that those same preferences (i.e. via email or point-of-care alerts) could also be the most effective modes of inviting an HCP to a survey. The challenge faced by this study was that email lists were incomplete or absent for the purpose of research invitations, due to local privacy regulations, and therefore postal mail invitations were the best option to ensure representativeness. Furthermore, we hypothesize that many invitations were likely discarded because they were mailed on behalf of a pharmaceutical company (identified as the least-trusted source of risk communication [41]) and assumed instead to be promotional materials. Additionally, given HCPs’ preference for electronic communication, as well as our experience that a great majority of HCPs take surveys online, even if a phone survey was offered, online surveys are probably most effective. However, offering multiple methods of taking the survey (i.e. online and over the phone) might have increased participation.

5 Conclusion

Although the HCP survey response rate was low (1.7%), the results indicate that the HCPs who responded to the aRMM effectiveness survey were generally well informed about phototoxicity and hepatotoxicity as risks associated with voriconazole treatment and preventive measures. However, the level of awareness of the risk of SCC of the skin was lower than the prespecified threshold of 80%. The level of educational value of the voriconazole aRMM tools cannot be meaningfully inferred from the results, which indirectly point to the SmPC or other resources being the main source of risk information for these HCPs. The results showed that small proportions of surveyed HCPs recalled receiving the printed aRMM tools and reported reading the entire HCP Q&A Brochure. One of the key lessons learned from this study is that engaging HCPs before designing and implementing an aRMM program could ensure a useful and focused aRMM program. Given the challenges experienced with survey implementation, which compromised the ability to make meaningful scientific conclusions, future programs would benefit from a combination of: (1) early and thoughtful formative research and pilot testing; (2) minor EU regulatory reforms such as those outlined in the ISPE SIG BRACE white paper that could lessen some of the existing operational barriers; and (3) adding explanatory questions to gain further context for explaining the study results.

Finally, it is evident that further research and alternative methods are needed to overcome the limitations of survey research to help risk minimization evaluations evolve.



The authors would like to acknowledge the study lead Emily Freeman, Ph.D. (formerly of Pfizer) who helped to design the study, and the other members of the Pfizer study team: Margaret Tawadrous, Nicola Hickling, and Sharon Reid. The authors would like to thank all of the healthcare professionals who participated in the survey. The survey was designed as a collaboration between Pfizer and United BioSource Corporation (UBC) and was administered and analyzed by UBC. Editorial/medical writing support, under the direction of the authors, was provided by Kristina Harrison, Ph.D., of CMC CONNECT, a division of McCann Health Medical Communications Ltd, Glasgow, UK, funded by Pfizer, in accordance with Good Publication Practice (GPP3) guidelines.

Compliance with Ethical Standards


This study was sponsored by Pfizer. The open access fee was funded by Pfizer. An honorarium payment for completing a survey was offered to HCPs in Spain (pursuant to local laws and country regulations).

Conflict of interest

Joanna Lem, Muhammad Younus, Shahrzad Moosavi, Jalal A. Aram, and Rachel E. Sobel are employees and shareholders of Pfizer, the manufacturer of voriconazole. Klaus Freivogel is an employee of United BioSource, a contract research organization that has received research grants from Pfizer to conduct the research. Anne Lewis is a former contractor employee of United BioSource and a current Principal of Clermont Consulting Group, LLC.

Ethical approval

The final prescriber survey protocol and survey questionnaire were submitted to local regulatory authorities such as ethics committees, where required.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Supplementary material

40290_2019_273_MOESM1_ESM.pdf (451 kb)
Supplementary material 1 (PDF 451 kb)


  1. 1.
    European Medicines Agency. Guideline on good pharmacovigilance practices (GVP), Module V—risk management systems (Rev 2); 2017. Accessed 13 March 2018.
  2. 2.
    European Medicines Agency. Guideline on good pharmacovigilance practices (GVP), Module XVI—risk minimisation measures: selection of tools and effectiveness indicators (Rev 2); 2017. Accessed 13 March 2018.
  3. 3.
    Rubino A, Artime E. A descriptive review of additional risk minimisation measures applied to EU centrally authorised medicines 2006–2015. Expert Opin Drug Saf. 2017;16:877–84.CrossRefGoogle Scholar
  4. 4.
    European Medicines Agency. Mycamine: EPAR product information; 2011. Accessed 13 March 2018.
  5. 5.
    US Food and Drug Administration. Vfend summary of product characteristics; 2008.,021267s024,021630s013lbl.pdf. Accessed 13 March 2018.
  6. 6.
    European Medicines Agency. VFEND summary of product characteristics; 2017. Accessed 13 March 2018.
  7. 7.
    Zabalza A, Gorosquieta A, Equiza EP, Olavarria E. Voriconazole and its clinical potential in the prophylaxis of systemic fungal infection in patients with hematologic malignancies: a perspective review. Ther Adv Hematol. 2013;4:217–30.CrossRefGoogle Scholar
  8. 8.
    Patel AR, Turner ML, Baird K, Gea-Banacloche J, Mitchell S, Pavletic SZ, et al. Voriconazole-induced phototoxicity masquerading as chronic graft-versus-host disease of the skin in allogeneic hematopoietic cell transplant recipients. Biol Blood Marrow Transpl. 2009;15:370–6.CrossRefGoogle Scholar
  9. 9.
    Riahi RR, Cohen PR. Voriconazole-associated phototoxicity. Dermatol Online J. 2011;17:15.Google Scholar
  10. 10.
    Vohringer S, Schrum J, Ott H, Hoger PH. Severe phototoxicity associated with long-term voriconazole treatment. J Dtsch Dermatol Ges. 2011;9:274–6.Google Scholar
  11. 11.
    Sheu J, Hawryluk EB, Guo D, London WB, Huang JT. Voriconazole phototoxicity in children: a retrospective review. J Am Acad Dermatol. 2015;72:314–20.CrossRefGoogle Scholar
  12. 12.
    Mansh M, Williams K, Binstock M, Hafeez F, Kim J, Glidden D, et al. Voriconazole exposure modifies risk for cutaneous squamous cell carcinoma, invasive aspergillosis and death in lung transplant recipients. Am J Transpl. 2014;14(Suppl 3):193.CrossRefGoogle Scholar
  13. 13.
    Wojenski DJ, Bartoo GT, Merten JA, Dierkhising RA, Barajas MR, El-Azhary RA, et al. Voriconazole exposure and the risk of cutaneous squamous cell carcinoma in allogeneic hematopoietic stem cell transplant patients. Transpl Infect Dis. 2015;17:250–8.CrossRefGoogle Scholar
  14. 14.
    Goyal RK. Voriconazole-associated phototoxic dermatoses and skin cancer. Expert Rev Anti Infect Ther. 2015;13:1537–46.CrossRefGoogle Scholar
  15. 15.
    Mansh M, Binstock M, Williams K, Hafeez F, Kim J, Glidden D, et al. Voriconazole exposure and risk of cutaneous squamous cell carcinoma, aspergillus colonization, invasive aspergillosis and death in lung transplant recipients. Am J Transpl. 2016;16:262–70.CrossRefGoogle Scholar
  16. 16.
    McLaughlin JM, Equils O, Somerville KT, Aram JA, Schlamm HT, Welch VL, et al. Risk-adjusted relationship between voriconazole utilization and non-melanoma skin cancer among lung and heart/lung transplant patients. Transpl Infect Dis. 2013;15:329–43.CrossRefGoogle Scholar
  17. 17.
    Zonios D, Yamazaki H, Murayama N, Natarajan V, Palmore T, Childs R, et al. Voriconazole metabolism, toxicity, and the effect of cytochrome P450 2C19 genotype. J Infect Dis. 2014;209:1941–8.CrossRefGoogle Scholar
  18. 18.
    Saito T, Fujiuchi S, Tao Y, Sasaki Y, Ogawa K, Suzuki K, et al. Efficacy and safety of voriconazole in the treatment of chronic pulmonary aspergillosis: experience in Japan. Infection. 2012;40:661–7.CrossRefGoogle Scholar
  19. 19.
    Practical approaches to risk minimisation for medicinal products: Report of CIOMS Working Group IX. Geneva: Council for International Organizations of Medical Sciences (CIOMS); 2014. Accessed 13 March 2018.
  20. 20.
    Groves RM, Fowler FJ Jr, Couper MP, Lepkowski JM, Singer E, Tourangeau R. Survey methodology. 2nd ed. New York: Wiley; 2009.Google Scholar
  21. 21.
    Collins J, Bonneh-Barkay D. Considerations for successful risk-minimisation strategies in the EU. Pharm Med. 2016;30:257–61.CrossRefGoogle Scholar
  22. 22.
    Clopper CJ, Pearson ES. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika. 1934;26:404–13.CrossRefGoogle Scholar
  23. 23.
    Vora P, Artime E, Soriano-Gabarro M, Qizilbash N, Singh V, Asiimwe A. A review of studies evaluating the effectiveness of risk minimisation measures in Europe using the European Union electronic Register of Post-Authorization Studies. Pharmacoepidemiol Drug Saf. 2018;27:695–706.CrossRefGoogle Scholar
  24. 24.
    Madison T, Huang K, Huot-Marchand P, Wilner KD, Mo J. Effectiveness of the crizotinib therapeutic management guide in communicating risks, and recommended actions to minimize risks, among physicians prescribing crizotinib in Europe. Pharm Med. 2018;32:343–52.CrossRefGoogle Scholar
  25. 25.
    Davis KH, Asiimwe A, Zografos LJ, McSorley DJ, Andrews EB. Evaluation of risk-minimization activities for cyproterone acetate 2 mg/ethinylestradiol 35 microg: a cross-sectional physician survey. Pharm Med. 2017;31:339–51.CrossRefGoogle Scholar
  26. 26.
    Madison T, Arias A, DiSantostefano R, Gilsenan A, Matus D, Primatesta P, et al. Evaluating the effectiveness of additional risk minimisation measures via surveys in Europe: challenges and recommendations; 2016. Accessed 28 Jan 2019.
  27. 27.
    EU GDPR Portal. EU General Data Protection Regulation (GDPR); 2018. Accessed 13 March 2018.
  28. 28.
    Center for Drug Evaluation and Research. Food and Drug Administration. Department of Health and Human Services. Risk evaluation and mitigation strategy assessments: social science methodologies to assess goals related to knowledge; 2012. Accessed 13 March 2018.
  29. 29.
    Burns KE, Duffett M, Kho ME, Meade MO, Adhikari NK, Sinuff T, et al. A guide for the design and conduct of self-administered surveys of clinicians. CMAJ. 2008;179:245–52.CrossRefGoogle Scholar
  30. 30.
    Cho YI, Johnson TP, Vangeest JB. Enhancing surveys of health care professionals: a meta-analysis of techniques to improve response. Eval Health Prof. 2013;36:382–407.CrossRefGoogle Scholar
  31. 31.
    Cook JV, Dickinson HO, Eccles MP. Response rates in postal surveys of healthcare professionals between 1996 and 2005: an observational study. BMC Health Serv Res. 2009;9:160.CrossRefGoogle Scholar
  32. 32.
    Wiebe ER, Kaczorowski J, MacKay J. Why are response rates in clinician surveys declining? Can Fam Physician. 2012;58:e225–8.Google Scholar
  33. 33.
    Agyemang E, Bailey L, Talbot J. Additional risk minimisation measures for medicinal products in the European Union: a review of the implementation and effectiveness of measures in the United Kingdom by one marketing authorisation holder. Pharm Med. 2017;31:101–12.CrossRefGoogle Scholar
  34. 34.
    Sobel RE, Bate A, Marshall J, Haynes K, Selvam N, Nair V, et al. Do FDA label changes work? Assessment of the 2010 class label change for proton pump inhibitors using the Sentinel System’s analytic tools. Pharmacoepidemiol Drug Saf. 2018;27:332–9.CrossRefGoogle Scholar
  35. 35.
    Banerjee AK, Zomerdijk IM, Wooder S, Ingate S, Mayall SJ. Post-approval evaluation of effectiveness of risk minimisation: methods, challenges and interpretation. Drug Saf. 2014;37:33–42.CrossRefGoogle Scholar
  36. 36.
    Gridchyna I, Cloutier AM, Nkeng L, Craig C, Frise S, Moride Y. Methodological gaps in the assessment of risk minimization interventions: a systematic review. Pharmacoepidemiol Drug Saf. 2014;23:572–9.CrossRefGoogle Scholar
  37. 37.
    Didona D, Paolino G, Bottoni U, Cantisani C. Non melanoma skin cancer pathogenesis overview. Biomedicines. 2018;6:6.CrossRefGoogle Scholar
  38. 38.
    Landsberg W, Al-Dakkak I, Coppin-Renz A, Geis U, Peters-Strickland T, van Heumen E, et al. Effectiveness evaluation of additional risk minimization measures for adolescent use of aripiprazole in the European Union: results from a Post-Authorization Safety Study. Drug Saf. 2018;41:797–806.CrossRefGoogle Scholar
  39. 39.
    Wingard JR, Carter SL, Walsh TJ, Kurtzberg J, Small TN, Baden LR, et al. Randomized double-blind trial of fluconazole versus voriconazole for prevention of invasive fungal infection after allogeneic hematopoietic cell transplantation. Blood. 2010;116:5111–8.CrossRefGoogle Scholar
  40. 40.
    Johnson PE, Dahlman G, Eng K, Garg R, Gottlieb S, Hoffman JM, et al. NCCN oncology risk evaluation and mitigation strategies white paper: recommendations for stakeholders. J Natl Compr Cancer Netw. 2010;8(Suppl 7):S7–27.CrossRefGoogle Scholar
  41. 41.
    Alqvist-Radstad J, Andric A, Baldelli I, Coleman AM, Cupelli A, de Vries S, et al. SCOPE Work Package 6 Healthcare Professional Survey—Medicines Safety Communications and their Effectiveness; 2016. Accessed 13 March 18.
  42. 42.
    Cowen EW, Nguyen JC, Miller DD, McShane D, Arron ST, Prose NS, et al. Chronic phototoxicity and aggressive squamous cell carcinoma of the skin in children and adults during treatment with voriconazole. J Am Acad Dermatol. 2010;62:31–7.CrossRefGoogle Scholar
  43. 43.
    Dal Pan GJ. Commentary on “The effectiveness of varenicline medication guide for conveying safety information to patients: a REMS assessment survey” by Enger et al. Pharmacoepidemiol Drug Saf. 2013;22:716–8.CrossRefGoogle Scholar
  44. 44.
    Frajzyngier V, Quintana A, Tucker H, Wible M, Hickman A, Baillon-Plot N, et al. An evaluation of the effectiveness of risk minimization measures for tigecycline in the European Union. Pharm Med. 2017;31:89–99.CrossRefGoogle Scholar
  45. 45.
    Smith MY, Morrato E. Advancing the field of pharmaceutical risk minimization through application of implementation science best practices. Drug Saf. 2014;37:569–80.CrossRefGoogle Scholar
  46. 46.
    Morrato EH, Smith MY. Integrating risk minimization planning throughout the clinical development and commercialization lifecycle: an opinion on how drug development could be improved. Ther Clin Risk Manag. 2015;11:339–48.CrossRefGoogle Scholar
  47. 47.
    Cunningham CT, Quan H, Hemmelgarn B, Noseworthy T, Beck CA, Dixon E, et al. Exploring physician specialist response rates to web-based surveys. BMC Med Res Methodol. 2015;15:32.CrossRefGoogle Scholar

Copyright information

© The Author(s) 2019

Open AccessThis article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (, which permits any noncommercial use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  1. 1.Epidemiology, Worldwide Safety and RegulatoryPfizer IncNew YorkUSA
  2. 2.Global Medical AffairsPfizer IncNew YorkUSA
  3. 3.Safety and Risk Management, Worldwide Safety and RegulatoryPfizer IncNew YorkUSA
  4. 4.United BioSource GmbHMunichGermany
  5. 5.Clermont Consulting Group, LLCCharlestonUSA

Personalised recommendations