Real-World Data and Randomised Controlled Trials: The Salford Lung Study
Traditional efficacy double-blind randomised controlled trials (DBRCTs) measure the benefit a treatment produces under near-ideal test conditions in highly selected patient populations; however, the behaviour of patients and investigators in such trials is highly controlled, highly compliant and adherent, and non-representative of routine clinical practice. Pragmatic effectiveness trials measure the benefit a treatment produces in patients in everyday “real-world” practice. Ideally, effectiveness trials should recruit patients as similar as possible to those who will ultimately be prescribed the medicine, and create freedom within the study design to allow normal behaviours of patients and healthcare professionals (HCPs) to be expressed. The Salford Lung Study (SLS) was a world-first, prospective, phase III, pragmatic randomised controlled trial (RCT) programme in patients with chronic obstructive pulmonary disease and asthma to evaluate the effectiveness of a pre-licensed medication (fluticasone furoate/vilanterol) in real-world practice using electronic health records and through collaboratively engaging general practitioners and community pharmacists in clinical research. The real-world aspect of SLS was unique, requiring careful planning and attention to the goals of maximising the external validity of the trials while maintaining scientific rigour and securing suitable electronic processes for proper interpretation of safety data. Key learnings from SLS that may inform the design of future pragmatic effectiveness RCTs include: (1) ensuring the trial setting and operational infrastructure are aligned with routine clinical care; (2) recruiting a broad patient population with characteristics as close as possible to patients in routine clinical practice, to maximise the generalisability and applicability of trial results; (3) ensuring that patients and HCPs are suitably engaged in the trial, to maximise the chances of successful trial delivery; and (4) careful study design, incorporating outcomes of value to patients, HCPs, policymakers and payers, and using pre-planned analyses to address scientifically valid research hypotheses to ensure robustness of the trial data.
KeywordsAsthma Chronic obstructive pulmonary disease Effectiveness Fluticasone furoate/vilanterol Primary care Randomised controlled trial Real world Salford Lung Study Usual care
Key Summary Points
Traditional efficacy double-blind randomised controlled trials (DBRCTs) are considered the “gold standard” study design for assessing the efficacy and safety of new medicines; however, their conduct in highly selected patient populations and in highly controlled settings limits the generalisability of their findings to patients seen in everyday clinical practice.
Pragmatic effectiveness trials conducted in the routine clinical care setting allow for the evaluation of the effectiveness of medicines in the presence of real-world factors related to patients, actual medication use, and healthcare systems, thus providing a more complete picture of the benefit/risk profile of a medicine to support healthcare decision-making.
In this article, we discuss the key features and advantages/limitations of pragmatic effectiveness randomised controlled trials (RCTs) compared with traditional efficacy DBRCTs, using the Salford Lung Study (SLS) programme as an illustrative example.
SLS was the world’s first prospective, phase III, pragmatic RCT to evaluate the effectiveness of a pre-licensed medication in a primary care setting using electronic health records and through collaboratively engaging general practitioners and community pharmacists in clinical research.
Key learnings from SLS that may help inform the design of future pragmatic effectiveness RCTs include: (1) ensuring that the trial setting and operational infrastructure are aligned with routine clinical care; (2) recruiting a broad population of patients with characteristics as close as possible to patients seen in routine clinical practice, to maximise the generalisability and applicability of the trial results; (3) ensuring that patients and local healthcare professionals (HCPs) are suitably engaged in the trial, to maximise the chances of successful trial delivery; and (4) careful study design, incorporating outcomes of value to patients, HCPs, policymakers and payers, and using pre-planned analyses to address scientifically valid research hypotheses to ensure robustness of the acquired data.
Pragmatic randomised effectiveness trials are designed to evaluate medicines in the “real-world” setting across a broad patient population [8, 9] (Fig. 1) and offer the opportunity to address issues faced by patients and healthcare professionals (HCPs) on a daily basis [3, 6], while retaining the benefits of random treatment allocation. Randomised effectiveness trials can complement traditional efficacy DBRCTs by filling the evidence gaps surrounding patient and physician experience, treatment adherence, and healthcare resource utilisation (HRU) and care costs, all of which are key to informing healthcare decision-making .
The phase IIIb, pragmatic effectiveness Salford Lung Study (SLS) RCT programme was designed to evaluate a pre-licensed inhaled corticosteroid/long-acting β2-agonist combination, fluticasone furoate/vilanterol (FF/VI), in patients with chronic obstructive pulmonary disease (COPD) and asthma in UK primary care . SLS was a world first, embracing the novel use of electronic health records (EHRs) to comprehensively enrol a broad spectrum of patients from across Salford and surrounding areas. Here, we discuss key features and advantages/limitations of pragmatic effectiveness RCTs versus traditional efficacy DBRCTs (focusing on respiratory trials), using SLS as an illustrative example. We also describe key learnings from SLS and discuss how these might help inform the design of future pragmatic effectiveness RCTs and respiratory treatment guidelines/healthcare policies.
Efficacy and Effectiveness RCTS: Overview and Major Differences
Comparison of traditional efficacy DBRCTs and pragmatic effectiveness RCTs
Traditional efficacy DBRCTs
Pragmatic effectiveness RCTs
Strict inclusion/exclusion criteria
Patients with significant comorbidities and severe disease typically excluded
Highly selected (“ideal”) patient population
Limited relevance to patients in routine clinical care setting
Good inhaler technique mandated
High adherence mandated
Broad inclusion criteria, minimal exclusion criteria
Patients with comorbidities and severe disease included
Broad, heterogeneous patient population
Greater relevance to patients in routine clinical care setting
Variable inhaler technique
Variable (and frequently poor) adherence
Designed to test efficacy under near-ideal conditions (i.e. where confounding factors are minimised)
Provide data on the efficacy and safety of a medicine, albeit in a highly controlled setting
Typically recruited in ambulatory care/outpatient centres
Randomisation (± stratification) and masking (e.g. double-blind, double-dummy) to limit bias due to systematic differences between treatment groups
Can assess experimental medicine versus a placebo or “gold standard” comparator
Treatment per protocol; generally treatment modifications are not permitted
Frequent study visits/monitoring
Adherence to treatment actively monitored and encouraged
Designed to test effectiveness in the presence of real-world factors
Provide data on the overall treatment strategy in a real-world setting
Conducted in routine clinical practice in primary care; patient management reflective of usual clinical care
Randomisation (± stratification) to limit bias due to systematic differences between treatment groups; typically open-label in design
Usually assess experimental medicine against usual care or established standard-of-care
Treatment modifications permitted based on physician’s clinical opinion
Few mandatory study visits; limited disruption to patients’ normal routine
No monitoring or active encouragement of treatment adherence; patients’ health behaviours as normal
Data have high internal validity, limited external validity
Endpoints often designed to enable regulatory approval/licensing
Data have high external validity
Often include additional endpoints of interest, e.g. healthcare resource utilisation, patient-reported outcomes
Treatment effect in the real world has to be estimated
Culturally accepted as most informative and therefore transferable, although external validity is weak
Data more generalisable to the overall disease population
Effect of healthcare systems and access to medicines and cultural factors may need to be considered
Examples of patient confounding/complicating factors in the real-world clinical practice setting
Existing diagnosis (often pragmatic or clinical)
Access to medical care
Non-adherence to prescribed medication (over- or under-treating)
Poor inhaler techniquea
Poor compliance with treatment advice and follow-up
Comorbidities/coexisting medical conditions
Cigarette smoking and/or recreational drug use
Variability in health literacy
Diversions and distractions caused by life and social events, crises, shift-work patterns, accidents and injuries
Almost all of these factors are excluded/altered by the strict eligibility criteria, conduct, and need for protocol compliance in traditional efficacy DBRCTs. Consequently, only a low proportion of primary care patients with COPD and asthma would be eligible for participation in typical efficacy trials [4, 11, 12, 13, 14] and the relevance of DBRCT results to patients in routine practice is limited [1, 7]. Effectiveness RCTs can therefore supplement data from efficacy DBRCTs by providing a more complete picture of the benefit/risk profile of a medicine to support healthcare decision-making.
Traditional efficacy DBRCTs versus pragmatic effectiveness RCTs: impact of study design aspects on acquired data
Process of care
Protocols demand atypical conduct from patients and HCPs, often requiring frequent, rigorous and prolonged study assessments
Optimal treatment compliance and inhaler device technique is strongly encouraged
Treatment pathway is strictly defined, and deviations from prespecified treatment may result in patients being withdrawn from study (restricting freedom of choice around patient care)
Rigorous assessment in efficacy DBRCTs allows for robust data acquisition and for patient safety to be closely monitored
By strongly encouraging optimal treatment compliance, DBRCTs are able to provide an accurate profile of treatment efficacy and safety for an intended dose
A downside of the highly controlled environment in which efficacy DBRCTs are conducted is the impact on the behaviour of both patients and physicians, which does not truly reflect the characteristics of a medicine when used in the real-world setting
In pragmatic effectiveness RCTs:
Care is aligned with that received in routine clinical practice, with minimal scheduled study visits/assessments and minimal disruption to patients’ everyday lives
Physicians have freedom of choice to modify patients’ treatment as deemed necessary
Study design permits freedom of choice around patients’ care and allows the normal behaviours of patients and HCPs to be expressed; thus, acquired data may be more generalisable to patients seen in everyday clinical practice
Data from pragmatic effectiveness trials more likely to be confounded by extraneous variables, not controlled for by virtue of the trial being conducted in a real-world setting
In efficacy DBRCTs, patients’ data are typically entered into eCRFs by investigators/dedicated study team members during or following scheduled study visits
The challenge for pragmatic effectiveness trial design is to adequately balance the delivery of highly accurate and complete data while minimising the level of interference that data entry and verification pose to routine practice. Effectiveness trials may rely on data extraction from patients’ EHRs or other spontaneous reporting systems, as well as eCRFs for data capture
In contrast to traditional efficacy DBRCTs, where neither investigator nor patient has knowledge of the assigned treatment, pragmatic effectiveness RCTs are typically open-label in design
Compared with eCRFs, EHR-based data capture has advantages in terms of allowing for remote data collection in real time (avoiding recall or transcription bias) and in providing the opportunity for long-term follow-up of the trial population after the study has completed
Disadvantages of EHR-based data capture include missing data, lack of representation of endpoints of interest, and potential issues with accessibility for research purposes in certain regions/countries
Conducting effectiveness trials in routine clinical care means that some investigators may be inexperienced in clinical research and processes must be implemented to manage this to ensure robust data collection (accordingly with resource, logistics, and cost implications)
An open-label study design creates the potential for bias in the acquired data due to behavioural effects that arise as a consequence of knowledge of the treatment that has been administered, e.g.:
Physicians may be biased in assigning causality of AEs to an investigational medicine rather than to a well-established standard-of-care treatment where common AEs are well known—this could have a positive or negative effect on AE reporting
There may be increased HCP/patient vigilance with a new medicine, resulting in higher rates of healthcare contacts than with a more familiar treatment option
Patients may merely have a preference for, and revert to taking, a more familiar treatment, thus impacting on adherence to the assigned study treatment
The Hawthorne effect—where individuals modify an aspect of their behaviour in response to their awareness of being observed—may apply and confound data collected in both traditional efficacy DBRCTs and pragmatic effectiveness RCTs; however, this effect may be less likely in effectiveness trials that are designed to minimise disruption to everyday clinical care
Trial eligibility criteria/patient population
The stringency of eligibility criteria for traditional efficacy DBRCTs versus pragmatic effectiveness RCTs will dictate the nature of the patient populations recruited, and this can have a profound effect on the data collected during such trials
Patients in traditional efficacy DBRCTs are usually recruited in ambulatory care/outpatient centres, tend to be healthier than the non-trial disease population, and often participate in multiple trials
In high-recruiting research centres, investigators may hold a database of patients who are “ready to enrol”. Such patients will meet trial inclusion/exclusion criteria as standard, are quick to learn and maintain excellent inhaler technique, intellectually capable, highly compliant with study protocols and procedures, and are familiar with a range of treatment devices and study assessments
Pragmatic effectiveness RCTs seek to recruit a broad participant population with characteristics as similar as possible to patients who will eventually be prescribed the medicine in routine clinical practice. To achieve this, such trials typically employ minimal inclusion/exclusion criteria
The highly selected patient populations in traditional efficacy DBRCTs allow for testing of the efficacy of a medicine under conditions where confounding factors are minimised; thus, data have high internal validity
The strict entry criteria/requirements for adherence to protocol in traditional efficacy DBRCTs may preclude otherwise eligible patients from participating (e.g. patients from deprived areas, for reasons including difficulties with/costs of getting to the research site, or working and family commitments). This has led to the concept of “persistent participators” in efficacy DBRCTs—a population that is non-representative of patients treated in real-world practice
Findings from efficacy DBRCTs have limited applicability/generalisability of the acquired data to the wider disease population—low external validity
An example would be the collection and interpretation of AEs. In a pragmatic effectiveness trial, by virtue of enrolling a broader population of patients (including those with comorbidities and more severe disease), it is likely that a higher incidence and/or wider variety of AEs will be reported than in an efficacy DBRCT evaluating the same medicine
These limitations are somewhat circumvented in pragmatic effectiveness RCTs
In pragmatic effectiveness RCTs, recruiting a broad participant population may introduce additional variability to the data set, and a greater number of patients may need to be enrolled to power the study to demonstrate treatment effect, compared with traditional efficacy RCTs
Endpoints in traditional efficacy DBRCTs (registrational trials in particular) are often dictated by outcomes of interest to regulatory authorities, often require frequent assessments and diary cards/electronic diaries, and serve as constant reminders of disease state and treatment response
In pragmatic effectiveness RCTs, it is desirable to select endpoints that are relevant to patient-centric goals for treatment and that physicians routinely use to assess patients and make treatment decisions, so as to optimise external validity and transferability of the data, and enhance value to clinicians, payers, and policymakers
In respiratory trials, for example, such endpoints would include exacerbations, hospitalisations, mortality, validated patient-reported outcomes, and quality-of-life measures
In effectiveness trials, it is desirable to minimise the impact of study assessments by selecting endpoints and a frequency of measures that ideally can be gathered with little or no impact on the patient or HCP, and where observer bias is controlled—crucial for an open-label study design
In pragmatic effectiveness RCTs, certain endpoints may be precluded because of the intensive monitoring that would be required (e.g. serial lung function, daily diaries)
In routine clinical practice, a patient’s treatment will be adjusted at the discretion of the treating physician
Treatment modifications are rarely permitted in traditional efficacy DBRCTs, but are allowed (albeit with potential restrictions) in pragmatic effectiveness RCTs, with implications for the analysis and interpretation of the study data (particularly important for safety evaluation)
In effectiveness RCTs where treatment can be modified, careful consideration must be taken as to whether specific study endpoints will be evaluated as ITT (i.e. according to randomised treatment group) or by actual treatment received
Effectiveness endpoints will typically be analysed as ITT, which is equivalent to comparing the treatment strategies being investigated in the effectiveness RCT
Safety data ought to be presented both by randomised treatment group and also by actual treatment
In traditional DBRCTs, efficacy and safety endpoints are typically analysed according to the ITT principle. Interpretation of data is more straightforward; randomisation group equates to treatment group and data can be analysed accordingly (e.g. safety events can be attributed to randomised treatment)
In effectiveness RCTs, analysis by actual treatment received allows for assessment of true exposure risk of a medication
In effectiveness RCTs, variation in the treatment being taken produces an additional level of complexity for data analysis, in that it precludes the direct comparison of randomisation groups from being equated with the safety of treatment A compared with treatment B
The Salford Lung Studies in COPD and Asthma: What Were Their Novel “Real-World” Aspects?
The SLS programme comprised two concurrent, 12-month, open-label, phase IIIb pragmatic RCTs designed to evaluate the effectiveness and safety of initiating once-daily inhaled FF/VI, compared with continuing usual care (UC) in patients with COPD or asthma in UK primary care [10, 15, 16, 17, 18]. All patients provided written informed consent for participation in SLS and the trials were conducted in accordance with the International Conference on Harmonisation Good Clinical Practice guidelines and the provisions of the 2008 Declaration of Helsinki. The trial protocols were approved by the National Research Ethics Service Committee North West, Greater Manchester South (approval numbers 11/NW/0798 and 12/NW/0455).
When SLS commenced, a full regulatory submission for FF/VI was under consideration by the European Medicines Agency that was based on extensive efficacy and safety data from completed RCTs . SLS was conducted to meet the need for effectiveness data for FF/VI to complement existing evidence from efficacy RCTs. SLS was the first study in the UK to take advantage of joint advice from the Medicines and Healthcare products Regulatory Agency (MHRA) and The National Institute for Health and Care Excellence (NICE). MHRA, responding positively to NICE’s enthusiasm for pragmatic data in a broad community, approved the study design with a pre-licensed medicine—a key factor enabling the study to proceed.
Maximising External Validity
Careful design and much background work went into ensuring that the delivery of SLS was aligned with routine care (e.g. normality of medicines supply; patient and HCP behaviours consistent with everyday clinical practice; interplay between patients, GPs, pharmacists in the community setting), while ensuring that the study conduct and data collection met the requirements of a phase IIIb RCT.
Collecting patients’ data via EHRs allowed us to measure the SLS COPD primary effectiveness endpoint (moderate/severe exacerbations) through surrogates triggered in the EHR (prescription of antibiotics and/or systemic corticosteroids or hospital admissions/visits associated with a respiratory cause ). This ensured that patients and doctors were essentially unaffected by the study for the entire follow-up period. In SLS asthma, the same primary effectiveness endpoint was not feasible because of the expected low frequency of exacerbations based on a pilot study (recruitment numbers would have been too high to achieve the required statistical power); instead, response on Asthma Control Test (ACT) was selected as the primary outcome. ACT was completed by patients at the baseline (randomisation) and week 52 (end of study) scheduled visits, and was additionally administered via telephone at weeks 12, 24, and 40 . Processes were implemented to ensure that ACT was administered with minimal interference to normal care (e.g. GPs were aware of ACT scores at baseline, but not thereafter; telephone ACT was administered by a study nurse blinded to treatment and who was trained not to provide advice to the patient, except under life-threatening circumstances).
All HCPs involved in SLS (GPs, nurses, pharmacists, and their staff) were trained to allow routine clinical practice to proceed, although consultation rates were higher in the FF/VI randomisation group than the UC group during the first 12 weeks [20, 21], predominantly for non-respiratory reasons, suggesting that GPs and patients did undergo an initial familiarisation period with the new therapy. Importantly, by extracting HRU data directly from EHRs, we were able to obtain a complete picture of the burden associated with COPD and asthma without bias due to recall or transcription of data, and were able to demonstrate a disproportionately high resource use for non-COPD/-asthma reasons [20, 21].
This “hands-off” approach really allowed normal patient and HCP behaviours to play out in SLS—quite unusual for a phase III trial, and a very positive aspect of the trial design, as it provides valuable information about how FF/VI performs when used in routine practice.
Maintaining Scientific Rigour
Through its prospective design, baseline randomisation/stratification procedures, and extensive a priori analysis plan, SLS achieved the scientific rigour characteristic of a traditional efficacy RCT. Much consideration went into the decision to allow asymmetric treatment modification in the trial design and the subsequent impact this would have on the data analyses (analysis by randomised treatment group or by actual treatment).
Furthermore, for the purpose of statistics and programming, the sponsor remained blinded to study treatment while the trial was ongoing and was only unblinded after all data had been collected and the study database had been locked, thus mimicking the approach taken in a typical DBRCT.
Safety Data Collection
SLS commenced with a pre-licensed medicine and our intention was to vigilantly collect and evaluate safety events in real time through patients’ EHRs. In recruiting a population inclusive of patients with comorbidities and severe disease, we anticipated that the study would accrue a large volume of safety data and that patients would experience multiple serious adverse events (SAEs) that would not be seen in trials where such comorbidities are excluded. An innovative approach for identifying potential AEs without interfering with patients’ normal routines was essential. The SLS safety data collection process has been published previously , but a key aspect of this was the creation of a consultant-led specialist safety team, who were alerted to review potential safety events in real time.
Over 7000 patients participated in SLS and both trials met their primary effectiveness endpoints, demonstrating the benefit of initiating FF/VI versus continuing UC [17, 18]. In SLS COPD, there was a statistically significant reduction for FF/VI versus UC in the mean annual rate of moderate/severe exacerbations, without increased risk of SAEs . In SLS asthma, the odds of patients being ACT responders (ACT total score ≥ 20 and/or improvement from baseline ≥ 3) at week 24 were significantly higher for FF/VI versus UC, without increased risk of SAEs . Consistent benefit of FF/VI over UC has also been demonstrated for various other endpoints, as demonstrated in secondary analyses of SLS COPD and asthma [20, 21, 22, 23, 24, 25, 26, 27, 28, 29].
SLS Approach to Effectiveness RCTS: Advantages and Limitations
Conducting effectiveness RCTs such as SLS in routine primary care requires access to the patient population of interest and good infrastructure, operational management, training/good clinical practice, and site engagement. It could be argued that such requirements could preclude the conduct of similar effectiveness studies in other geographical locations .
As SLS comprised phase IIIb trials of a pre-licensed medication requiring detailed safety monitoring, the studies were time- and resource-intensive to design and the operational logistics were highly complex. There exists a perception that effectiveness studies are simpler to design and less expensive to implement than traditional efficacy RCTs, but our experience with SLS suggests quite the opposite; however, this may not necessarily reflect requirements for all real-world effectiveness studies, which should be designed on a case-by-case basis.
SLS commenced with a phase of pre-licensed FF/VI in the UK primary care setting and this had implications for the acquired trial data set, which should be considered when designing similar future effectiveness RCTs. For example, in SLS COPD, higher rates of all-cause primary care contacts for FF/VI versus UC were observed in the first 3 months of the study, which may have been driven by additional scrutiny of the then-unlicensed FF/VI . Patients and physicians were allowed to modify treatment, and this required an additional level of consideration around the underlying reasons for modification and had implications for data analysis. In SLS, we were able to determine actual HRU and care costs (as opposed to the usual modelled costs) for patients with COPD and asthma, which is highly relevant for routine clinical practice in the UK. Furthermore, patients from deprived areas, who may be ineligible for, or unwilling to participate in, traditional phase III RCTs were recruited to the SLS. Salford, UK, is an urban location with areas of deprivation. Asthma patients were equally represented in deprivation categories, but COPD patients are over-represented in deprived areas. In an analysis of outcomes by deprivation, we found that deprivation did not impact the main outcomes of the SLS trials, thus supporting the recruitment of participants from all socioeconomic strata to provide data that are generalisable to routine clinical practice [26, 27, 28].
To reflect the routine clinical care setting, the UC randomisation group comprised many different inhaled maintenance therapies as the comparator for initiation of FF/VI, which could also be varied over the course of the study. Caveats of this design aspect include a limited capacity to prospectively evaluate FF/VI against a specific UC treatment, and inability to equate a UC option with an established standard-of-care in COPD or asthma. Allowing broad UC therapies and treatment modifications in SLS has limitations for data analysis (precludes direct, head-to-head comparisons of FF/VI versus UC, or of FF/VI versus one particular UC treatment), but advantages in that analysis by actual treatment could be conducted (e.g. safety reporting).
How Might SLS Inform HCPS and Decision-Makers?
For most newly approved medicines, evidence from efficacy DBRCTs is insufficient to fully guide physicians in choosing optimal treatment for their patients. Pragmatic effectiveness trials can fill the gap by providing data on the overall treatment strategy in routine clinical practice while maintaining the strength of an RCT  and are a valid option for addressing issues that patients, clinicians and policymakers face on a daily basis. Knowledge of the overall effectiveness of a medicine in the intended patient population, taking into account real-world factors related to patients, actual medication use, and healthcare systems, will ultimately help HCPs make more-informed treatment choices for their patients. Furthermore, this study is the first to provide HCPs with answers on how initiating FF/VI (having been treated previously with other medicines) may impact exacerbation rates and other outcomes versus continuing on those other medicines.
In respiratory clinical care, there has tended to be a focus on symptom management; however, patients are often more concerned with how their symptoms make them feel and the impact of symptoms on their everyday lives . Health care is increasingly adopting a patient-centric approach, which considers patients’ perspectives regarding the impact of disease and its treatment. Clinical trials should, therefore, incorporate appropriate patient-reported outcomes (PROs) and quality of life (QoL) outcomes in their design. In SLS asthma, several PRO effectiveness endpoints were prospectively assessed (including ACT, Asthma Quality of Life Questionnaire, Work Productivity and Impairment Questionnaire: Asthma, and the EuroQol 5-Dimensions 3-Levels questionnaire). Initiation of FF/VI was associated with consistent benefits in PROs versus continuing UC, demonstrating that the observed improvement in asthma control (measured by ACT) for FF/VI translates into patient-perceived benefits in health-related QoL . Follow-up interview-based studies conducted in subsets of patients who completed SLS have also provided important additional findings on patient-centred outcomes relevant to respiratory care in routine clinical practice [24, 32, 33].
For healthcare policy decision-makers, data from effectiveness trials can provide a more balanced view of the overall benefit/risk of a medicine, including HRU and cost-effectiveness—critical factors for consideration by resource-limited health services . However, despite Health Technology Assessment groups expressing a desire to see more pragmatic studies describing effectiveness, for many, their dossier restrictions do not allow unblinded studies to be included in their assessments. There have been few studies like SLS in phase III and so payers and regulators have little experience of the nuances of such data sets and how to respond to them within their regulation. However, unless they do so, this will have a negative impact on sponsors’ willingness to fund effectiveness studies.
Learnings From SLS: How Can These Be Applied to Future RCTS?
SLS design aspects—issues requiring greater effort/input to resolve compared to DBRCTs, based on opinion of SLS investigators and operational staff
SLS design aspect
Recruitment of GP investigators
Almost all GPs had not previously taken part in clinical trials
GPs are busy and do not have space in their practices or the time to conduct clinical studies
Appointed GP ambassadors who recruited practices and “sold” the value of the study in their locality
Recruited a team of approximately 50 research nurses to support GP investigators
Patient recruitment was initially slow in the SLS
No previous experience of recruiting in this environment
GPs very busy, limited time for recruitment
Personal contact from patient’s own GP critical for recruitment
Adopted a project management approach to patient recruitment and consent, local advertising and pharmacy approaches to patients
Medicines supply and GCP management, pharmacy involvement
Pharmacies are usually in competition for business
Few pharmacists had prior clinical research experience
Requirement for pharmacy study-specific SOPs
Created a pharmacy steering group to oversee training and SOP development, endorsement of pharmacy chain superintendent pharmacists 
Study endpoints, analysis and powering
No prior data on which to base our power calculations, our statisticians had not previously dealt with studies like this
Endpoints had to be of value but be measured with minimal interference to patient care
Considerable debate with the SLS Scientific Committee to decide on endpoints
Numerous reviews of statistical plans and endpoints—very different to a DBRCT, where our confidence is higher
Randomisation and stratification of patients by asthma severity
Since very few baseline investigations were performed, it was impossible to stratify according to lung function or usual measures of disease severity
Novel approaches to stratification were developed, such as the issuing of a “dummy prescription” by GP at baseline assessment, which allowed us to stratify according to intended treatment
Electronic collection of pharmacy dispensing data
Pharmacy systems in the UK are primarily stock control and labelling systems, and many different systems are used
Bespoke solution created, which took an incredible amount of work
Safety monitoring to GCP standards
This had not been done previously and as we were using EHR triggers to detect certain study endpoints and safety signals, we had to think completely differently to safety monitoring in a DBRCT
Worked with the sponsor’s pharmacovigilance team to build a robust safety system
Had a consultant physician-led safety team (two physicians and four nurses) monitoring signals on a daily basis 
Data quality and standards
Use of EHRs and effort required to ensure that data was of high enough standard to meet GCP requirements
Sponsor’s experience from data governance was a need to take EHR to a higher level
Implemented a much higher than usual investment in data cleaning and data quality
The EHR needed an additional programme to collect relevant (and delete irrelevant) data. Level of quality and governance not adequate to start with and required an audit → fix → audit approach
Setting and Infrastructure
The underlying operational infrastructure was the key to delivering two large phase IIIb trials evaluating a pre-licensed medicine out of local GP practices and pharmacies in and around Salford, UK. Salford was an ideal location for the trials owing to its relatively static population served by a single hospital with an integrated, real-time EHR connection with surrounding primary care practices and linking with patient-level prescription information (Salford Integrated Record, SIR) . A bespoke information technology infrastructure was developed by NorthWest eHealth to extract data from the SIR for the purpose of effectiveness research [10, 19]. Over 2100 GPs, nurses, and pharmacy staff in and around Salford were trained in good clinical practice in SLS.
Critical to the delivery of SLS was the unique involvement and collaboration of community pharmacies. A key component of the integrity of the effectiveness design was maintaining the normality of repeat prescribing and dispensing in a situation where a pre-licensed medicine was being evaluated. Extensive training and process development permitted pharmacists/pharmacy staff to participate in a phase III trial, despite most having no prior clinical research experience. SLS demonstrated that pharmacies normally competing for prescription business can work in a collaborative and supportive manner for the benefits of patients. Specific aspects of how pharmacy involvement was successfully achieved in the SLS will be the subject of another publication .
Patient Population: Inclusiveness and Applicability
SLS showed that by employing limited eligibility criteria, it is possible to recruit to a phase IIIb effectiveness RCT a broad population of patients that are representative of those in everyday clinical practice, including from socioeconomically deprived areas.
SLS COPD enrolled approximately half of all eligible patients with COPD in the target geographical area . These patients had a high disease burden and more symptoms, more frequent exacerbations, more comorbidities, and more SAEs/pneumonia SAEs compared with patients in large, registrational, efficacy RCTs in COPD . Furthermore, over half of SLS COPD patients were categorised in the most deprived quintile by postcode [26, 28]. Notably, only at most 30% of SLS COPD patients would have been eligible for a typical regulatory phase III COPD exacerbation study , definitively demonstrating that COPD patients enrolled in traditional efficacy RCTs are not representative of patients in primary care.
The applicability of SLS findings to patients in routine clinical care is supported by a previous study demonstrating similarity in the characteristics of SLS COPD patients with other primary care patient populations across Europe . Further support comes from a recent observational study demonstrating that patients in the SLS COPD UC group were similar to a matched cohort of non-trial COPD patients in England from the Clinical Practice Research Datalink database . Evidence for a Hawthorne effect was observed, with a higher frequency of COPD exacerbations recorded in SLS patients than in non-trial patients; however, the largest effect was observed through behavioural changes in patients and GP coding practices . SLS data have, therefore, contributed to the development of novel methods for evaluating the presence of an operating Hawthorne effect for future effectiveness trials conducted in everyday clinical practice.
Patient and HCP Engagement
Our experience with SLS underscores the importance of carefully designing pragmatic effectiveness RCTs to maximise chances of success in routine practice while ensuring operational feasibility. Engaging patients and HCPs in effectiveness research is extremely challenging. Initially, patient enrolment was slow in the SLS and we had to revisit our strategies for recruitment. The key was the involvement of patients’ own GPs in recruitment and obtaining of consent. Following on from SLS, additional qualitative research has been conducted to understand the drivers for patient and HCP engagement in the studies and how participation in future effectiveness trials might be enhanced . Though key learnings from SLS will be the subject of a separate publication, our findings should be highlighted around the overall positive experience of patients and HCPs who participated in SLS and the importance of (1) local advertising to raise community awareness of study recruitment; (2) site/investigator engagement, ensuring that through extensive training on Good Clinical Practice, the study design and delivery was aligned with routine care; (3) provision of research nurse support at study sites, which was key to study delivery; (4) ease and convenience of study assessments; and (5) a need for improved study results dissemination .
SLS incorporated outcomes of interest outside those typically included in traditional respiratory efficacy DBRCTs (e.g. HRU/care costs, PROs, patient exit interviews), which has provided valuable supporting data for the benefit of initiating FF/VI versus continuing UC in patients with COPD and asthma in routine practice [20, 21, 22, 23, 24, 32, 33].
Furthermore, allowing asymmetric treatment modifications in the study design necessitated a novel approach to safety evaluation: highly comprehensive safety analyses were conducted both by randomised treatment group and by actual treatment at the time of an event.
The scientifically rigorous collection of real-world data in SLS offers major opportunities for future studies examining new research questions.
The real-world design aspect of SLS was unique, requiring careful planning and attention to the goals of maximising external validity while maintaining scientific rigour and securing suitable electronic processes for safety data collection. Key learnings from SLS that may help inform the design of future pragmatic effectiveness RCTs include: (1) ensuring that the trial setting and operational infrastructure are aligned with routine clinical care; (2) recruiting a broad population of patients with characteristics as close as possible to patients seen in routine clinical practice, to maximise the generalisability and applicability of the trial results; (3) ensuring that patients and HCPs are suitably engaged in the trial, to maximise the chances of successful trial delivery; (4) careful study design, incorporating outcomes of value to patients, HCPs, policymakers and payers; and (5) using pre-planned analyses to address scientifically valid research hypotheses to ensure robustness of the acquired data.
The authors thank the participants of the Salford Lung Studies.
The Salford Lung Studies in COPD and asthma were funded by GlaxoSmithKline plc. (HZC115151/ClinicalTrials.gov identifier NCT01551758 and HZA115150/ClinicalTrials.gov identifier NCT01706198). All authors had full access to all of the data from the SLS COPD and asthma trials and take complete responsibility for the integrity of the data and accuracy of the data analyses.
Medical Writing, Editorial, and Other Assistance
Editorial support in the development of this article (in the form of writing assistance, collating author comments, assembling tables/figures, grammatical editing, and referencing) was provided by Emma Landers, PhD, at Gardiner-Caldwell Communications (Macclesfield, UK) and was funded by GlaxoSmithKline plc.
All named authors meet the International Committee of Medical Journal Editors (ICMJE) criteria for authorship for this manuscript, take responsibility for the integrity of the work as a whole, and have given final approval to the version to be published.
David A. Leather has employment with, and stock/share ownership in, GlaxoSmithKline plc. Loretta Jacques has employment with, and stock/share ownership in, GlaxoSmithKline plc. Rupert Jones has grants from AstraZeneca and GlaxoSmithKline plc.; personal fees from AstraZeneca, Boehringer Ingelheim, Chiesi, GlaxoSmithKline plc., Novartis and Nutricia. Ashley Woodcock has received speaker fees and expenses from GlaxoSmithKline plc.; advisory board member for, and expenses from, Chiesi; personal fees from Axalbion and Reacta Biotech; Chair of Medicines Evaluation Unit. Jørgen Vestbo has received research grant from Boehringer Ingelheim; honoraria from AstraZeneca, Boehringer Ingelheim, Chiesi, GlaxoSmithKline plc., and Novartis for consulting/presenting. Mike Thomas has received speaker fees from Aerocrine, GlaxoSmithKline plc., and Novartis; honoraria from Aerocrine, Boehringer Ingelheim, GlaxoSmithKline plc., MSD, Novartis, and Pfizer. Mike Thomas is a recent a member of the BTS SIGN Asthma guideline steering group and the NICE Asthma Diagnosis and Monitoring guideline development group. Rupert Jones was supported by the National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care South West Peninsula (NIHR CLAHRC South West Peninsula). Ashley Woodcock and Jørgen Vestbo are supported by the National Institute of Health Research Biomedical Centre in Manchester (NIHR Manchester BRC).
Compliance with Ethics Guidelines
This review article is based on the previously reported Salford Lung Studies in COPD and asthma. All patients provided written informed consent for participation and the trials were conducted in accordance with the International Conference on Harmonisation Good Clinical Practice guidelines and the provisions of the 2008 Declaration of Helsinki. The trial protocols were approved by the National Research Ethics Service Committee North West, Greater Manchester South (approval numbers 11/NW/0798 and 12/NW/0455). Our article does not report on any new studies with human participants or animals.
Data sharing is not applicable to this article as no data sets were generated or analysed during the current study.
Real-world data and randomised controlled trials: The Salford Lung Study. A video (MP4 233891 kb)
- 18.Woodcock A, Vestbo J, Bakerly ND, et al. Salford Lung Study Investigators. Effectiveness of fluticasone furoate plus vilanterol on asthma control in clinical practice: an open-label, parallel group, randomised controlled trial. Lancet. 2017;390(10109):2247–55.Google Scholar
- 21.Browning D, Bakerly ND, New JP, et al. Healthcare resource utilisation (HRU) in the Asthma Salford Lung Study (SLS asthma). Eur Respir J. 2018;52:PA5036. Poster presented at the 2018 European Respiratory Society International Congress.Google Scholar
- 22.Woodcock AA, Boucot I, Browning D, et al. The clinical effectiveness of fluticasone furoate/vilanterol (FF/VI) in everyday clinical practice: patient-reported outcomes in the Salford Lung Study (SLS COPD). Am J Respir Crit Care Med 2017;195:A5476. Poster presented at the 2017 American Thoracic Society International Conference.Google Scholar
- 25.Jacques L, Bakerly ND, New JP, Svedsater H, Lay-Flurrie J, Leather DA. Effectiveness of fluticasone furoate/vilanterol versus fluticasone propionate/salmeterol on asthma control in the Salford Lung Study. J Asthma. 2018;16:1–10.Google Scholar
- 26.Jones R, Nicholls A, Browning D, et al. Deprivation in the COPD Salford Lung Study (SLS) is associated with higher healthcare costs but does not moderate the main outcomes. Thorax. 2017;72(Suppl 3):A96–97. Poster presented at the 2017 British Thoracic Society Winter Meeting.Google Scholar
- 27.Collier S, Jones R, Lay-Flurrie J, et al. Deprivation is associated with higher healthcare use in the Salford Lung Study in asthma (SLS asthma). Thorax. 2018;73 (Suppl 4):A120. Poster presented at the 2018 British Thoracic Society Winter Meeting.Google Scholar
- 28.Jones R, Nicholls A, Browning A, et al. Impact of socioeconomic status on participation and outcomes of the Salford Lung Studies. ERJ Open Research (in press).Google Scholar
- 34.Leather DA, Howard S, Haydock G, Stephens L. Community pharmacy: a crucial enabler in creating the effectiveness study environment in the Salford Lung Studies (Manuscript submitted to International Journal of Pharmacy Practice).Google Scholar
- 35.Driessen M, Boucot I, Müllerova H, et al. Patient demographics and clinical characteristics in the UK Salford Lung Study (SLS COPD) and the European ACCESS study. Eur Respir J. 2017;50:PA940. Poster presented at the 2017 European Respiratory Society International Congress.Google Scholar
- 37.Gemzoe K, Crawford R, Caress A, et al. Patient and healthcare professional (HCP; general practitioner [GP] and practice manager [PM]) insights to facilitate engagement in future effectiveness studies: learnings from the Salford Lung Studies (SLS). Am J Respir Crit Care Med. 2019;199:A6227. Poster presented at the 2019 American Thoracic Society International Conference.Google Scholar
Open AccessThis article is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, which permits any non-commercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc/4.0/.