Optimising recruitment into trials using an internal pilot
Recruitment to trials can be difficult. Despite careful planning and research that outlines ways to improve recruitment, many trials do not achieve their target on time and require extensions of funding or time.
We describe a trial in which an internal pilot with embedded qualitative research was used to improve recruitment processes and inform recruitment projections for the main trial. At the end of the pilot, it was clear that the sample size would not be met on time. Three steps were taken to optimise recruitment: (1) adjustments were made to the recruitment process using information from the qualitative work done in the pilot and advice from a patient and public involvement group, (2) additional recruiting sites were included based on site feasibility assessments and (3) a projection equation was used to estimate recruitment at each site and overall trial recruitment.
Qualitative work during the pilot phase allowed us to develop strategies to optimise recruitment during the main trial, which were incorporated into patient information packs, the standard operating procedures and training sessions with recruiters. From our experience of feasibility assessments, we developed a checklist of recommended considerations for feasibility assessments. For recruitment projections, we developed a four-stage projection equation that estimates the number of participants recruited using a conversion rate of the number randomised divided by the number screened.
This work provides recommendations for feasibility assessments and an easy-to-use projection tool, which can be applied to other trials to help ensure they reach the required sample size.
ISRCTN, ISRCTN92545361. Registered on 6 September 2016.
Consolidated Standards of Reporting Trials
National Health Service
National Institute for Health Research
Patient information leaflet
When designing trials, good estimates of recruitment targets are needed. This enables study timeframes and budgets to be properly calculated and enables practicalities, such as the number of study centres, to be planned for. Despite careful planning, many trials struggle to achieve their recruitment targets within the expected timeframe. In a survey of UK clinical trial units, this was found to be the most commonly listed inefficiency in trials . Numerous trials have stopped because they have not been able to recruit sufficient participants or have finished with fewer participants than planned . Extensions of time and funding are required in around half of publicly funded trials [3, 4, 5]. In recent years, research has addressed trial recruitment, which has led to recommendations of strategies that may influence or improve recruitment, such as telephone reminders, financial and non-monetary incentives, envelopes with handwritten addresses, addressing external influences, using a patient-centred approach, making interventions available only in the trial and a dedicated trial manager [3, 6, 7, 8, 9, 10]. Despite this, questions remain unanswered about how to improve recruitment [11, 12].
Many authors have published on the difficulties in recruiting patients to trials and achieving complete samples . Recruitment targets are often overly optimistic. Lasagna’s law—developed based on a trial in which only 100 patients were recruited from a potentially eligible population of over 8000 —states that investigators overestimate the number of patients who may be available for a trial. The same phenomenon is described in Meunch’s third law, which states that any recruitment estimate should be divided by ten . Despite much work that explores how to enhance recruitment, Lasagna’s law still holds true . Less attention has been paid to how to estimate realistic recruitment rates in the first instance. Pilot phases (either internal or external) in trials are of particular value to the estimation of recruitment rates and strategies and are part of good trial design and practice [17, 18]. Internal pilots are useful for bringing unexpected practicalities to light that may be seen only once a trial is in the delivery phase.
Because of recruitment challenges, trial teams often need to expand to more centres or seek a time or funding extension to facilitate recruitment to target, while others do not recruit enough patients to answer their question. This is an inefficient use of resources, including time, money and effort for patients, researchers and funders [19, 20]. The key to achieving recruitment targets is to set them realistically by conducting a thorough feasibility assessment and using the information from this to create projections. This can be done using information from an internal pilot study. This article presents a case study of a randomised controlled trial in which we used information from an internal pilot phase to improve recruitment processes, undertake site feasibility assessments and create a projection tool to inform implementation of changes to the trial processes. Methods for predicting recruitment can be done using statistical packages [21, 22], but many researchers and trial managers do not have the training, time or resources to utilise them.
Our aim is to provide an overview of implementable methods of optimising recruitment, which research teams can use to increase the likelihood that recruitment targets are met.
Case study: the STAR trial
This example describes the STAR trial, a randomised controlled trial that evaluated the clinical effectiveness and cost-effectiveness of a new care pathway for patients with chronic pain after a knee replacement. The trial was planned to run at four sites with a target sample size of 380. The full protocol has been published . Recruitment was projected equally between centres with an estimated recruitment rate of 3 patients per month for all four sites combined during a 6-month pilot and 15 patients per month thereafter for a total of 30 months. Chronic pain after a knee replacement affects about 20% of patients, and therefore around 1 in 5 patients undergoing a total knee replacement would be eligible for the STAR trial [24, 25, 26]. Screening processes for the trial were piloted before the trial started and demonstrated a conversion rate of 7.6%, which is around 7–8 randomised patients per 100 screened . The trial screening and recruitment processes are outlined in the flow chart in Additional file 1.
Adjustments were made to the recruitment process using information from qualitative work done in the pilot and advice from a patient and public involvement group.
Additional recruiting sites were included based on site feasibility assessments.
A projection equation was used to estimate recruitment at each site and overall trial recruitment.
Step 1: embedded qualitative research
Recent research has demonstrated the benefit of using qualitative methods to inform refinements to trial processes and to optimise recruitment and retention rates [28, 29]. As part of the work within the pilot, we employed qualitative methods to identify and understand the motivations and possible barriers to participation of our target patient population. We used a dynamic approach commensurate with the trial design and the aims of the qualitative work , which were to identify barriers to recruitment and retention, and to optimise the accessibility of trial information and informed consent. Our approach involved audio-recording recruitment consultations between patients and recruiters, and follow-up telephone interviews with participants after they had been randomised.
Some patients thought that the randomisation result reflected a calculated ‘need’ for the intervention based on their responses to the questionnaire. Further analysis showed that recruiters explained the use of questionnaires and entry of responses into a database just prior to explaining randomisation, so the link appeared natural.
During the recruitment consultations, one patient was unsure about the term ‘ongoing treatment’ in reference to a statement in the patient information leaflet (PIL) that patients’ ongoing treatment would not be affected.
One participant thought that they would be attending the clinic on multiple occasions as part of a group, which concerned them because of work commitments.
Some participants experienced problems when selecting answers to questions in the online version of the outcome questionnaire.
Based on this information, the project team made the following changes to the recruitment process, PIL and questionnaires. We set up a working group to develop an improved explanation in the PIL of randomisation and the need for fair comparisons in trials. Moreover, we updated the standard operating procedures for recruiters to ensure that they made a clear distinction between completion of the questionnaire and the randomisation procedure. We expanded the description of usual care provided in the PIL to ensure that patients understood that all participants, regardless of group allocation, would still be able to access medical care as usual. We clarified in the PIL that patients would attend only one clinic appointment. Finally, the online version of the questionnaire was modified to ensure that participants could select options more easily.
During training sessions with recruiters, findings from the qualitative work were presented and discussed, ensuring that recruiters felt confident and supported during the main trial phase, and regular contact was maintained thereafter. Regular reports from the qualitative findings were also presented at trial management group meetings, steering committee meetings and patient and public involvement meetings, for discussion and reflection. The qualitative work helped us to develop strategies to optimise recruitment during the main trial.
Patient and public involvement
We collaborated with a study-specific patient and public involvement group to seek advice on patient documents and how to improve recruitment and retention methods . The group met on five occasions before the trial and offered advice on the content and format of the plain English summary, PILs, questionnaires, resource-use diaries and interview topic guides. The group met three times during the pilot to review trial progress and give advice on standard operating procedures. Advice given included that research staff making phone calls or home visits should give patients plenty of time to answer the telephone or door as a painful knee can make getting up to walk difficult, especially after sitting for a long time. At the end of the pilot, we reviewed the findings with the group, who made recommendations about patient involvement in researcher training and updates to recruitment standard operating procedures, including how to explain usual care. The group met regularly throughout the trial and will give advice on how to communicate findings to patients and the public.
Step 2: site feasibility assessments
A feasibility assessment is an evaluation of whether there is capacity and capability for a trial to be carried out at a site or group of sites. This trial management process is distinctly different from a feasibility study . It is widely used in commercial research when selecting participating sites for clinical trials and in National Health Service (NHS) research departments in the United Kingdom when deciding whether there is capacity and capability to undertake delivery of a trial as a recruiting site. The process may help to identify barriers to recruitment and establish realistic recruitment targets and timelines . An overview of the process and its importance is covered in the National Institute for Health Research (NIHR) good clinical practice training, which focuses on four considerations: patient population, study team, clinical support services, and equipment and facilities . It is also important to consider external factors that may influence recruitment, such as changes in clinical guidelines or practices . Assessing feasibility after a trial has opened can be useful in identifying issues with recruitment and informing the selection of additional sites. The key aspects of our feasibility assessments are described below.
- Patient population
Gain accurate estimates of how many patients are available.
Consider patient burden when estimating recruitment rates.
Consult a patient and public involvement group.
Embed qualitative research within the internal pilot.
Consider regional differences in patient populations.
- Study sites
Use multiple methods to identify sites, e.g. registries, local research departments and networks.
Send an expression of interest form to potential sites to collect feasibility information.
- Expression of interest forms should ask for the following information:
○ The trial team: training and experience, protected time available, cover arrangements
○ A description of the current standard practice or treatment
○ How the current standard differs from or conflicts with the trial intervention
○ The number of potentially eligible patents and how this number was calculated
○ Whether all (or some) clinicians support the recruitment of their patients
○ Why some clinicians may not want to be involved
○ Any conflicting trials currently running or being set up
○ The number of trials the principal investigator is currently overseeing
○ Whether support is available for additional costs
- Training, equipment and materials
Consider how much training is required for staff, and whether this will be provided centrally or on site.
Plan well in advance to account for busy schedules.
Specify if any specialist equipment or testing is needed, and clarify how or if the site can accommodate this.
Approval processes will differ between sites: be prepared for delays and ask for timelines.
Ask each site what they need to move forward and stay in regular contact.
First, we estimated the patient population at each potential trial site by searching local hospital systems, requesting data from hospital informatics departments and consulting data published by the UK National Joint Registry. Where electronic information was not available, we manually counted entries from clinic or procedure lists. Next, we estimated the number of patients who may want to take part in the trial. Regional variations in recruitment between each site were discussed with local research teams, who gave advice based on their previous experiences with the same or similar patient populations. Populations at recruiting sites were assessed using regular screening log submissions.
Information published by the UK National Joint Registry identified hospitals performing a high volume of total knee replacements. Local research and development departments and research networks were consulted to help identify contacts who could be sent expression of interest requests. These requests were sent to many potential centres with a brief (one page) summary of the trial, either directly or via an intermediate contact, such as a local research network. The information required to conduct a feasibility assessment for a new site was requested in the expression of interest questionnaire, which can be found in Additional file 3. A deadline was placed on the return of the questionnaire.
The expression of interest asked for the number of eligible patients and how this figure was obtained, for example from hospital informatics departments. At one high-volume potential site, only half the clinicians agreed to allow the participation of patients they had treated. Thus, since not all the clinicians on some sites may be willing for the patients under their care to be recruited, we adjusted the targets and projections. An additional site was required to make up for the lost screening population. We considered each clinician’s reasons for not including patients, as these may be the same at other sites. We also investigated whether other studies were being run involving the same patient population, so that we had a sense of whether there was a risk that patients might be overburdened or invited to take part in other studies that might preclude their involvement in the trial, or vice versa. This trial involved excess treatment costs. We found it was important to be transparent to sites about these costs as well as about payments.
Step 3: projections
After the pilot, we found that it was important not simply to divide the required sample size by the number of participating sites and assume each site would recruit equally. Some sites had differing numbers of available patients and recruitment targets were, therefore, weighted accordingly. Setup times were also considered, including a time lag between the commencement of screening and enrolment of the first participant. We used projections to estimate the monthly and overall recruitment from each site and adjusted these as the trial progressed to account for higher or lower recruitment. Our model used the estimated total number of participants to predict recruitment using a conversion rate of the number randomised divided by the number screened. The four-stage process is outlined below.
Stage 1: quantify the patient population
The Stage 1 metric is the total number of available potential participants to be screened in a given amount of time. We first clarified the different circumstances for each site. It was important to understand how their current clinical practice differs from the research intervention and how this might affect patient recruitment. Reliable information about the patient population was sourced from the National Joint Registry or electronic hospital databases to avoid using unrealistic estimates. Screening logs were submitted monthly by each recruiting site and were used to clarify the patient population. Stage one in the equation below represents the total number of patients available for screening across four sites (1456) in one year, or 12 months.
Stage 2: adjust for mitigating factors
The Stage 2 metric is the percentage of participants estimated to be eligible and willing to take part, or the conversion rate. We adjusted for known mitigating factors from the literature to calculate a potential conversion rate. After a systematic review, we thought it reasonable to assume that approximately 20% of patients with a total knee replacement would experience long term pain [24, 25, 26]. We then accounted for those patients deciding not to take part. Rather than set the conversion rate at 20%, we estimated that 50% of these patients would not meet the eligibility criteria and that 50% of those remaining would decline to take part, which leads to a potential conversion rate of 5%. The screening and randomisation logs can be used to calculate actual conversion rates by dividing the number of randomisations by the number screened. This can be used to project recruitment by site as well as the trial overall. Stage two in the equation below is 20% x 50% x 50%, or a conversion rate of 5%.
Stage 3: further adjustments
The Stage 3 metric is an adustment for expected losses to recruitment. Further adjustments were made to account for time periods during which all specialities experience challenges in recruitment to research studies. For example, the end of year holiday period is known to present challenges. We assumed that December and January would each lose half the potential recruitment. Stage three in the equation below represents 11 full recruitment months out of 12 (11/12), or a loss of 1 month recruitment per year.
Stage 4: project recruitment
The Stage 4 metric is the total recruitment period. Once we had estimated our screening population and conversion rate, we applied our projection equation. This gave us an idea of what overall recruitment would be if all sites started on day one and recruited for the full 30 months. Stage four in the equation below represents the total recruitment period of 30 months.
This indicated that the current four sites in the pilot would not be sufficient to attain a sample size of 380. The information was then entered into a spreadsheet to project recruitment and estimate how many additional sites would be required to achieve the additional 214 randomisations, based on the patient population. Projections for each potential site were calculated to take into account the impact on overall trial recruitment. Five new sites were opened using a staged approach over time.
Trial delivery stage
Monthly screening log returns were used to calculate actual conversion rates. These were also used to observe whether sites were able to screen the agreed number of patients. When targets were not met, we worked with the site team to investigate ways to increase screening numbers. For example, we asked if there were particular operating lists or surgeons that were being missed or whether screening searches were being focused on a particular geographic area. Projections were then adjusted based on actual recruitment and conversion rate data. Actual and estimated data were included in the same projection by using actual data on current sites and projected data based on the trial average and a potential new site’s screening population.
Feasibility assessments and recruitment projections are useful tools in designing a trial. Combined with an internal pilot and qualitative work, they allow trial management teams to set realistic recruitment targets and identify ways to improve recruitment, increasing the likelihood that the sample size is met. Using these tools during the pilot of a trial can optimise recruitment into the main trial and ensure delivery on time. Our internal pilot and the development of easy-to-use site feasibility and projection tools have helped to improve recruitment for the main trial, and key learning points can be applied to other trials.
The feasibility assessment that we have described is specific to the context of our trial; however, a similar approach could be applied to other trials with appropriate modifications. The main points to consider as outlined by the NIHR good clinical practice training are the patient population, trial staffing and training, clinical support services, and equipment and facilities. After the feasibility data have been collected and the sites confirmed, the setup is important to consider when setting projections of realistic targets.
Adjusting projections based on screening data is contingent on the submission of accurate screening logs. It has been argued that, despite the Consolidated Standards of Reporting Trials (CONSORT) guidelines, it is not worth collecting screening data , while others report the necessity of collecting screening data to comply with CONSORT guidelines and to report on the generalisability of the trial results [38, 39]. When published, screening data are not only important for transparency, but are also helpful for estimating the patient population in future trials. Screening data collected within a trial are valuable information, which may be used to project future recruitment.
Although the methods described are specific to one trial, the concepts can be applied to many trials, both in the delivery phase and design phase. Information obtained from an internal pilot, including qualitative findings, can be used for site feasibility assessments and to create recruitment projections. This information may be used to improve recruitment procedures and processes and can inform a decision on whether to expand to additional trial sites. Use of these tools may help trial teams increase the likelihood that the sample size will be met.
The authors would like to thank the study participants, members of the PEP-R group, the STAR Patient Panel, the trial steering committee, the Bristol Randomised Trials Collaboration and Professor Tim Peters for their input into this study. This publication is part of the part the STAR Programme and as such it benefits from the involvement of the whole STAR team.
This article presents independent research funded by the NIHR under its Programme Grants for Applied Research programme (RP-PG-0613-20001). This study was also supported by the NIHR Biomedical Research Centre at the University Hospitals Bristol NHS Foundation Trust and the University of Bristol. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care. The STAR research team acknowledge the support of the NIHR, through the Clinical Research Network.
Availability of data and materials
The participants in this research provided informed written consent for their data to be shared anonymously with other researchers. Anonymised qualitative data from this study are available only to bona fide researchers after the University of Bristol Data Access Committee has approved their request. Please contact firstname.lastname@example.org in the first instance.
WB, VW, AM and RGH contributed to the design of this research. WB and AM were involved in the acquisition and analysis of the data. WB drafted the manuscript and all authors revised it critically for important intellectual content. All authors read and approved the final manuscript.
Ethics approval and consent to participate
A favourable opinion for conduct of this trial was issued by the National Research Ethics Service Committee South West – Central Bristol Research Ethics Committee (reference 16/SW/0154). Relevant health research authority and local research and development approvals were obtained, and all participants provided written informed consent.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
- 3.Walters SJ, Bonacho Dos Anjos Henriques-Cadby I, Bortolami O, Flight L, Hind D, Jacques RM, et al. Recruitment and retention of participants in randomised controlled trials: a review of trials funded and published by the United Kingdom Health Technology Assessment Programme. BMJ Open. 2017;7:e015276.CrossRefGoogle Scholar
- 7.Edwards PJ, Roberts I, Clarke MJ, Diguiseppi C, Wentz R, Kwan I, et al. Methods to increase response to postal and electronic questionnaires. Cochrane Database Syst Rev. 2009;(3):MR000008.Google Scholar
- 8.Gillan MG, Ross S, Gilbert FJ, Grant AM, O’Dwyer PJ. Recruitment to multicentre trials: the impact of external influences. Health Bull (Edinb). 2000;58:229–34.Google Scholar
- 15.Bearman JE, Loewenson RB, Gullen WH. Muensch’s Postulates, Laws and Corollaries, Biometrics Note No 4. Bethesda: Office of Biometry and Epidemiology, National Eye Institute; 1974.Google Scholar
- 18.White D, Hind D. Projection of participant recruitment to primary care research: a qualitative study. Trials. 2015;16:473.Google Scholar
- 23.Wylde V, Bertram W, Beswick AD, Blom AW, Bruce J, Burston A, et al. Clinical- and cost-effectiveness of the STAR care pathway compared to usual care for patients with chronic pain after total knee replacement: study protocol for a UK randomised controlled trial. Trials. 2018;19:132.CrossRefGoogle Scholar
- 25.Pinedo-Villanueva R, Khalid S, Wylde V, Gooberman-Hill R, Soni A, Judge A. Identifying individuals with chronic pain after knee replacement: a population-cohort, cluster-analysis of Oxford knee scores in 128,145 patients from the English National Health Service. BMC Musculoskelet Disord. 2018;19:354.CrossRefGoogle Scholar
- 33.Boyatzis RE. Transforming qualitative information: Thematic analysis and code development. Thousand Oaks: Sage; 1998.Google Scholar
- 36.Research NIfH. Introduction to GCP: National Institue for Health Research; [25/07/2018]. Available from: https://www.nihr.ac.uk/our-research-community/clinical-research-staff/learning-and-development/national-directory/good-clinical-practice/our-courses/introduction.htm. Accessed 4 Jan 2019.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.