Background

The 2014–2016 Ebola virus disease (EVD) (Ebola) epidemic in Liberia, Guinea and Sierra Leone resulted in over 28,000 reported cases and over 11,000 deaths [1]. Sierra Leone was the most severely affected country, with 8706 laboratory confirmed cases and 3956 deaths [1]. During the epidemic, surveillance goals were to detect Ebola cases promptly to provide an effective response including rapid diagnosis, case isolation and management, contact tracing, safe burials, and identification of transmission chains [2, 3]. Contact tracing for Ebola is defined as the identification and follow-up over 21 days of individuals who have been in contact with a person confirmed to have been infected with Ebola [2]. Previous epidemics have been controlled through contact tracing and isolation, which limits onward transmission [4].

Contact tracing was a core surveillance activity in the 2014–2016 epidemic [5, 6]. Ebola contact tracing operations used paper-based systems supplemented by phone calls and Short Message Service (SMS). Challenges identified in previous epidemics include incomplete identification of contacts and inefficiencies in paper-based reporting systems, including missing contact lists, inadequate data collection and training, transcription errors and delays in steps from identification of contacts to isolation of suspected cases among those contacts [2, 5, 7,8,9,10].

The national system for contact tracing in Sierra Leone during the 2014–2016 Ebola epidemic was that, after the initial alert of a suspected or probable case, a case investigation team (CIT) investigated to determine whether the person was a suspected or probable Ebola case. If the case definition was met, the CIT conducted an initial line listing of contacts of the case. The case (suspected or probable) was then escorted to an Ebola Treatment Centre (ETC) or holding centre. If laboratory-confirmed as EVD, contact tracing was initiated, using a paper Daily Reporting Form (DRF). Challenges specific to contact tracing during the 2014–2016 Ebola epidemic included insufficient trained staff, community resistance to report Ebola cases and contacts and mistrust of contact tracers, limited telephone and internet connection, difficulties in reaching remote areas and the high number of Ebola cases that occurred within a short period [5, 9]. Due to these challenges, we established a study to assess the ability of an mHealth app to improve contact tracing through using mobile phones.

mHealth approaches for epidemic investigations

Use of mobile devices to support medical and public health practice (mHealth) can improve health outcomes in low-income settings [11,12,13,14], due to the low cost of roll-out, mobility of devices, ease of use and flexible deployment compared with other methods e.g. computers. mHealth applications have the potential to reach rural populations with low levels of income and literacy, and to reduce time to collect data, distance travelled to collect and return information and the cost of information delivery [11, 12, 15, 16]. mHealth applications have been used for epidemic tracking of tuberculosis (TB) and Japanese encephalitis, but few mHealth applications have been designed and evaluated for contact tracing. One app designed for contact tracing for TB in Botswana reduced the median time required to complete the TB contact tracing form and improved data quality compared to paper-based contact tracing [17]. However, the full potential of mHealth in addressing disease epidemics is largely unexplored.

During the 2014–2016 Ebola epidemic, several mHealth interventions were developed to improve patient care, health worker safety, public awareness, surveillance and vaccine monitoring [18,19,20,21,22,23,24,25,26]. Studies have reported on the use of mobile phones for community event-based surveillance through reporting of sick community members with symptoms suggestive of Ebola to community and district monitors through audio phone calls and SMS messages to support surveillance initiatives [20, 27]. One of the few apps for Ebola contact tracing was developed by the Earth Institute at Columbia University. This consisted of a smartphone application developed using the CommCare application for collecting and analysing contact tracing data in Guinea [10]. The system demonstrated the potential to improve data access and improve the quality of surveillance data.

Evaluating mHealth interventions

Most studies of mHealth applications are small-scale or pilot implementation studies, with few rigorously designed evaluation studies [28,29,30]. This is likely due to inherent challenges in introducing technologies in complex emergencies, including the feasibility of implementing technology during an ongoing epidemic [10] .

Objective

Our hypothesis was that the mHealth system would be more effective for contact tracing than the standard paper-based system. The original aims were to i) evaluate the feasibility of introducing a smartphone-based electronic data-capture and management system, through the development of a smartphone app for Ebola contact tracing and monitoring in Port Loko District, Sierra Leone, and ii) compare the effectiveness of the app with the standard paper-based system of contact tracing through a cluster randomised trial. The objectives were to i) assess the time between a new Ebola case being laboratory-confirmed and their household contacts first being visited by a contact tracer, and ii) evaluate the operational requirements needed to introduce effective mobile-based contact tracing and monitoring in Sierra Leone.

Due to the decline in Ebola cases in Port Loko District during the first 6 months of 2015, the study design was changed on the June 8, 2015 to a proof-of-concept (demonstration) project. The study aim was revised to describe the feasibility and effectiveness of an electronic data capture and management system to improve monitoring of contacts of confirmed Ebola cases, and to provide guidance on how best to implement the system. The objectives remained unchanged.

The objective of this manuscript is to describe the results of this proof-of-concept study to assess the feasibility and effectiveness of implementing the electronic data capature and management system for contact tracing through comparison with the standard paper-based system for contact tracing.

Methods

Setting and participants

The study was conducted in Port Loko District, northern Sierra Leone, and was a collaboration between the London School of Hygiene & Tropical Medicine (LSHTM), Innovations for Poverty Action (IPA) and International Medical Corps (IMC), working closely with the District and National Ebola Response Teams. In Port Loko District, the overall technical supervision of surveillance and contact tracing activities was managed by the World Health Organization (WHO). The lead agency for the management and supervision of surveillance activities including contact tracing activities was Marie Stopes International until the end of March 2015, and was taken over by GOAL Global on April 1st 2015. UNFPA were responsible for the management and supervision of contact tracers.

Eligible study participants were all 26 Contact Tracing Coordinators (CTCs) and 86 Contact Tracers (CTs) operating in the District during the study period (April 13 to August 31, 2015). The list of eligible CTCs and CTs was obtained from partners responsible for contact tracing activities in the District.

Recruitment and informed consent

Written informed consent was obtained from eligible CTCs and CTs who consented to take part in the study. Consent was not required from individual Ebola contacts as the smartphone app mirrored the existing paper-based system. The only additional information collected was to record the time of the first contact tracing visit.

Study definition - Ebola contact

The study definition of an Ebola contact was that used by the Sierra Leone National Emergency Response Centre, i.e. a person without any disease signs and symptoms who had physical contact with an Ebola case (alive or dead) or the body fluids of an Ebola case within the last three weeks. Physical contact included sharing the same room/bed, caring for a patient, touching body fluids, or closely participating in a burial [31]. An Ebola contact was defined as a listedFootnote 1 contact of a laboratory-confirmed Ebola case in Port Loko District diagnosed during the study period.

The line listing of Ebola contacts was undertaken using the Ministry of Health and Sanitation, Ebola Virus Disease Contact Line Listing form. This was done by the CIT and involved asking household members of the Ebola case to report on the individuals that the Ebola case had come into contact with that met the definition used by the Sierra Leone National Emergency Response Centre. Data on each person who had contact with the Ebola case (Ebola contacts) was collected, including their name, demographic and location information, date of last contact with the Ebola case, type of contact, head of household and contact details. The initial line list was then updated once the suspected or probable Ebola case was confirmed by the laboratory. Once the Ebola case was confirmed, contact tracing of the line listed contacts started.

Original study design

The original design was a two-arm cluster-randomised trial to be conducted in all 11 Chiefdoms of Port Loko District. The Chiefdoms were randomised in a 5:6 allocation ratio to receive one of the following: i) Control arm: standard paper-based contact tracing system, involving CTCs operating at Chiefdom level within the District, together with a team of CTs based in villages throughout the Chiefdom using a paper-based system supplemented by ad-hoc phone calls and SMS messaging for contact tracing of contacts of confirmed Ebola cases ii) Intervention arm: the CTCs and their CTs were provided with smartphones installed with the Ebola Contact Tracing (ECT) app (Fig. 1). CTCs and CTs were trained and supported to use this system. The trial started on April 13, 2015, with the intervention introduced in 6 randomly selected Chiefdoms and ongoing paper-based data collection in the remaining 5 Chiefdoms.

Fig. 1
figure 1

Design of the Ebola Contact Tracing smartphone app study. Key. * The surveillance team goes back to the house of the suspected case to obtain an updated list of contacts following laboratory confirmation. ** District Health Management Team. # Viral Hemorrhagic Fever. † Contact Tracing Coordinators visit the DHMT six days a week (Monday to Saturday). $ Ebola Contact Tracing

Full size image

Modified study design

Due to the small number of Ebola cases, the design was changed on June 8, 2015 to a proof-of-concept study in order to introduce the app to all 11 Chiefdoms in the District. This decision was taken to maximise the data collected using the app given the small number of new Ebola cases and that these had occurred within a short period of time. Following the recommendation of local WHO staff, the paper-based system for contact tracing continued to be used alongside the ECT app as it was the standard national system and was considered essential for surveillance. The date and time of the Ebola confirmation for all cases was from the Public Health England (PHE) laboratory in Port Loko District.

Qualitative methods

Qualitative methods formed part of a process evaluation to evaluate the intervention. This included semi-structured interviews and focus group discussions (FGDs) with district-level data entry and management staff, CTCs and CTs on their use of mobile technology prior to the study, use of the smartphone app for contact tracing, training and ideas for improvements to the app. Semi-structured interview schedules and a topic guide were developed that addressed key areas in relation to the study objectives, study operation, implementation and delivery. The semi-structured interviews and focus group discussions were conducted in either English or Krio and were audio recorded.

Development and testing of the Ebola contact tracing (ECT) app

The intervention (“ECT app”) was a customised three-tier smartphone application developed using Dimagi’s CommCare platform [32], an open-source mobile platform for electronic data capture and case management. It was developed from February–April 2015 by a developer in the USA (SQ) working with the study team. The primary reasons for using CommCare were i) its case management feature, which enables information for an index Ebola case to be linked to multiple contacts per case; and ii) the ability to share data on the Ebola cases and their contacts between multiple users (e.g. CTCs, CTs and the study team). The app was pre- and pilot tested, and refined iteratively before deployment in mid-April 2015. Due to the timeline, app structure and feasibility of making technical changes, not all required changes could be made to the app, however, mitigating action was taken to overcome these issues (Additional file 1: Table S1). The smartphones were second-hand Android phones (4G HTC MyTouch) donated by the UN Mission for Ebola Emergency Response (UNMEER).

The app was developed following an assessment of the structure of the contact tracing system in the District, and discussions with national and local stakeholders including the National Emergency Response Centre Surveillance Pillar, the WHO Country Office, the Port Loko District Health Management Team (DHMT), stakeholders from the DERC, District Council, US Centers for Disease Control, WHO (Port Loko), Marie Stopes International, UNFPA and GOAL Global (Additional file 2).

Phased introduction of the ECT app

A phased introduction of the intervention (the ECT app) was formally approved by the Sierra Leone Ministry of Health and Sanitation and the WHO (Freetown national office) on February 9, 2015. This involved pilot testing the intervention in one ‘low risk’ Chiefdom (with no confirmed Ebola cases within the previous 21 days) from March 5–10, comprising a 2-day training workshop and a 4-day simulation exercise. Following feedback and revisions to the app, a second pilot test was conducted from March 16–18. The results were presented to stakeholders, and authorisation was granted to phase in the intervention.

Study training

The main study training was on April 14–24, 2015 for all 26 CTCs from the 11 Chiefdoms. This included an initial half-day of training on the study design and use of the paper DRF for contact tracing. Those in the intervention arm Chiefdoms had a further 2.5 days training after the initial training on use of the smartphone and ECT app. This training was prior to the modified design of a proof-of-concept study. Training for CTCs from the control arm Chiefdoms subsequently included in the proof-of-concept study took place from June 9–20. Prior to this training, the paper-based system was the sole contact tracing system in the control Chiefdoms.

Data analysis

The data collected via the app was stored on a central server on CommCareHQ and exported into Excel and checked before being transferred to Stata version 14 for data management. Data from the paper-based DRFs were entered into Excel, and exported to Stata for data management. Descriptive analyses were used to estimate the duration from laboratory confirmation of an Ebola case to time of first visit to any one of their contacts monitored using the app or paper, and the median duration of each step of the chain (confirmation of Ebola case to Ebola case registration, to Ebola contact assignment by CTCs using the app, to first visit by the CT; Table 1). Results were summarised by case, and also over all the contacts to give equal weight to each contact, as the number of contacts per case varied substantially.

Table 1 Features and functionality of the Ebola Contact Tracing (ECT) App
Full size table

Qualitative data were translated and transcribed into English. All data were anonymised and any identifying information was removed prior to the analysis. The data was then analysed using thematic analysis to identify, analyse and report patterns (themes) from the data using NVivo software [33]. A six-step approach was undertaken for the thematic analysis as described by Braun and Clarke (2006) [33].

Ethics

Ethics approval was received from The London School of Hygiene & Tropical Medicine Observational/Interventions Research Ethics Committee (reference 8749–01) and the Sierra Leone Ethics and Scientific Review Committee (SLESRC). The study received clearance and support from the Republic of Sierra Leone Ministry of Health and Sanitation, and the National Ebola Response Centre Surveillance Pillar (responsible for Ebola contact tracing and monitoring activities nationally).

Results

Ebola cases

Of the 43 EVD cases identified during the study period, the contacts of 25 were monitored only using the standard paper-based system for contact tracing (“paper-based cases”). Contacts of the remaining 18 cases were also monitored on the ECT app (“app-based cases”). The cases were confirmed between the 13th April and 17th July 2015. The median age of paper-based and app-based cases was 26 years (interquartile range (IQR) 20–44) and 29 years (IQR 18–40), respectively (Table 2). Approximately half the cases were female (60% for the paper-based cases and 50% for the app-based cases; p = 0.52). All but one of the paper-based cases were from one Chiefdom. The 18 app-based cases were distributed across 5 Chiefdoms (Table 2).

Table 2 Characteristics of cases and contacts (13th April – August 31st 2015)
Full size table

Contacts of Ebola cases

A total of 408 contacts were identified for the 25 paper-based cases (“paper-based contacts”) (average 16 per case), and 646 contacts for the 18 app-based cases (“app-based contacts”) (average 36 per case). The age and sex distribution of contacts were similar by type of case (Table 2).

Standard paper-based system for contact tracing

The paper DRF was returned for the contacts of only 9 of the 25 paper-based cases (36.0%) and for 157 of the 408 contacts of these cases (38.5%) at the end of the study period. The data collected were of poor quality, including errors on the forms (including missing or obviously incorrect time of first visit, Ebola case IDs and dates of first and last visit). A common error was that the time of contact visit reported by the CT on the DRF was reported as prior to the laboratory confirmation of the index case for 63 (40%) of the paper-based contacts. Among the remaining 94 contacts with completed forms, the median duration from laboratory confirmation of the Ebola case to first reported contact visit by the CT was 16.9 h (IQR 15.9–61.8 h). Fewer than half of the contacts (n = 43, 44%) were reported to have been monitored for the full 21 days.

Influence of training on paper-based contact tracing

One Ebola case was confirmed the day before training started, and served as an example case during training. The quality of paper-based contact tracing data for this case was higher than for subsequent paper-based cases. For example, all forms for the 32 contacts of the case were returned, and 31 of these had time-of-first-visit data in line with the standard protocol of CTCs collecting the list of contacts from the DHMT between 8 and 9 am each morning and contact visits being made around midday. In contrast, for the 6 subsequent cases from another Chiefdom, 65 contacts were all recorded as having time-of-first-visit at 8.30 am (mainly on the same day as each other) which according to the standard protocol was not possible.

App-based reporting for contact tracing

Of the 18 app-based cases, 2 were ‘secret burials’Footnote 2 and were not laboratory-confirmed. For the 16 confirmed app-based cases entered on the ECT app for contact monitoring, 556 contacts were registered (Table 3). Of these, 524 contacts were assigned by CTCs (94.2%) for contact tracing, and 384 (62.5%) were visited. The 172 contacts who were not assigned or visited were from 5 different cases each from a different Chiefdom. The median time from laboratory confirmation to case registration on the ECT app was 18.0 h (IQR 12.1–30.0), from case registration to contact assignment by CTCs was a median of 23.4 h (IQR 3.5–46.1) and from median time of contact assignment to median time of first visit by a CT was 22.8 h (IQR 11.7–108.7; Table 3). Overall, the median time from laboratory confirmation to first recorded visit by the CT was 70.2 h (IQR 61.9–124.5) (equal weight per case). Key factors associated with the delays are described in Table 3.

Table 3 Time from laboratory confirmation to first visit for the 16 confirmed cases, plus two secret burials
Full size table

Table 4 summarises results at contact level. The median duration from contact registration on the ECT app to assignment by CTCs was 4.33 h (IQR 1.4–47.5 h) but the median duration from contact assignment to first recorded visit by CTs was 24.9 h (IQR 18.1–122.7 h). The total median duration from the start of case registration on the ECT app at the district to the first recorded visit by the CT was 73.2 h (IQR 26.2–216.8 h) (equal weight per contact). The median number of days monitored for the 384 contacts was 10 days (IQR 5.5–16). This varied by Chiefdom with one Chiefdom having a median recorded follow-up of just 3 days (IQR 1–4).

Table 4 Duration of each step from case registration to first visit of each contact, using the ECT app
Full size table

Qualitative results

Results from the process evaluation indicated that few (< 5%) CTCs and CTs had previously used a smartphone but both CTCs and CTs preferred the ECT app for contact tracing, reporting that it was faster and more accurate to use than the paper-based system. Reported advantages of the ECT app over the paper-based system included the elimination of the CTC’s travel time to receive the daily lists of contacts from the District, the ability to monitor accurately large numbers of contacts, and the relative ease of transporting a phone compared with carrying paper forms. Suggested improvements related to using the app included the need for better network coverage (outside the control of the study or even the Ministry of Health and Sanitation); improved battery life and quality of phones; the need for further training on synchronising the data (“syncing”) between the phone and the server; the need for increased compensation to offset phone charging fees (due to decreased battery life of the donated second–hand phone batteries due to some CTCs and CTs charging their phones using alternative methods of removing and charging the phone battery only using ‘charging pegs’ at telecentres (charging booths) instead of taking the entire phone for charging); better strategies for overcoming distances to charging booths; and more refresher training for contact tracing and monitoring using the study phones.

Discussion

To our knowledge, this is the first study to attempt to estimate the time to first contact visit from confirmation of an Ebola case for contact tracing using an app- compared with paper-based system. Our findings demonstrate proof-of-concept that an electronic system for data capture and management can improve data completeness and accuracy of Ebola contact tracing and monitoring through improved data completeness and monitoring. However, the study also illustrated the challenges of conducting high-quality contact tracing in an emergency setting [2, 5, 9, 34].

Previous studies comparing paper-based methods of data collection with electronic methods have tended to show the electronic method was superior [17, 35]. Our study showed that advantages of the app over the paper-based system included a more accurate, timely, and permanent record of information, and improved data completeness, quality and security. The challenges in collecting high-quality data using a paper-based system was shown by the poor quality of this data.

Operational requirements needed to introduce effective mobile-based contact tracing/monitoring during an Ebola epidemic include effective coordination with technical partners, understanding the structure of the evolving contact tracing and monitoring process in real-time, accessing study hardware and software, and the technical expertise to design, develop, pilot and deploy the app during the epidemic. The results are applicable to other low-income settings that face similar geographical and infrastructural conditions.

One key finding was the long interval between Ebola case and contact registration and first visit by the CT when using the ECT app. This was, on average, longer than the interval for the paper-based cases that predated the introduction of the app, though the data are not fully comparable and may be partly due to the highly-clustered nature of Ebola cases and their contacts, in space and time, with the addition of new contacts several days after the initial laboratory confirmation of the case increasing the duration between registration and first visit. The need to adhere to national protocols meant that a parallel approach of using both paper and the app was in place, which may have affected the efficiency and accuracy of the app-based method. The duration could have potentially been shortened through technical improvements in the system (e.g. batch assignment of contacts rather than single assignment of each contact by a CTC to one or more CTs), but these were impractical due to the underlying technical structure of the app. Other causes reflect structural limitations of mHealth approaches in certain settings (including poor network coverage).

Strengths and limitations of the ECT app-based system

A key challenge was the complexity of designing and implementing an app-based system in an emergency context. Few studies have attempted to design and evaluate a mHealth intervention during an epidemic. It was necessary to minimise any risk to the existing paper-based system, and steps taken to mitigate this risk included the phased introduction of the ECT app and continued use of the paper-based DRF alongside the app as recommended by technical partners.

The major strength of the app-based system was the creation of a more accurate, timely, and permanent record of information stored in a centralised data management and data storage system in the main CommCare HQ. The system allowed for the real-time transfer of information to CTCs and CTs once the Ebola case and contacts were registered at the District level. The app-based system had in-built validation and mandatory data entry fields to improve data entry and data accuracy. Other strengths of the app-based system were the reduction in data entry work, improved data security, data sharing, and monitoring of CTC and CT activity with the production of key performance data. For example, the paper-based system meant CTs had to carry large volumes of paper to monitor contacts. Further, additional contacts could be easily added to the app-based system and assigned for contact tracing. This reduced the need for travel by CTCs to the DHMT offices to collect the Ebola contact line list, which was the daily process for the standard paper-based system. Findings from the qualitative results found that CTCs and CTs preferred to use the app over the paper-based system for contact tracing.

We were unable to test our original trial hypothesis that the app reduced the time from confirmation-to-visit by the CT as the data quality for the paper-based system was inadequate for this estimation. However, the duration for some app-based steps were longer than anticipated, reflected the multiple challenges (Table 3). For example, the time from confirmation-to-registration on the ECT app (median 18 h) reflected the standard protocol whereby the surveillance teams had to revisit the home of the Ebola case to review and, where necessary, revise the contact list after case confirmation and subsequent delays in receiving the new updated list of contacts back to the district level. This was also meant to occur for the paper-based system; however, this protocol was not always followed.

One challenge in using the app was assignment of long lists of Ebola contacts. This was due to a technical feature related to the app that resulted in errors in assignment that potentially contributed to a time delay. Further issues included the time-consuming assignment of long lists of Ebola contacts to multiple CTs, some of whom did not have their ECT app-installed smartphones switched on, or who may having forgotten how to use the app if there was a long delay between assignments. Steps to mitigate this included giving CTs pocket-sized instructions on key steps on the use of the app, field support from a study monitor and refresher training.

The average number of Ebola contacts for the paper-based system was lower than that for the app-based system (16 contacts per case compared with 36 contacts per case). The paper-based contact line listing was undertaken before contact tracing started, so the difference in the average number of Ebola contacts per case is unlikely to be due to the method used, but may reflect the varying number of contacts of Ebola cases over time and the high degree of clustering in time and space. Further, surveillance activities were enhanced in the district in June, which may have increased the number of contacts identified per case. The increased number of contacts per case during the app-system period may also reflect the increased duration from registration-to-first-visit by the CTs for the app-based versus paper-based system. The majority of cases with longer confirmation-to-visit times were from Chiefdoms with multiple contacts per case and this may have affected the process e.g. through manual errors in assigning contacts and system errors with monitoring features.

Other challenges included poor network coverage and technical challenges (e.g. corrupted memory cards and software errors), short battery life and charging issues and the time lag between the training and emergence of real Ebola cases and contacts for monitoring. Mitigating action included testing the phones before deployment and providing CTCs and CTs with spare phones/batteries, power banks for charging and changing the network provider (Additional file 1: Table S1).

Developing an app during an epidemic had additional challenges, including the speed with which the app needed to be developed, limited time for pre- and pilot testing due to the pressure to respond quickly to the emergency and difficulties in updating app design and technical features after deployment. The complex design structure of the contact tracing system had to be reflected in the technical design of the app platform. Mitigating action to overcome these issues included simulated testing of Ebola cases and contacts to test the system, further refresher training and training and the development of strategies to reduce issues related to the difficulties with the app design (some of which are listed in Additional file 1: Table S1).

Strengths and limitations of the paper-based system

The main strength of the paper-based system was the use of a simple method of data collection. However, our study was motivated by recognised challenges in this system, including in-built delays such as the physical collection of forms from the DHMT each morning (a round trip of approximately 5–6 h depending on the Chiefdom). The geographical challenges particular to Port Loko District and the challenges for surveillance activities have been reported by others involved in surveillance activities in the District [36].

Our study identified major problems with i) data archiving of the standard paper-based system, with the majority of contact tracing forms not being returned to the DHMT at the end of the monitoring period; and ii) data quality, with missing and inaccurate information. However, contact tracing may have sometimes been conducted more efficiently than the recorded data implied - for example, direct reports by phone circumventing the paper-based system. This may have improved the time-to-first-visit using the paper-based system but not improved the app-based system as the timings for the app-based system were automatically generated.

Recommendations to improve contact tracing

i) Addressing human resource issues is key especially the dissatisfaction with the level of financial reimbursement for the role, travel to often remote locations and hostility from local communities. These personnel challenges have been reported in other studies on Ebola contact tracing [2, 5, 7,8,9,10, 36].

ii) Improved monitoring and evaluation including stronger technical oversight of contact tracing activities and improved monitoring of CTs in particular to monitor their contact tracing activities and improved strategies to obtain DRFs, for example, setting specific days that DRFs should be returned to the District, and incentives/rewards for timely and complete data submission.

iii) For the paper-based system, we recommend that future improvements should address operational delivery issues of the paper-based system including, improved filing, data archiving and data entry systems, and greater co-ordination between organisations involved.

iv) For the app-based system, a two-tier system from registration at the District to visitation by the CT, rather than our three-tier structure would have been preferable, as the long delays were often between contact assignment to first visit. Our monitoring showed that the assignment stage presented major problems, even though CTCs were trained and received refresher training on this stage. One of the considerable constraints of the app-based system was trying to match to a paper-based system. A complex system created with paper and voice communication in mind can be difficult to replicate using the consistent logic required by software.

Other recommendations would include use of an in-country app design team where possible, further support on the technical aspects of the design, and high-quality (not second-hand) phones.

Conclusion

This study illustrates the complexities, challenges and opportunities of implementing an app-based system for epidemic contact tracing and monitoring. Despite these challenges, this proof-of-concept study shows that implementing mHealth applications even in a low-connectivity and low-resource environment like rural Sierra Leone is possible, even in an emergency. Having more time to develop and test the app and conduct training workshops in a non-emergency setting should result in a more comprehensive and successful app that could be further integrated with the national health system.