1 The Data Collection Challenge

The conflict in Mali in 2012 broke out after a long period of political and economic stability. It began when armed separatist groups occupied the northern desert and semi-desert regions. A period of instability followed, during which an estimated 36% of the total population from the affected regions fled to the south of Mali and to neighboring countries. The crisis had dramatic effects on public infrastructure and service and reduced people’s mobility and their access to markets. It also led to the destruction and theft of assets and shook investor confidence. Farmers were cut off from their fields, artisans were unable to sell their produce as tourism came to a halt, traders were unable to move, and breeders with high numbers of livestock were forced to leave conflict-affected areas for safer places, losing many of their animals to theft along the way. The crisis reinforced the feeling of neglect by the Malian state among those from the affected areas, while simultaneously strengthening cross-border ethnic loyalties and economic ties. The conflict officially ended with the Peace Accords signed in May and June 2015, but the North remains insecure, as it has become a safe-haven for terrorists and criminals.

The crisis created distrust between different ethnic groups and among people of different religious affinities. Social cohesion weakened, and interactions became more restricted, inducing a feeling of fear. About one in three living in Timbuktu or Gao reported in July 2016 that they did not feel safe at home at night; in Kidal, this rose to two in three. Many people distanced themselves from social networks, neighbors became estranged, mixed marriages ended, and even within families, members became wary of each other. Animosity was also expressed toward the government. By July 2016, 53% of the population had lost confidence in the government, and confidence in the judicial system ranged from 66% in Timbuktu to as low as 8% in Kidal.

Collecting data in these circumstances is challenging, especially for emissaries of the central government. In fact, since the outbreak of the conflict in 2012, agents of the National Institute of Statistics have been unable to collect any data in Kidal or elsewhere in northern Mali. Data, however, was urgently needed to monitor the developments post-signing of the Peace Agreement. The Peace Agreement had established conditions for the restoration of stability and economic recovery, and called for development planning and new investments in the north, as well as the creation of a monitoring system to assess the impact of assistance on security, socio-economic development, and wellbeing.

The Permanent Monitoring System (PMS) was created to respond to this data challenge. It consists of an observatory that relies on local enumerators living in northern Mali, who collect data on a monthly basis. The PMS amasses information from a representative sample of households living in the targeted areas,Footnote 1 and from local authorities, clinics, schools, and markets, where commodity prices are collected.Footnote 2

2 The Innovation

When enumerators from outside a community are not welcome or when travel to and within a region is dangerous for outsiders, one solution is to work with locally recruited enumerators who reside in the area. The use of such ‘resident’ enumerators is usually discouraged, particularly for consumption surveys as experience shows that due to limited possibilities for supervision, data quality tends to erode over time, while respondents tend to grow tired of answering detailed questions repeatedly about the various items they have consumed. For this reason, many consumption surveys have shifted from collecting consumption data using diary-methods to approaches that rely on recall. In the former approach it is often necessary for enumerators to stay in the village for up to a month; using recall methods survey teams can stay in the village for much shorter periods of time.

There are major advantages to an approach that relies on enumerators that reside for longer periods in a village. Among these are that resident, locally recruited enumerators know the survey areas well. This reduces many of the complexities associated with insecurity, local grievances, or language. The latter is a critical advantage. Ethnolinguistic fractionalization is high in Africa and in many locations, the ability to speak the language of choice of the respondent is key to the success of a survey. When enumerators cannot phrase questions in the language that respondents are most comfortable with, responses may be wrong or biased.

Another advantage of using resident enumerators is that, contrary to survey teams that visit an enumeration area for a short period of time, resident enumerators have ample time to carry out interviews. Capitalizing on this, enumerators of the PMS were asked to administer five different survey instruments including a household survey that contained multiple modules among which socio-demographic characteristics and income-generating activities, including detailed questions on agriculture, livestock, fisheries, and entrepreneurial activities. The survey collected information on assistance received, the return of refugees and internally displaced people, the health of household members, and food security. The household survey also covered shocks that households might have experienced, possession of assets, and access to services. A part of the questionnaire was devoted to subjective questions about the implementation of the Peace Agreement, perceptions of security, and priorities for initiatives that could consolidate peace and security in the region.

A second survey instrument was used to interview local authorities (mayors, traditional authorities, and local chiefs) to collect information about local initiatives and interventions, as well as information about the evolving security situation. A third survey was administered to assess the operations of health care centers in the surveyed villages. This survey assessed the impact of the crisis on their functioning, the presence and return of staff that had fled during the conflict, the assistance they received, and their needs in terms of supplies and equipment. A fourth survey was conducted in primary schools in the surveyed villages. Like the clinic survey, it assessed the presence of staff and the return of teachers who had fled during the crisis, the assistance the schools had received, and the school’s needs. The fifth and final survey instrument collected information on prices of a selected list of commodities to gauge the changes in the cost of living in different localities.

Another advantage of resident enumerators is that they are in a better position to deal with ‘moving’ populations such as herders, of which there are many in northern Mali. Herders move about, few have mobile phones, and even if they do possess them, they are often out of range of a telephone network. This means that phone interviews as discussed in Chapters 2 and 3 are not feasible, particularly if non-random non-response is to be avoided. The ability to deal with moving respondents is determined by the time availability and local knowledge of resident enumerators. Locally recruited enumerators know where to find pastoralists as they regularly gather at specific locations to water their animals, or as they move from pasture to pasture following well-established grazing patterns. Herders are not the only mobile population. In many places in Africa, farmers also move. During the growing season, many remain at their fields in temporary shelters only to return to their village after the harvest. Enumerators selected from the village can follow households to their farms for interviews. They know the area and, unlike survey teams visiting enumeration areas for only a short period, have more flexibility in when to carry out an interview. They can meet respondents in the evening, early in the morning, at the market, or at the place where the respondent carries out his or her business.

Once enumerators have developed good relationships with the respondents, and respondents have confidence in them, resident enumerators are more likely to elicit accurate information, particularly when questions are sensitive, and the enumerator is able to emulate that responses will be kept confidential. This is another advantage of resident enumerators: it dispels fear and creates trust, trust that can be difficult to establish between people from different localities or ethnic groups. The importance of this cannot be underestimated in a (post)conflict situation, as it is not uncommon for respondents in insecure locations to fear reprisals for having provided information to an outsider, no matter how innocuous the information may seem. However, if the enumerator stays with the respondents in the village, it signals trustworthiness and allays such fears.

The flexibility and level of trust the locally recruited enumerators built in northern Mali allowed them to collect high-quality data and to assure high response rates over the course of a year. Between January 2016 and January 2017, the highest household non-response rate encountered was 4.4% in October 2016 in Gao, when ten households did not respond to the survey; however, they resumed their participation in January 2017. During this particular month, the northern regions experienced 21 attacks and bomb explosions, including six in Gao, four in Timbuktu, and two in Kidal. Still these insecurity events did not disrupt the survey or cause the response rate to drop.

There were two clear challenges when it came to relying on resident enumerators. The first is that it may be difficult to identify skilled enumerators in the communities of interest. Particularly in remote locations, the number of suitable candidates may be limited. Few people are likely to have experience with survey data collection and finding people with a certain level of formal education may be difficult. In the case of the northern Mali survey, the pool of eligible candidates was further reduced by the requirement that prospective enumerators had their own means of transport, allowing them to move about easily, while at the same time, assuring a greater sense of ownership and responsibility than one might expect for a project provided means of transport. A second challenge is supervision.

Hiring and managing enumerators was delegated to a local firm with extensive experience in data collection, and with a robust network in northern Mali.Footnote 3 It advertised the positions on its website and mobilized its contacts in the region to publicize the job opportunity. To assure the independence and objectivity of enumerators, the firm avoided relying on local authorities for recruitment. Those applying were expected to send proof of their education (diplomas) and of the possession of a motorbike (as a means of transport); these were later checked during enumerator training and the first supervision mission. Enumerators were informed that they would not be engaged full-time, and could take up other employment as long as they were available for this activity for at least one year, and during the first two weeks of each month.

Enumerators who had finished at least secondary education were sought, but it proved challenging to find people with that level of education living in remote villages, as less than 5% of the population of northern Mali has completed secondary education or higher (INSTAT 2012). In the end, and after devoting much effort to identifying and hiring enumerators in each enumeration area, it was not possible to find sufficiently qualified people for each location. As a solution, certain enumerators were required to cover two or three villages close to one another, a solution that caused few problems since enumerators had their own motorbikes to move between villages.

For those who did qualify, wages were high. To complete some 20 questionnaires over a two-week period every month, enumerators were paid approximately US$350 per month, plus a premium of US$600 every quarter and again at the end of the operation. These premiums were needed to assure the continued participation of the enumerators, as other organizations active in the area were offering competitive salaries. Although the budget for enumerator fees was relatively high, the overall cost of one round of data collection was reasonable: about US$30,000 for approximately 800 questionnaires (12 households per enumeration area, plus school, clinic, district leaders, and price questionnaires), or less than US$40 per questionnaire. The reason for this relatively low unit cost, despite high salaries is due to the minimal expenses incurred for transport, printing and communications, and meals and lodging (Fig. 1).

Fig. 1
figure 1

(Source World Bank 2016)

Map indicating the location of enumerators across northern Mali

In the end, 35 enumerators were hired. Since traveling to northern Mali was not recommended for people not from the area, all enumerators were invited to Bamako for one week of training. In addition to becoming familiar with the survey material, questionnaires and manuals, much emphasis was placed on how to behave, as the aim was for the enumerators and respondents to develop an ongoing relationship for a period of more than one year. Hence, the training emphasized the importance of confidentiality, the importance of maintaining good relationships with respondents and local authorities, and the necessity of remaining neutral when collecting data.

Maintaining data quality was not an issue, as the response rates presented in Table 1 illustrate. Not only were enumerators motivated, as demonstrated by the fact that none dropped out of the exercise, but the use of tablets to collect data and the ability to remotely supervise the enumerators’ actions improved data quality dramatically. The tablets registered the data and the time of data collection, along with the GPS coordinates of where the data were entered. This para-data allowed the firm supervising the data collection to assess whether the enumerators had indeed visited households for interviews, and to assess the average response time. The use of Computer-Assisted Personal Interviewing (CAPI) thus solved an important supervision problem that might otherwise have affected the quality of data collected by local enumerators operating under limited supervision.

Table 1 Response rate (percentage of households that answered the survey)

To facilitate data collection using tablets, enumerators were trained in the use of CAPI techniques. Different questionnaires (for households, schools, clinics, and district leaders, and price questionnaires) were programmed in CSPRO, and a server was installed in the office of the firm supervising the work. Using CAPI allowed enumerators to send data to Bamako as soon as they completed a survey and had access to the internet. Though phone network coverage is limited in northern Mali, the network exists, at least in the urban center of each district. It was agreed that at least once a week, enumerators would move to a location that had network coverage to transfer their data to the server. At the beginning of each month, when enumerators were within reach of the phone network, they were paid using mobile payment systems such as Orange Money. Enumerators also downloaded new or updated questionnaires at these times. Relying on CAPI thus allowed the team to dynamically change the questionnaires used. Core questions typically did not change, but the questionnaires were adapted regularly to respond to new requests for information from development agencies and the government. Questionnaires were also changed in response to events on the ground and enumerators were expected to report noteworthy events, the distribution of material to farmers and breeders, and the functioning of schools and clinics. Their feedback was then used to update the questionnaires.

CAPI was not used everywhere: in some villages in Kidal, paper questionnaires continued to be used as respondents had expressed concerns about the use of tablets. They feared enumerators might use the GPS capacity of the tablet to order drone strikes. In these few instances, enumerators filled in paper questionnaires and subsequently transferred the responses onto the CAPI system, before electronically sending the responses to the server in Bamako.

The firm visited each enumeration area every six months for additional supervision, exposing the supervision team to insecurity while traveling, but once in the villages, the team was generally given a warm welcome. The team would meet with local authorities, including traditional and religious authorities, to (re)explain the objectives of the activity, and to request continued collaboration. The team also met with citizens at large, stressing how the enumerators were working in the interests of the whole community by striving to collect good information on the issues affecting their villages.

These efforts were successful. Quality data was collected throughout the entire period and for more than one year, the PMS informed the government of Mali and international organizations on changes in the situation in northern Mali. Best of all, none of the enumerators were harmed, nor was any survey respondent affected by violence that could in any way be associated to the survey.

3 Key Results

From September 2015 to January 2016, 35 enumerators covered 672 households across 56 villages and city areas, administering the five different types of survey instruments. Some key results are presented below. Food insecurity was found to mostly affect households in Gao and particularly in the early months of the year, when more than one-quarter of Gao’s citizens lived in a state of food insecurity; this declined to around one-fifth between March and October. In Kidal, food insecurity was found to be much less of an issue, with less than 10% of households living in a state of food insecurity throughout 2016. In January 2017, however, food insecurity became a more serious issue in Kidal, and 19% of households were affected. In Timbuktu, few were affected by food insecurity throughout the duration of the survey (Fig. 2).

Fig. 2
figure 2

(Source Authors’ calculations based on data from the Permanent Monitoring System)

Percentage of households living in a state of food insecurity

Despite the Peace Agreement, the surveyed households’ sense of security decreased considerably during 2016. Between January and December 2016, the percentage of the population who were comfortable at home at night decreased from 79 to 47% in Gao, from 91 to 8% in Kidal, and from 74 to 63% in Timbuktu. The pattern was the same for feelings of security during the day: Between January and December 2016, the percentage of population who felt secure going out alone during the day decreased from 81 to 64% in Gao, from 95 to 13% in Kidal, and from 72 to 69% in Timbuktu (Fig. 3).

Fig. 3
figure 3

(Source Authors’ calculations based on Mali Permanent Monitoring System)

Perceptions of security

Following the conflict, confidence in government was low, especially in Kidal, where less than 20% of the population was found to have confidence in the Malian government. Confidence levels barely changed throughout 2016. In Gao, the percentage of the population having confidence in the government never exceeded 70%. In Timbuktu, levels of confidence were generally higher, but they fluctuated quite considerably over time. Confidence levels were not much different in terms of the judicial system, with particularly low levels in Kidal, higher in Gao, and highest in Timbuktu, where over the course of 2016, confidence in the judicial system decreased from 70 to 51% from January to December (Fig. 4).

Fig. 4
figure 4

(Source Authors’ calculations based on Mali Permanent Monitoring System)

Confidence in the government and the judicial system

The patterns of confidence in the government and the judicial system carry over with respect to trust in people from other ethnic groups and foreigners. In Gao and Timbuktu, the percentage of the population with trust in people from other ethnic groups was relatively high compared to Kidal, but decreased over time, from 77% in Gao and 75% in Timbuktu in January 2016, to 71% in Gao and 56% in Timbuktu in December 2016. In Kidal, levels of trust were found to be significantly lower, at around 40% and falling during 2016. Trust in foreigners was virtually non-existent in Kidal, at less than 10%, but much higher (and rising) in Timbuktu and Gao. In contrast, the population across all three locations was found to have high levels of confidence in religious and traditional authorities. The quasi-totality of the population in the three regions indicated that they had confidence in religious leaders, and the same pattern held true for traditional leaders in Gao and Timbuktu, but not in Kidal, where confidence in traditional leaders was found to be much lower, at 63% in January 2016, and declining over time (Fig. 5).

Fig. 5
figure 5

(Source Authors’ calculations based on Mali Permanent Monitoring System)

Confidence in people

During 2016, the problems faced by healthcare centers remained largely unresolved. Some problems, such as the lack of medication and lack of staff, even increased between January and December 2016, although staff absenteeism declined considerably over the same period. Other problems, such as the lack of infrastructure, became less pressing, but generally speaking, very limited progress was made in restoring health services. The state of schools was similar. The lack of teachers decreased from 24 to 16% over 2016, as did the lack of school materials, declining from 24 to 20%. However, other issues became more pressing, including the lack of classrooms and the absence of school feeding (Fig. 6).

Fig. 6
figure 6

(Source Authors’ calculations based on Mali Permanent Monitoring System)

Problems reported by health facilities and schools

4 Lessons Learned and Next Steps

Implementation of the PMS was surprisingly straightforward, not least because World Bank staff collaborated closely with a high-quality survey firm with experience in northern Mali. Two major challenges, the hiring of enumerators and ensuring data quality, have been discussed already. A third challenge proved to be financing. While the data produced were well-received and in demand, and even though each round of data collection was relatively inexpensive, after 15 months of continuous data collection, the team failed to identify the funding needed to continue the exercise.

Fortunately, an alternative was identified. While financing for generalized data collection proved hard to find, funding for third-party monitoring of project activities was available. The mix of terrorism and armed violence rendered field supervision by donor representatives impossible. At the same time donors desired to invest more in the north to support the peace process. Because donor representatives were not able to visit project sites in northern Mali, they started to rely on third-party monitors. Often, these are local NGOs that are also involved in reconstruction activities (raising concerns about conflicts of interest), or specialized outsider firms with a higher risk appetite, at a commensurate price. Irrespective of their nature, these third-party monitors collect information, for example on the progress of a construction project, which is a task familiar to the resident enumerators used for this project. The resident enumerators were thus retrained to act as third-party monitors. Relying on local enumerators for third-party monitoring is new, and the World Bank is testing this approach against an alternative of visits by experts from local NGOs. This is ongoing, but if the results of the continuous data collection in northern Mali offer any guidance, it seems likely that the local enumerators, equipped with tablets, cell phones, and motorbikes, will be able to provide quality data at a fraction of the cost that is usually paid for third-party monitoring.