Introduction

Donors have funded rural development projects, including research for development (R4D) (sensu lato Høgh-Jensen et al. 2010) for many decades. Agricultural and forestry R4D projects help address international development goals related to enhancing food security, reducing poverty and achieving sustainable management of natural resources. Collectively, agricultural research investments have been shown to generate impressive economic benefits (Lindner et al. 2013; Maredia and Raitzer 2012; Raitzer 2003). However, these economic benefits usually relate to a small number of highly successful projects (Lindner et al. 2013). Individual R4D projects may or may not include activities to facilitate widespread adoption of the research findings. Therefore, it is important for donors to improve their understanding of what enables some R4D projects to have large-scale impacts that contribute to transformational development outcomes.

Agroforestry involves the combination of woody perennials, herbaceous plants, livestock and people, and their interactions with one another in farming and forest systems (Sinclair 1999). It has been practiced by farmers in developing countries for centuries, primarily to support their subsistence, but increasingly to provide important sources of income. In this paper, the term agroforestry will include both traditional agroforestry, where trees are grown in conjunction with crop and livestock management, and smallholder woodlots where the tree component is part of the smallholder’s livelihood system. Agroforestry research seeks to enhance the benefits for farmers and the environment from adopting agroforestry practices. Research approaches have included both narrow-focussed research on tree growing and wider systems-focused research, covering the farming system and the value chain for the agroforestry products (Bartlett 2019). However, with a few exceptions, the researcher-designed agroforestry practices have not achieved widespread adoption (Franzel et al. 2004).

For 35 years, the Australian Centre for International Agricultural Research (ACIAR) has invested in international research partnerships in the agriculture, fisheries and forestry sectors (ACIAR 2017). ACIAR funds R4D projects, conducted by Australian or Consultative Group on International Agricultural Research (CGIAR) scientists working collaboratively with scientists from the partner countries, to address a research priority identified by the partner country. ACIAR projects seek to generate knowledge, technologies and capacity to achieve better decision making, changed agricultural practices and policies that, in turn, generate positive scientific, economic, social or environmental impacts (ACIAR 2014). Under ACIAR’s forestry program, about three-quarters of its 101 projects completed over a 30 year period have related to aspects of agroforestry and smallholder planted forests (Bartlett 2016a).

Very few studies have attempted to identify sets of factors that facilitate success in international development projects (Ika et al. 2012). Systematically evaluating the relative success of multiple R4D projects is fundamental to improving understanding of how and why development interventions work or don’t work and what factors contribute to project success. In this paper, success is considered to have two dimensions. The first is the extent to which planned research outputs are achieved and adopted by “next users”, such as the participating scientists, viz. “achievements”; the second is the extent of the impacts resulting from wider adoption of the research outputs by “end users”, viz. “impacts” (Pearce 2010).

Because R4D projects are implemented in different countries, covering different development issues and a wide range of implementation contexts, it can be costly and challenging to evaluate and compare the success of multiple projects. Also, for individual R4D projects, the nature and extent of impacts generated will be partly dependent on where a project is situated on the impact pathway along the research to development continuum, as well as the nature and effect of any influences outside the project that affect the adoption of the research findings. To address these challenges, Bartlett (2016b) developed an evaluation methodology that can be used to evaluate the relative success of completed R4D projects using the existing project records. In related research, Bartlett (2018) identified 37 factors that can enhance or diminish the success of R4D projects, including 15 ‘key success factors’, and demonstrated the relationships that exist between these factors and the evaluated relative success of forestry R4D projects. This paper evaluates the relative success of 15 completed ACIAR-funded agroforestry R4D projects, considers factors that have affected project success and examines aspects of three of the more successful projects that are contributing towards the achievement of transformational development outcomes.

Methodology

Fifteen agroforestry-related projects completed in the past decade were purposively selected from ACIAR’s database of forestry projects in order for the sample to maximise the number of countries where such projects had been implemented and, where possible, to include some interrelated-projects within a country. The details of the 15 selected projects are shown in Table 1.

Table 1 Summary information for selected completed ACIAR agroforestry-related projects
Full size table

The relative success of each of the 15 projects was evaluated by the author using qualitative data, drawn from internal ACIAR project records, and the score-card matrix methodology described by Bartlett (2016b). The records examined included: project documents; annual project reports; annual assessments and mid-term reviews conducted by the ACIAR program manager; final project reports; external end-of-project reviews; adoption studies and external impact assessments; and any project-related publications. While not fully overcoming potential biases associated with self-reporting of achievements by project staff, the use of multiple sources of available data provided a degree of triangulation, through the perspectives of program managers and independent external reviewers of projects, as well as those of project participants.

For each project, the evaluation considered four criteria related to research achievements: project design; results achieved; collaboration; and publications; and for four criteria related to research impacts: capacity building outcomes; scientific outcomes; economic outcomes; and social and policy outcomes. For each criterion there was a standard evaluation question and an indication of the nature of the evidence sought (see Bartlett 2018).

Table 2 Framework for evaluating relative success of R4D projects
Full size table

In evaluating the relative success of each project, the available evidence was considered and a score for each criterion assigned by the author, to the nearest 0.5, up to the maximum score allowed for each criterion (see Table 2). The maximum scores for two criteria ‘results achieved’ and ‘scientific impacts’ were weighted more highly because it was an evaluation of R4D projects with well-defined activities and expected scientific outputs. Scores totalling up to a maximum of ten were assigned for both research achievements and research impacts. Scores of 0.0–5.0 were categorised as low achievements or low impacts; scores of 5.1–10.0 categorised as high achievements or high impacts. This classification generates four categories of project success: high achievements–high impacts; high achievements–low impacts; low achievements–low impacts; and low achievements–high impacts.

To explore the relevance of key success factors to the success of these projects, 37 scientists were selected using a purposive strategy, because they had worked as project leaders, project coordinators or leading collaborating researchers on one or more of the 15 agroforestry projects. Each scientist was requested to assess the relative importance of the 15 key success factors identified by Bartlett (2018) in relation to the project that they had worked on. Each researcher was given 100 points to allocate across the key success factors that they considered relevant to their project. They were requested to allocate higher numbers of points to the factors that had the greatest influence on project success and to indicate whether they considered that individual factors contributed to enhanced or diminished success. Responses were received from 27 partner scientists, who had worked on 12 of the agroforestry projects included in this study. Seventeen responses related to the projects evaluated by the author as having high achievements–high impacts, nine responses related to high achievements–low impact projects, and only one response related to low achievements–low impacts projects.

The responses were analysed by the author in two ways. All 27 responses were analysed to determine the overall priority ranking by the project participants for the 15 key success factors. Previous research by Bartlett (2018) had shown that the strongest causal relationships between the key success factors and evaluated success existed in the high achievements–high impacts projects and the low achievements–low impacts projects. Therefore, given the relative number of responses received for these two categories in this study, the 17 responses relevant to the six high achievements–high impacts projects were analysed as a subset to determine the relative importance of key success factors in the most successful projects.

Results

Evaluation of relative success

The results from the evaluation of each of the 15 agroforestry projects by the author against the eight evaluation criteria are presented in Table 3.

Table 3 Evaluation scores for 15 agroforestry projects against the 8 evaluation criteria
Full size table

The scores for the four evaluation criteria for research achievements varied between projects and the sums of these scores for each project ranged from 4 to 10, with all but two of the projects achieving scores greater than 5. The scores for the four evaluation criteria for research impacts varied between projects and the sums of these scores for each project ranged from 2 to 9. Only six of the 15 projects that were evaluated achieved scores of greater than 5 for impacts achieved, demonstrating the challenges that many R4D projects experience in achieving significant impacts.

The combined results from the evaluation of the research achievements and research impacts for the 15 agroforestry projects are presented in Fig. 1. This shows that the relative success of the projects varies considerably across the sampled agroforestry projects, with six projects evaluated as having high achievements–high impacts, seven projects having high achievements–low impacts, and two projects having low achievements–low impacts.

Fig. 1
figure 1

Relative success evaluation scores for 15 agroforestry projects

Full size image

The six projects that had the highest levels of success (as indicated from their achievements and impacts scores) were:

  • FST/2008/007—Advanced breeding and deployment for tropical acacias (Vietnam)

  • FSC/2012/043—Enhancing evergreen agriculture in East Africa – Trees for Food Security (Ethiopia, Rwanda, Uganda, Burundi)

  • FST/2014/099—Enhancing private sector development of a Canarium industry (Papua New Guinea)

  • FST/2011/076—Enhanced livelihoods from agroforestry & community forestry (Nepal)

  • FST/2005/177—Improving profitability from smallholder teak agroforestry (Indonesia)

  • FST/2012/039—Integrating timber and non-timber forest products in agroforestry (Indonesia)

The two projects that had the lowest levels of success were:

  • FST/2004/057—Improved management of teak and paper mulberry plantations (Laos)

  • FST/2012/043—Enhancing economic opportunities from smallholder forestry (Solomon Islands)

Analysis of key success factors

The analysis of the total data set from the 27 returns (see Table 4), shows that these project scientists considered that all 15 of the key success factors had an effect on the success of the various agroforestry projects. The total scores for individual success factors ranged from 121 (for site selection and scientific rigour of trials) to 245 (for leadership and project management).

Table 4 Project scientists’ ranking of 15 key success factors
Full size table

In this analysis, the 27 project scientists considered that all 15 key success factors were relevant to the success of their projects. However, as the cumulative scores for individual factors ranged from 121 to 245, some were considered to have had more influence on project success. The two key success factors they considered to be the most important were “good leadership and project management” and “collaborative scoping and design”.

The analysis of the 17 responses relevant to the six high-achievements–high impacts projects (see Fig. 2) showed a similar distribution pattern to that from the total data set, with the same two key success factors considered most important. However, there were some differences in the ranking of the factors related to project implementation, most notably the greater importance of “links to impact pathway and user benefits”.

Fig. 2
figure 2

Relative importance of key success factors in the six high achievement–high impact projects

Full size image

Discussion

Understanding project success

Evaluating the relative success of multiple agroforestry R4D projects provides a comparison of each project’s achievements and impacts as well as insights into which projects may be contributing to transformational outcomes. This study has found that 40 percent of the sampled ACIAR agroforestry projects had high achievements and high impacts, 47 percent had high achievements but low impacts, while 13 percent had both low achievements and low impacts. Interestingly, the high achievements–high impacts projects occurred in all four geographic regions (Eastern Africa, South Asia, South-East Asia and PNG-Pacific) where ACIAR has been implementing forestry projects.

The two projects with the highest relative success evaluation scores had quite different characteristics in terms of research focus and duration of the research collaboration. The highest scoring project (FST/2008/007) had a comparatively narrow focus on improving tropical Acacia germplasm in Vietnam and followed on from four related R4D projects conducted over a 15 year period. In contrast, the second highest scoring project (FSC/2012/014) involved three different but related themes of agroforestry research, had activities in four East African countries and wasn’t linked to previous ACIAR projects. This demonstrates that it is possible to achieve high levels of project success regardless of whether or not the project teams have previously worked together, or the breadth of research activities and where they sit on the research to development continuum.

The two projects with the lowest relative success evaluation scores were both teak agroforestry projects which had some similarities and some differences. The lowest scoring project (FST/2004/057), which was implemented in northern Laos, focussed on spacing and thinning of teak plantations, improving teak germplasm and exploration of potential non-timber forest products for use as companion plantings in agroforestry systems. The second lowest scoring project (FST/2012/043), which was implemented in Solomon Islands, focussed on developing farmer-friendly teak silvicultural systems, improving germplasm and promoting teak agroforestry systems. Both were first phase R4D projects and had overly ambitious project designs given they were new research collaborations in countries where the research capacity was weak. This meant that both projects struggled to achieve all the planned research outputs and the impacts were largely confined to building researcher capacity and some dissemination of scientific information.

When a donor funds a series of related R4D projects over a period of ten to twenty years, generally the expectation is that the achievements and impacts from each successive project will be greater than from the previous project. However, in an evaluation of 30 forestry R4D projects Bartlett (2018) found that this expectation is not always met. In this study, there were five pairs of linked agroforestry projects, covering activities in Laos, Indonesia, Papua New Guinea, Solomon Islands and Vanuatu. This study found that for two of these pairs the relative success of the latter projects improved, but for the other three pairs the relative success actually declined for the latter projects (see Fig. 3). For the two Papua New Guinea projects (FST/2010/013 and FST/2014/099), where the success increased from high achievements–low impacts to high achievements–high impacts, the second project was better designed with a stronger focus on exploring the activities that would create new markets for the agroforestry product. For the two Laos projects (FST/2004/057 and FST/2012/041), where the success increased from low achievements–low impacts to high achievements–high impacts, the level of collaboration between partners improved enabling more of the planned outputs to be achieved, which in turn generated greater economic and scientific impacts from the research.

Fig. 3
figure 3

Relative success trends in linked agroforestry projects

Full size image

When the level of success declined between projects, the reasons generally related to the challenges of achieving increased levels of results and/or scaling out project activities, or problems with collaboration between partners. For the two Indonesian projects (FST/2005/177 and FST/2012/039), both of which were evaluated as high achievements–high impacts, the decreased relative success was related to the second project’s inability to achieve as many of the planned outputs and expected impacts when its activities were scaled out to four other areas. For the two Vanuatu projects (FST/2005/089 and FST/2012/042), both of which were evaluated as high achievements–high impacts), the decreased success related to the project’s inability to complete many of the planned outputs in the second phase and a significant deterioration in the relationships and collaboration within the project team. For the two Solomon Islands projects (FST/2007/020 and FST/2012/043), the relative success declined from high achievements–low impacts to low achievements–low impacts between the two projects. The overly ambitious design of the second project and the inability of the project team to implement many of the planned activities or to generate scientific results contributed to the declining success.

In this study, the participating scientists collectively considered that all 15 of the previously identified key success factors were relevant to the success of their agroforestry projects and some individuals involved in the less successful projects identified that inadequate attention to some factors had diminished the project’s success. For example, in the case of the least successful project in Solomon Islands (FST/2012/043) both ‘collaborative scoping and design’ and ‘selection and commitment of partner institutions’ were identified as factors that diminished success; meaning that for this project there was insufficient effort made during project design to ensure the planned research activities had strong ownership by the partner scientists and that the partner institutions were genuinely committed to providing counterpart resources to work with the Australian scientists. Conversely, analysis of the responses from the two most successful projects (FST/2008/007 and FSC/2012/014) indicated that these same two key success factors plus two other factors ‘leadership and management’ and ‘scientists commitment, collaboration and focus’, both relevant to project implementation, were all considered important contributors to those two project’s success. This demonstrates that all these key success factors do need to be actively considered and managed during project design and project implementation, both by the donor organisation and the scientists participating in the project.

Contributions to transformational development

While many international development agencies seek to achieve transformational outcomes from their investments, there is currently no accepted definition of the concept of transformational development. The World Bank Group (2016) defined transformational interventions as “an individual or a series of interventions that support deep, systemic, and sustainable change with the potential for large-scale impact in an area of a major development challenge”. There are also challenges in categorically demonstrating transformational development outcomes from small budget R4D agroforestry projects. However, it is possible that some such projects may either contribute substantially towards, or stimulate the catalytical programs and changes necessary in partner countries to achieve, such outcomes. To do this, the R4D projects would need to be well designed with a clear theory of change and then achieve the intended outcomes. Such projects would invariably achieve high evaluation scores for both project achievements and project impacts.

Three of the agroforestry R4D projects from this study can potentially claim to have contributed towards transformational development, as they achieved relative success evaluation scores of at least eight for both project achievements and project impacts. These projects were: the Vietnam “Breeding for tropical Acacias” project (FST/2008/007); the East African “Trees for Food Security” project (FSC/2012/014); and the Papua New Guinea “Development of a Canarium industry” project (FST/2014/099). The activities conducted under each of these projects were quite different as were their operating contexts, but each project resulted in enhanced agroforestry knowledge and technologies. The outputs from these projects are being used by the project partners, and more broadly within the partner countries, to facilitate improved adoption of agroforestry systems, development of new markets for agroforestry products and enhanced livelihoods from the sale and processing of agroforestry products. The nature of each of these projects and the salient features that have contributed towards transformational development outcomes are examined in the following paragraphs.

Project FST/2008/007 focused on advanced breeding and deployment for tropical acacias in Vietnam, particularly increasing the number of hybrid Acacia clones, refining germplasm deployment strategies and developing polyploid varieties. It is linked to agroforestry because in Vietnam over 300,000 smallholder farmers have tropical acacias planted as part of their farming system utilising germplasm arising from the long-standing ACIAR collaboration. When the project commenced in 2009, there was about 400,000 hectares of Acacia plantings, including 150,000 hectares of A. mangium x A. auriculiformis hybrid all derived from about 15 commercially-licenced clones. The project evaluated another 550 hybrid clones, with the early growth of many new clones equalling or exceeding that of the commercial clones, and established a second-generation seed orchard using superior hybrid clones (Griffin et al. 2015). By 2011, the recorded area of Acacia plantings had increased to 960,000 hectares, with an estimated additional unrecorded area of 649,000 hectares located on small farms (Midgley et al. 2017). In 2014, Vietnam produced 23 M m3 of acacia wood sourced from land designated as forest plantations and small farm plantings, with the small plantings alone contributing an estimated US$300 M to smallholder farmer livelihoods (Midgley et al. 2017). Clearly this ACIAR project, and the series of predecessor projects have contributed very significantly to transforming livelihoods and lives of large numbers of Vietnamese smallholders who now practice Acacia-based agroforestry.

The features of this R4D agroforestry intervention and the broader operating environment that have contributed towards transformational development outcomes include:

  • Acacia agroforestry involves a simple farmer-friendly technology making it easy for farmers to adopt—they just plant the seedlings and harvest trees after 5–6 years

  • Twenty years of R4D collaboration between Australian scientists and scientists at the Vietnam Academy of Forest Sciences to domesticate and improve tropical Acacia germplasm

  • Substantial investment in postgraduate capacity building of committed local partner scientists linked to the R4D project

  • Existence of a very effective national germplasm dissemination system, providing easy access to high quality Acacia germplasm

  • Vibrant and competitive markets for Acacia logs, enabling farmers to access these markets and receive good returns from their agroforestry practices

Project FSC/2012/014 was ACIAR’s largest single agroforestry project, which also had substantial co-investment from the World Agroforestry Centre. The project sought to enhance smallholders’ food security through research on options and approaches for increasing crop production where farmers incorporated agroforestry practices. Its activities focussed on semi-arid and sub-humid zones in Ethiopia, Rwanda, Uganda, and Burundi. Research activities included identifying locally-appropriate agroforestry options, improving understanding of farmers’ tree growing preferences and extension systems, predicting the interactions between trees and crops and the impacts from adopting agroforestry at field, farm and landscape scales, and facilitating adoption through improved access to appropriate germplasm and tree-growing knowledge. Its activities were linked to national programmes to scale up the use of trees within farming systems. The project substantially improved understanding of the contexts and patterns of tree adoption by farmers (Iiyama et al. 2016) and tree-crop interactions for wheat (Dilla et al. 2019) and maize (Sida et al. 2018) crops, and enhanced predictive capability through development of tree-crop interaction modelling within the APSIM family of predictive models (Luedeling et al. 2016). It also established 1600 participatory agroforestry trials with farmers in three countries, and facilitated improved adoption of agroforestry by over 30,000 farmers through the establishment of six Rural Resource Centres, which provided training, and supply of improved tree germplasm and business opportunities for farmer groups and unemployed youth (Muthuri et al. 2017). Its activities are transforming lives of many farmers and being scaled up by other donors and agencies.

The features of this R4D agroforestry intervention and the broader operating environment that have contributed towards transformational development outcomes include:

  • Existence of supportive government policies and programs promoting extensive application of agroforestry

  • Developing and disseminating locally-appropriate agroforestry systems that match farmers’ interests, and potential markets

  • Including a focus within the project design on improving adoption of agroforestry practices

  • Utilising effective local mechanisms for dissemination of germplasm and tree-growing knowledge with links to national agroforestry programs and activities of non government organisations (NGOs)

  • Strong linkages to national agencies responsible for designing and implementing agroforestry programs

  • Ability of the World Agroforestry Centre to provide effective year-round support in partner countries and to engage at the Ministerial and senior official level to overcome implementation challenges

Project FST/2014/099 was a somewhat unusual agroforestry R4D project, in that it focussed primarily on developing value-added markets for nuts from the indigenous tree Canarium indicum. These trees are widely grown in agroforestry gardens by the people of Papua New Guinea and produce highly nutritious nuts. However, at the time the project commenced there was no private-sector entities purchasing and processing these nuts. The project followed on from a decade of previous R4D on nut processing techniques, market research and the development of a pilot processing factory located within the National Agricultural Research Institute. The project’s activities included scaling up production from the pilot processing factory, working with potential private sector investors to launch a range of processed nut products into urban markets, and supporting women with the development of small nut processing enterprises. In 2018, the project’s Galip Nut Company (https://www.galipnuts.net/) launched a range of natural, roasted and peeled nut products in supermarkets and duty free stores in Port Moresby, with demand exceeding supply. By 2018, the quantum of unprocessed nuts purchased for processing had risen to over 200 tonnes/annum, with the 65 tonnes of nut purchased in 2017 generating about AUD 135,000 for 659 farmers. The project is collaborating with four new private sector entities that are investing in nut processing in Papua New Guinea (unpublished ACIAR reports). While there is still more to be done to fully develop the fledgling Canarium nut industry in Papua New Guinea, this project has demonstrated the significant potential for this new agroforestry-based product to facilitate widespread transformational development for communities in this country. It also demonstrates that sometimes, in order to create markets for agroforestry products where none currently exists, both the donor and the collaborating project partners need to be willing to step beyond the normal boundaries of a R4D agroforestry project and have a strong commercial focus.

The features of this R4D agroforestry intervention and the broader operating environment that have contributed towards transformational development outcomes include:

  • High numbers of existing planted Canarium trees with the nuts being part of local diets for centuries

  • Inclusion of funding in the project for operating a pilot nut processing facility to supply local markets

  • Investments in product development, value-chain and market research as well as capacity building of partner researchers in commercial nut processing operations

  • Effective partnerships developed with private sector actors to trial commercial processed nut sales

  • Government recognition that Canarium nuts are one of three agricultural products with potential for widespread commercialisation

Conclusions

The success of agroforestry R4D projects is of interest to the donors that fund them, to the governments of the countries in which they are implemented and to the recipients of the benefits that come from these projects. Evaluating the relative success of 15 ACIAR-funded agroforestry R4D projects has provided valuable insights into the effectiveness of these projects in terms of what they set out to do, the relevance of previously identified key success factors to the projects with the greatest and least success and provided a way of identifying projects that have potentially contributed to transformational development outcomes. It has also shown that in situations where one R4D project follows on from a related project, the relative success of the second project may not always be greater than the first project. The existence of this evaluation method provides donors with an option to compare the relative success of related projects and, if necessary, to adjust project activities or resources during implementation to achieve greater success. It has also enabled the identification of three very different agroforestry R4D projects that have contributed to transformational development outcomes.

In the main, R4D projects develop knowledge, technologies, capacity and approaches to dissemination to the intended beneficiaries. Rarely do they have the resources to support large-scale dissemination or adoption programs. The extent to which the outputs from R4D projects are adopted in developing countries depends largely on the interest and ability of partner governments, NGOs and other development partners to disseminate the results and to provide appropriate supporting programs to enable widespread adoption to occur.

Key salient features of the three R4D agroforestry projects and their operating contexts that have enabled them contribute to transformational outcomes included: the development of simple farmer-friendly, locally-appropriate agroforestry technologies; the existence of supportive government policies and programs, including effective mechanisms for dissemination of germplasm and tree-growing knowledge to farmers; the engagement of NGOs and private sector entities; and the willingness of the donor to invest in value-added product development and research on efficient value-chains and potential markets. Each of these is highly linked to the key project success factor “links to impact pathways”.