Introduction
International development actors are increasingly investing in efforts to evaluate the impact of their work, with the aims both of understanding their effectiveness and generating useful learning on how to make programmes more effective. Most impact evaluations take the form of in-depth studies of particular projects or programmes, implemented in particular contexts. While the resulting reports are often useful for understanding effectiveness on a local level, producing generalized insights or learning remains difficult. In this paper we seek to bridge this gap through the use of statistical meta-analysis, a technique that pools data from multiple studies, in order to seek patterns across those studies and to allow for the analysis of questions which are not answerable with any one study in isolation.
A conventional meta-analysis (such as those carried out by the Cochrane Collaboration or the International Initiative for Impact Evaluation, 3ie) involves consolidating multiple studies of a single type of intervention, with the aim of drawing general conclusions about the effectiveness of that intervention.1 In contrast, this meta-analysis consolidates data on the effectiveness of a single organisation, Oxfam GB, across 16 projects that aimed to build resilience in rural communities. These data come from Oxfam’s ‘Effectiveness Reviews’, a series of impact evaluations carried out each year since 2011 on randomly selected projects. The projects evaluated were implemented in various locations around the world and were diverse in their scale and activities, but they were evaluated against a common objective – that of building resilience to shocks, stresses and uncertainty – using a consistent quasi-experimental methodology.
In this paper, we present the results of the meta-analysis and discuss what can be learned from those results. Overall, the projects evaluated are found to have had a significant positive effect on resilience. The aggregated project effect across the 16 evaluations is estimated to be approximately 0.4 standard deviations in terms of Oxfam’s ‘resilience index’, a composite index of context-specific indicators of resilience. There are significant positive effects across each of the three components of this resilience index – absorptive capacity, adaptive capacity and transformative capacity.
Surprisingly, there does not appear to be any connection between the size of impact and the scale of the project, nor the time-frame over which the interventions are implemented. However, there are large differences between regions of the world: projects evaluated in Asia are estimated to have much greater impact, on average, than those in Africa and Latin America. We investigate whether these regional differences could be caused by differences in the nature of the projects in different regions, or whether they result from the measurement approach or the evaluation methodology itself. We conclude that it seems unlikely that these factors could fully explain the difference in results between Asia and the other two regions.
We also find evidence of an important gender difference in resilience: female-headed households were generally found to have lower scores on the resilience index than were maleheaded households. This gender difference is small in magnitude, but it is consistent across the components of resilience and across the contexts where the evaluations were carried out. If female-headed households are generally less resilient than male-headed households, it is important also to consider whether the activities of resilience-building projects tend to have greater impact among female-headed or male-headed households. In fact we find no evidence that the projects evaluated overall had greater or lesser impact among female-headed households than among male-headed households.
The meta-analysis also provides useful insights about the approach used to measure resilience in these evaluations. Firstly, we investigate whether there is potential for using a standard set of resilience indicators across all evaluations, rather than adapting the indicators to the local context in each case. We find that such a standardized resilience index would produce lower estimates of projects’ impact on average – but it could arguably provide a more accurate measure of resilience, less at risk of biasing the selection of indicators towards the priorities of any particular programme. However, this also means that a standardized index would be less sensitive to the more immediate changes that are likely to arise from programme activities. Secondly, we test whether households that we assess as being more resilient in fact generally suffer less when large-scale crises occur. We find evidence that households with higher scores for the resilience index reported losing fewer livestock during recent crises (droughts or floods), but that there is no relationship between the resilience index and losses of crops. The result for livestock losses provides at least some reassurance that the resilience index is a meaningful measure of households’ ability to deal with shocks.
This paper is structured as follows. Section 2 introduces the evaluations included in the metaanalysis, the evaluation methodology, and the approach used to measure resilience. Section 3 presents the results of the meta-analysis and examines differences in the results by region and by gender of the household head. Section 4 discusses ways in which the meta-analysis can help to inform the resilience measurement approach. Section 5 concludes by discussing what can be learned from the results of this meta-analysis, both for programme design in general and for the measurement of resilience specifically.
Read full report here.
Original source: Oxfam.
Posted on 25 October 2017
