Please follow and like us:
Evaluation Purpose and Questions
This performance evaluation examines the appropriateness and effectiveness of livelihoods programs for refugee returnees in Burundi funded by DOS/PRM (PRM) during fiscal years 2009-2013 and implemented by three implementing partners (IPs). The purpose of this evaluation is to provide PRM with guidance and learning to improve PRM’s livelihoods programming strategy as well as its capacity to support the design and implementation of livelihoods projects that contribute to helping refugees and returned refugees become self-reliant.
The evaluation seeks to answer the following overarching questions, each of which included several sub-questions that have been thoroughly addressed in the body of this report.
1. What types of assistance/programs were provided?
2. Who are the recipients of assistance/programs?
3. Were PRM-supported programs designed and implemented using best practices?
4. What was the impact of the programs/assistance?
PRM Livelihoods Strategy
Building sustainable livelihoods is believed to facilitate achievement of PRM’s goal of durable solutions and supporting self-reliance for refugees. In May 2014, PRM adopted an internal livelihoods strategy, which outlines the following three goals:
1. Improve design and implementation of livelihoods programming;
2. Develop and disseminate tools and guidance for program officers and refugee coordinators; and
3. Exert diplomatic efforts to improve livelihoods prospects for populations of concern.
World Relief (WR) began programming in the Makamba and Rutana Provinces of Burundi in 2005 however, the projects for which it received PRM funding and that were evaluated as part of this contract began in 2009. WR’s livelihood activities included the distribution of seeds and chickens as well as the establishment of village savings and loans associations (VSLAs). Christian Outreach for Relief and Development (CORD) has worked with PRM funding in Burundi since 2009 with a focus on peacebuilding. CORD’s livelihoods activities targeted HHs with assistance including seeds, tools, animals, and cuttings as well as training in animal husbandry, agricultural production, and composting. Food for the Hungry (FH) also began implementing livelihoods activities with PRM funding in 2009. FH’s main activities included seed and tool distribution, goat distribution, and training in animal husbandry, stable construction, and improved agriculture techniques.
Evaluation Design, Data Collection Methods, and Limitations
This mixed-methods evaluation employed the standard rapid appraisal methods of document review, preliminary interviews, key informant interviews (KIIs), focus group discussions (FGDs), and site visits within Ruyigi, Rutana, and Makamba provinces. The two-person evaluation team interviewed various stakeholders, including PRM staff members, IP staff members, UNHCR, communal authorities, program beneficiaries, non-beneficiaries, and local partner organizations. Several limitations affected the data collection and analysis, including lack of baseline and monitoring data, selection and response bias, restricted fieldwork timeframe, and the absence of baseline and robust monitoring data, among others.
FINDINGS AND CONCLUSIONS
Evaluation Question 1: What types of assistance/programs were provided?
IPs in Burundi included livelihoods activities as part of a larger package of interventions for returned refugees, IDPs, and “stayees” (individuals who remained in Burundi during the conflicts). In most cases, livelihoods activities constituted the smallest component of IPs’ annual budgets. Most livelihoods activities were designed as a one-time distribution of goods rather than a long-term intervention with on-going services. A small number of activities included training components for certain participants, while other activities focused on the creation of income-generating and savings and loans activities. It was difficult for the team to assess the extent to which IPs’ activities were able to meet beneficiary needs and preferences due to the lack of useful baseline data. Needs assessment data, as well as systematic information gathered on individual and HH situations, community and market contexts, as well as ethnic and gender dynamics would have greatly facilitated this evaluation and enabled the team to feasibly answer this evaluation question. The substantial time lapse between the end of the programs and the evaluation made it very problematic for participants to recall their experiences; identify changes over the course of the programming; and differentiate between the inputs and activities they received from FH, CORD, or WR in light of the many other organizations also providing nearly identical support, distributions, and training.
Evaluation Question 2: Who are the recipients of assistance/programs?
All IPs discussed targeting criteria in the implementation of their livelihoods activities, however criteria were loosely defined and clear policies and procedures for applying and following the criteria were lacking. Feedback from village leaders and association members underscores the ad-hoc nature in which many activities were targeted and participants were selected. The team found a range among interviewed participants regarding levels of vulnerability, with some participants in need of assistance to meet basic needs and others who appeared well off and not in need of assistance. Several activities were not designed to assist the most vulnerable due to requirements of either a financial or asset-based contribution such as land, while the targeting and selection processes of other activities were not accessible to all eligible community members due to the way in which the process was executed at the village level.
The team found an overwhelming lack of sustainability due to the one-time nature of activities and the absence of sufficient, on-going support, particularly for individuals who are more vulnerable.
Evaluation Question 3: Were PRM-supported programs designed and implemented using best practices?
The team found the activities to be poorly designed and poorly implemented with minimal sustainability and few tangible outcomes. In addition to the weaknesses in activity design and implementation, the team found a widespread deficiency of technical knowledge and experience among IP program staff about livelihoods and livelihoods interventions. Moreover, the evaluation team found that IPs were not engaged in routine, systematic monitoring of their activities during the period of implementation. In some cases where monitoring data was collected by the IPs, this data was found to be quite accurate, although insufficient for the purposes of determining if a livelihoods program effected a change in the self-sufficiency of the beneficiaries. In particular, the absence of outcome-level indicators limits the depth of learning about these programs that can be achieved.
Evaluation Question 4: What was the impact of the programs/assistance?
Due to the lack of baseline data, the team is not able to determine the impact of the activities or attribute changes in beneficiaries’ livelihoods to the IPs’ interventions. All aspects of the activities for which the team was able to collect information are discussed in the other sections of this report.
IPs need to ensure that their field-based staff members in charge of implementing and overseeing livelihoods programs are properly trained and skilled in the field/discipline of livelihoods. HQ staff need to develop systems and tools to provide their field-based colleagues with stronger technical guidance and support to do their jobs, as field-based staff lack education and training in livelihoods.
IPs should include, at a minimum, one full staff position dedicated to monitoring livelihoods programs. Requiring existing staff members and refugee social worker assistants to undertake program monitoring is not feasible. Including M&E directly in budgets, both in terms of staff time and additional, needed resources, will help to ensure that program monitoring is given the necessary attention and dedication. While having a staff person for each project may not be feasible or necessary, a country-level or regional coordinator may be well-suited for such work.
IPs need to place more importance on developing livelihoods activities that are appealing to the needs of women and girls. Activities should be based on sound evidence as collected through a gender analysis with a focus on livelihoods capacities and gaps. Activities should not only demonstrate a nuanced and informed design, but also an implementation approach that considers how best to recruit and support women and girls to maintain their participation in activities as well as to find success following the termination and/or completion of activities.
PRM should consider its capacity for supporting IPs working on livelihoods assistance. This includes the presence of a clear strategy with well-articulated priorities and operational procedures. This framework should be used as the basis for how awards are made and how IPs are monitored and evaluated over time.
PRM should ensure that needs assessments are conducted by each IP prior to program design and implementation. Needs assessments should also seek to understand the community context in terms of land ownership and use, markets, politics, ethnicity, gender, social structures and networks, wealth, and religion, among others.
PRM should insist on proper M&E of its programs and should require sufficient M&E budget lines in all proposals for livelihoods programs. Specifically, PRM should require all IPs to develop project logical frameworks that clearly state their program or activity objectives and the series of activities that will produce the required outputs, outcomes, and intermediate results to those objectives. The project logical framework should include the necessary custom and standard indicators (as offered by PRM) to monitor program or activity implementation over time. Indicator targets can also be set where necessary. The logical framework should be designed in a way that demonstrates careful consideration of the pre-design and implementation assessments that have been conducted of the needs and context.
PRM should review its vulnerability criteria for livelihoods programs vis-à-vis the types of activities IPs are proposing to implement. This review should be made in consideration that program participants will be well-placed to find success and ultimately to benefit from the activity or program being implemented and that proposed activities are appropriately designed for inclusion of vulnerable populations.
Distributed by APO (African Press Organization) on behalf of U.S. Department of State.
Source:: Evaluating the Effectiveness of PRM Livelihoods Programs for Returned Refugees in Burundi, August 2015