Project Evaluator

Background

Projects background and context

«Empowered Partnership for Sustainable Development – pilot phase»

The Project builds on UNDP’s strong experience in supporting and monitoring the implementation of SDGs in Ukraine. UNDP’s strategic plan focuses on key areas including poverty alleviation, democratic governance and peacebuilding, climate change and disaster risk, and economic inequality. UNDP provides support to governments to integrate the SDGs into their national and sub-national development plans and policies.

In the context of the country’s reforms towards EU integration, and in particular its decentralization reform, many responsibilities lie now with Ukrainian local authorities, and solutions for building more sustainable cities and communities can be both generated and implemented at the local level. One key requirement for initiating and promoting sustainable development changes is to build and nurture multi-stakeholders collaborations (challenge-driven partnerships) that generate solutions to local sustainable development challenges and find resources for their implementation. The quality of local administrations as partners is crucial to ensure that these partnerships are effective and bring in concrete results.

The Project’s overall goal is to contribute to a more efficient public administration, capable to interact and work, in a transparent manner, with the business sector, civil society, and local communities in order to promote sustainable development and Ukraine’s approximation towards the EU and to plan and implement gender-equitable recovery measures in response to COVID-19 crisis, ensuring the protection of rights of the most vulnerable and continuity of the provision of the public services.

Its specific purpose is to pilot at regional/local level (oblasts, municipalities, or territorial communities) the formation of multi-stakeholders (local and regional governments and councils, academia, civil society, and business) partnerships to address sustainable development challenges, including those emerged with the COVID-19 pandemic on the basis of the best international practices. In particular, the project aimed to facilitate collaborations between the public administrations and multiple stakeholders, reaching over administrative geographical boundaries to define and address common sustainable development challenges in local communities, including challenges related to the impact of COVID-19; and to develop and test mechanisms for joint problem-solving, resource pooling and implementation, as well as strengthen organizations’ capacities to collaborate effectively on selected common sustainable development challenges.

Duties and Responsibilities

SCOPE AND OBJECTIVE OF THE ASSIGNMENT

The main objective of the assignment is to conduct the forward-looking Final Evaluation of the Project “Empowered Partnership for Sustainable Development – pilot phase”. The purpose of the evaluation is to analyse the implementation of the project in 2019-2021 and formulate lessons learned; and provide recommendations for scale-up and future initiatives in challenge-based partnerships.

The key product expected is a comprehensive evaluation report (up to 30 pages without annexes, single spacing, Myriad Pro font, size 11), which includes, but is not limited to, the following components:

Executive summary (up to 3 pages);
Introduction;
Assessment of scope and objectives;
Assessment approach and methods;
Development context and project background;
Data analysis and key findings and conclusions;
Lessons learned and recommendations for similar interventions (including viable ideas on areas which could be sharpened and further optimized in future interventions);
Annexes;
List of people interviewed; interview questions, etc.

This final evaluation will assess projects performances against the review criteria: relevance, effectiveness, efficiency, sustainability and impact, in line with UNDP Evaluation Guidelines. More specifically, it will cover, but not be limited to, the following areas and preliminary questions:

RELEVANCE

The report will examine the extent to which the project is relevant to the:

Country context: How relevant was the project to the interventions target groups, including Government’s needs and priorities? To what extent was the project aligned with the policies and strategies of the Government, SDGs as well as UNDP Country Programme Document/United Nations Partnerships Framework?
Target groups: To what extent was the project relevant to address the needs of vulnerable groups and gender issues (both at project and stakeholder’s level)? To what extent did the initial theory of change for the project take those groups into consideration?
Does the project remain relevant taking into account the changing environment while taking into consideration the risks/challenges mitigation strategy? Was there a need to reformulate the project design and the project results framework given changes in the country and operational context.
Does the SymbioCity approach, used in the project, remains relevant and scalable in the evolving context?
What can be done additionally to better capture the needs of vulnerable groups and gender issues?
What measures can be taken to improve the relevance of the project?
To what extent has the project contributed to gender equality, the empowerment of women and the human rights-based approach?

EFFECTIVENESS

Assess the overall performance of the project with reference to its respective project document/cost-sharing agreement, strategy, objectives and indicators, and identify key issues and constraints that affected the achievement of project objectives. Were the planned objectives and outcomes achieved in the framework of the key project components?
What are the results achieved beyond the logical framework? What were the supporting factors? What are the main lessons learned from the partnership strategies and what are the possibilities of replication and scaling-up? How can the Project build on or expand the achievements?
How have stakeholders been involved in project implementation?
What measures can be taken to improve the effectiveness of the project?
What can additionally be done to better capture the needs of vulnerable groups and gender issues?
Assess the project effectiveness at addressing the challenges around which the partnerships were formed?

EFFICIENCY

The extent that to which:

The project cost was effective? Was the project using the least cost options? Have resources (funds, human resources, time, expertise, etc.) been allocated strategically to achieve the relevant outputs and outcomes?
Has the project produced results (outputs and outcomes) within the expected time frame? Was project implementation delayed, and, if it was, did that affect cost effectiveness or results?
Are the project’s activities in line with the schedule of activities as defined by the project team and annual work plans? Are the disbursements and project expenditures in line with budgets?
Was the project management, coordination and monitoring efficient and appropriate?
Assess the criteria of select project partners’ selection.
What can additionally be done to improve the efficiency of the project?

SUSTAINABILITY

Sustainability is understood as the likelihood of continued benefits after the project ends. Assessment of the sustainability of project results will be given special attention.

To what extent are project results (impact, if any, and outcomes) likely to contribute after the project ends? Define the areas that produced the most sustainable results, and the most promising areas requiring further support and scaling-up in the course of future interventions.
Is there sufficient public/stakeholder awareness in support of the project’s long-term objectives?
Is the projects activity likely to continue, be scaled up, replicated and increasingly contribute to the development after the project? Define which of the platforms, networks, relationships development in the framework of the Project that have the highest potential for further scaling up and/or replication.
Are there any social or political risks that may jeopardize the sustainability of project results?
Do the legal frameworks, policies, and governance structures and processes within which the project operates pose risks that may jeopardize the sustainability of project benefits?
To what extent were capacity-building initiatives for partner organizations adequate to ensure sustainability? What could be done to strengthen exit strategies and sustainability?
Identifying possible priority areas of engagement, offer recommendations for the next phase of the Project.
To what extent do mechanisms, procedures and policies exist to allow primary stakeholders to carry forward the results attained on gender equality, empowerment of women, human rights and human development?

IMPACT

Has the Project contributed or is likely to contribute to long-term social, economic, technical changes for individuals, communities, local governance self-bodies and other institutions related to the project?
What difference has the project made to the direct beneficiaries, involved in the implementation of the initiatives, as well as indirect beneficiaries (target communities)?
To what extent has the project promoted positive changes in gender equality and the empowerment of women? Were there any unintended effects?

The final list of evaluation questions and tools to be proposed by the evaluator and agreed with UNDP.

EVALUATION APPROACH AND METHODOLOGY

Methodology

The scope of the final evaluation will cover all activities undertaken in the framework of the Projects. Given the forward-looking nature of the evaluation, the Evaluator will: a) compare planned outputs of the project to actual outputs and assess the actual results to determine their contribution to the attainment of the project’s objectives, as well as b) provide clear recommendations to UNDP, based on identified lessons learned in key areas of project implementation.

The evaluator will be required to use different methods to ensure that data gathering and analysis deliver evidence-based qualitative and quantitative information, based on diverse sources: desk studies and literature review, statistical analysis, individual interviews, focus group meetings, surveys and direct observation. This approach will not only enable the final evaluation to assess causality through quantitative means but also to provide reasons for why certain results were achieved or not and to triangulate information for higher reliability of findings. The concrete mixed methodological approach will be detailed in the inception report and stated in the final report. All data provided in the report should be disaggregated by gender and vulnerability.

The evaluator is expected to follow a participatory and consultative approach ensuring close engagement with UNDP Country Office (CO), project team, government counterparts, international partner organisations all stages of the evaluation planning and implementation. The evaluation will assess the extent to which the project was successfully mainstreamed with UNDP strategic priorities, including eradicating poverty, accelerating structural transformations for sustainable development and building resilience to crises and shocks.

The evaluation of project performance will be carried out against the expectations set out in the Project Logical Framework/Results Framework, which provides performance and impact indicators for project implementation along with their corresponding means of verification. All indicators in the Logical Framework need to be assessed individually, with final achievements noted. An assessment of the project M&E design, implementation and overall quality should be undertaken. The evaluation will assess the key financial aspects of the project, including project budget revisions. Project cost and funding data will be required from the project, including annual expenditures. Variances between planned and actual expenditures will need to be assessed and explained.

The conclusions related to the implementation of the project should be comprehensive and balanced, and highlight the strengths, challenges and outcomes of the project. They should be also well-substantiated by the evidence and logically linked to the assessment findings. In addition, they should also provide insights into the identification of and/or solutions to important problems or issues pertinent to project beneficiaries, UNDP and SIDA.

The recommendations for the project should identify how best practices and achievements of the project can be scaled up or proliferated to increase the positive impact of similar intervention countrywide. The recommendations need to be supported by an evidential basis, be credible, practical, action-oriented, and define who is responsible for the action - to have potential to be used in decision-making.

The evaluator should provide a proposed design, methodology of evaluation (methods, approaches to be used, evaluation criterion for assessment of each component to be proposed), detailed work plan and report structure to UNDP prior to the start of fieldwork; these documents and the list of partners and other stakeholders to visit should be agreed with UNDP. While proposing the methodology, the Consultant should be guided by UNDP approach to project evaluations.

The evaluator is expected to develop and present a detailed statement of evaluations methods/approaches in the inception report to show how each objective and evaluation criterion will be assessed.

The methodology will be based on the following:

Desk review of the documents listed below (but not limited to):
The original project documents, monitoring reports, action plans, M&E frameworks, and financial documents (such as the cost-sharing agreement with donor);
Notes from meetings involved in the project (such as Board meeting minutes, National Steering Committee meeting minutes);
Other project-related material produced by the project (such as datasets, publications, audio-visual materials and consultancies reports).
Interviews with the relevant UNDP Country Office and the project’s management and staff, SIDA and various national and sub-regional authorities dealing with projects’ activities as necessary, to provide in-depth briefing on the projects, results, context of partnerships with different stakeholders etc. as well as vision for future.
Interviews and focus groups discussions with projects partners at the national and local levels and beneficiaries.

De-briefing session will be arranged for discussing the evaluation findings, results and recommendations.

Deliverables

The Consultant should provide the following deliverables for the evaluation of the project “Empowered Partnership for Sustainable Development – pilot phase”:

Deliverable #

Task description

Days and timing

Payment breakdown

Deliverable #1

Conduct desk research of core documentation (small grants agreements, project documents, annual work plans and progress reports 2019-2021, project implementation plans, board meeting minutes, with annexes etc.). The set of documents to be reviewed will be prepared by UNDP. Develop an evaluation strategy and plan.

Output: The inception report (with detailed description of the methodology and evaluation matrix) is produced; annotated structure of the report is developed; a toolkit for gathering data (questionnaire and interview plans, a questionnaire for a beneficiary satisfaction survey) is designed. All documents are submitted to UNDP for final approval.

within 5 days after signing agreement

20%

Deliverable #2

Conduct necessary consultations, field visits, interviews with the project staff and partners. Examine how stakeholders assess the project and what their concerns and suggestions are. Clarify issues that emerge from the preliminary analysis of the project requiring hard and soft data to substantiate their reasoning. Collect and analyse feedback from the partners.

Produce a draft report of the evaluation covering all items detailed in the paragraph #2 of the present TOR with definition of the lessons learned and recommendations for the follow-up phase of the project.

Output: draft report produced and submitted for UNDP comments (UNDP review will take up to 10 days). Initial findings discussed in a wrap-up session with Project team and UNDP CO (can be done on-line via video conference).

within 20 days after inception report approval

40%

Deliverable #3

Collect, review and incorporate comments from UNDP into the final version of the evaluation report.

Prepare a detailed PowerPoint presentation of the evaluation study and present (in English) the results during the meeting between UNDP, SIDA in Kyiv, Ukraine (can be arranged remotely via Skype depending on meeting arrangements. If travel occurs, UNDP will cover all related travel expenses).

Consultations regarding UNDP expectations from the presentation will be held with the Contractor prior to the event.

Output: Final evaluation report containing all required annexes indicated in paragraph #3 of the present TOR, submitted to UNDP, SIDA for final review and approval. PowerPoint presentation prepared and delivered during the joint meeting of interested parties (to cover major findings and lessons learned from the evaluation as defined in section 3 of this TOR with diagrams/pictures, where applicable).

within 5 days after revision of the draft report

40%

The detailed structure of the final report should be agreed with UNDP and reflect all key aspects in focus. Payment will be based upon satisfactory completion of deliverables. 100% of the total amount shall be paid upon completion of the Deliverables 1-3.

MONITORING/REPORTING requirements

The consultant will interact with UNDP project and CO staff to receive any clarifications and guidance that may be needed. He/she will also receive all necessary informational and logistical support from UNDP CO and the Project. On a day-to-day basis, consultant’s work will be coordinated with UNDP Project Manager. The satisfactory completion of each of the deliverables shall be subject to the endorsement of the UNDP CO Partnership and Coordination Officer.

The consultant will inform UNDP of any problems, issues or delays arising during the implementation of the assignment and take necessary steps to address them.

The key product expected is two comprehensive evaluation reports (with parameters indicated above in section 2)

The report must be as free as possible of technical jargon in order to ensure accessibility to its wide and diverse audience. The Report should be prepared in English.

All reports and results are to be submitted to the UNDP in electronic form (*.docx, *.xlsx, *.pptx, and *.pdf or other formats accepted by UNDP).

Competencies

COMPETENCIES

Demonstrates integrity by modeling the UN’s values and ethical standards.
Promotes the vision, mission, and strategic goals of UNDP.
Displays cultural, gender, religion, race, nationality and age sensitivity and adaptability.
Treats all people fairly without favoritism.
Fulfills all obligations to gender sensitivity and zero tolerance for sexual harassment.

Required Skills and Experience

EXPERIENCE AND QUALIFICATIONS REQUIREMENTS

Education: Advanced University degree (Master’s or PhD) in Economics, Management, Mathematics, Social Sciences, Public Administration, Business Administration or other relevant area;
Relevant professional experience: At least 3 years of work experience in the area of economic development, poverty reduction, community engagement, sustainable development, private sector development, SME and/or business support of business membership organizations development including participatory planning, monitoring and evaluation;

Experience in evaluation: At least, 3 accomplished complex evaluations projects where the candidate was the author or co-author especially in economic development, private sector development fields, understanding of gender aspects (a reference to or copies of previously developed knowledge materials including analytical reports, research papers, case studies materials, etc. to be provided);
Proven knowledge of monitoring and evaluation methodologies, a summary of a proposed evaluation methodology is to be provided (up to 2 pages);
Languages proficiency: Excellent knowledge of Ukrainian and/or Russian, English.