Monitoring and Evaluation Guidelines

www.iaea.org/technicalcooperation

Monitoring and Evaluation

Guidelines

Technical Cooperation Projects

Contents

List of Acronyms ......................................................................................................................... 3

1. INTRODUCTION ................................................................................................................... 4

1.1. Purpose and audience of the manual .................................................................................. 4

1.2. Features of the IAEA’s technical cooperation programme ................................................. 4

1.3. Structure of the manual ...................................................................................................... 5

2. CONCEPTS AND RATIONALE ............................................................................................... 6

2.1. Basic definitions ................................................................................................................... 6

2.2. Difference between monitoring and evaluation ................................................................. 7

2.3. Monitoring & Evaluation Criteria ........................................................................................ 7

2.4. Rationale for Monitoring and Evaluation ............................................................................ 9

2.5. Considerations for M&E within the TC programme ............................................................ 9

3. PLANNING FOR MONITORING AND EVALUATION ........................................................... 12

3.1. Starting point: the Logical Framework Approach (LFA) .................................................... 12

3.2. Indicators ........................................................................................................................... 16

3.3. Data Collection/M&E Tasks ............................................................................................... 19

3.4. Frequency of and Responsibilities for M&E tasks ............................................................. 21

3.5. Risks related to monitoring and evaluation implementation ........................................... 23

4. IMPLEMENTING MONITORING AND EVALUATION FOR TECHNICAL COOPERATION

PROJECTS: PRINCIPLES AND TOOLS ......................................................................................... 24

4.1. Principles of monitoring and evaluation within the TC programme ................................. 24

4.2. Monitoring and evaluation Tools for TC projects .............................................................. 26

Resource Documents ................................................................................................................ 29

ANNEXES......................................................................................................................... 30

A. Example-1 of LFM: Improving a regulatory framework .................................................... 31

B. Example 1 of M&E matrix: Improving the regulatory Framework .................................... 34

C. Example-2 of LFM: Radiotherapy services ........................................................................ 36

D. Example 2 of M&E Matrix: Radiotherapy services ............................................................ 38

E. EXample of Work/Action Plan ........................................................................................... 40

F. Project Progress Assessment Report (PPAR) Template .................................................... 41

G. Guidelines for preparation of PPAR ................................................................................... 44

H. Example of Project Progress Assessment Report (PPAR) ................................................. 47

I. Guidelines for Field Monitoring Missions .......................................................................... 50

J. Checklist of Specific Questions for Monitoring ................................................................. 52

K. Guidelines for self-evaluation ............................................................................................ 53

L. Sample of information Gathering Tools/Methods ............................................................ 55

LIST OF ACRONYMS

Counterpart

DAC

Development Assistance Committee

DTM

Designated Team Member

FMM

Field monitoring mission

IAEA

International Atomic Energy Agency

IFAD

International Fund for Agricultural Development

LFA

Logical framework approach

LFM

Logical Framework Matrix

M&E

Monitoring and evaluation

M&EM

Monitoring and evaluation matrix

MoV

Means of Verification

Member State

NLA

National Liaison Assistant

NLO

National Liaison Officer

OECD

Organisation for Economic Co-operation and Development

OIOS

Office of Internal Oversight Services

PCMF

Programme Cycle Management Framework

PMA

Programme Management Assistant

PMO

Programme Management Officer

PPAR

Project Progress Assessment Report

RASIMS

Radiation Safety Information Management

System

Regional Division

SEPO

Successes, failures, potentials and obstacles

SWOT

Strengths, weaknesses, opportunities and threats

Technical cooperation

TCPC

Division of Programme Support and Coordination

TCQAS

Technical Cooperation Quality Assurance Section

Technical Department

ToR

Terms of reference

Technical Officer

WFP

World Food Programme (of the United Nations)

1. INTRODUCTION

1.1. PURPOSE AND AUDIENCE OF THE MANUAL

This document supplements other guidelines and manuals already developed in the framework of the

technical cooperation (TC) programme

of IAEA. It aims to clarify concepts and scope, and to provide

guidance and tools for results monitoring and evaluation of TC projects.

The counterparts

(CPs) of TC projects are the primary audience targeted by the manual. It will help

counterparts to better implement, monitor and self-evaluate their projects and, ultimately, to better

demonstrate accomplishments in contributing to the achievement of Member State development goals, as

stipulated in the IAEA Medium Term Strategy 2012-2017 (p.3):

“The Agency will enhance its role in promoting the advantages of nuclear technology and applications where

they have an added value for addressing basic human and socio-economic development needs and in

promoting capacity building in Member States. Activities in human health, cancer treatment, food security,

water resource management, industrial applications and environmental monitoring will contribute towards the

achievement of the Millennium Development Goals and any follow-up initiative”.

National Liaison Officers (NLOs) and Agency staff members involved in the delivery of TC programme are an

important secondary audience for the manual. It is expected that the manual will contribute to enhancing

their knowledge and skills in monitoring TC projects and also backstopping the respective CPs.

Finally, the manual is intended for all other TC programme stakeholders. It will improve understanding and

knowledge for monitoring and evaluation within the TC programme context.

1.2. FEATURES OF THE IAEA’S TECHNICAL COOPERATION PROGRAMME

The IAEA’s TC programme is the main mechanism through which the IAEA helps Member States to build,

strengthen and maintain capacities in the safe, peaceful and secure use of nuclear technology in support of

sustainable socioeconomic development. Key areas of intervention include human health, agriculture and

food security, water and environment, energy planning and nuclear safety and security.

The following documents can be mentioned:

 Designing IAEA Technical Cooperation Projects using the Logical Framework Approach

 TC Programme Quality Criteria

 Roles and responsibilities in the formulation of the technical cooperation programme

 Policy for Projects (National, Regional and Inter-Regional)

 TC Programme Planning and Design Glossary

See http://pcmf.iaea.org for more details

According to the TC glossary, the CP is an institution or individual in the Member State that manages the project and

thus plays a primary role in project planning and implementation.

The design and management of the TC programme is guided by various IAEA policy documents. Key

documents include

a) The IAEA Statute;

b) The Revised Guiding Principles and General Operating Rules to Govern the Provision of Technical

Assistance by the Agency (INFCIRC/267);

c) The IAEA Medium Term Strategy 2012-2017;

d) The Technical Cooperation Strategy: The 2002 Review (GOV/INF/2002/8/Mod1) (TCS);

e) The Revised Supplementary Agreement Concerning the Provision of Technical Assistance (RSA);

f) General Conference TC resolutions and Board of Governors decisions.

Various key principles are derived from these policy documents, which guide how TC activities are designed

and managed. The TC programme is developed according to the principle of shared responsibility by the

Member State and the Secretariat, with the leading role taken by the country.

The programme is needs driven and is developed through a consultative process with all programme

stakeholders to identify development needs, gaps and priorities where nuclear technology has a competitive

advantage. National projects are designed by the counterparts; regional projects are designed by a lead

country selected from among the Member States of a region.

The Technical Co-operation Strategy: “The 2002 Review (GOV/INF/2002/8/Mod.1)” states:

“The technical co-operation (TC) programme of IAEA is part of the Agency’s mandate “to accelerate and

enlarge the contribution of atomic energy to peace, health and prosperity throughout the world.” The IAEA’s

role under this programme is that of a scientific and technical agency making a discrete but significant

contribution to sustainable development goals through the development and transfer of nuclear science and

technology. This transfer takes place primarily through the provision of training, expert advice and equipment

— designed to build, strengthen and maintain Member State capacity for using nuclear technology in a safe,

secure and sustainable manner. Technology transfer is underpinned by the Agency’s technical expertise,

quality control capabilities and information networks.”

1.3. STRUCTURE OF THE MANUAL

The manual is divided into three chapters, followed by several Annexes.

Chapter 1 clarifies monitoring and evaluation (M&E) concepts, as well the rationale for undertaking M&E for

TC projects. It also presents considerations for M&E within the context of the TC programme.

Chapter 2 describes the Logical framework approach (LFA), the Logical Framework Matrix (LFM) and the

results hierarchy. It also presents the M&E Matrix and describes its different elements.

Chapter 3 presents the principles for M&E within the TC context, and introduces the following M&E tools:

Project Progress Assessment Report (PPAR), field monitoring missions (FMMs) and Self-Evaluation that are

suggested to TC projects.

Concrete examples and/or guidelines on each tool are included in the Annexes.

See http://wwwtc.iaea.org/tcdocumentrepository

2. CONCEPTS AND RATIONALE

2.1. BASIC DEFINITIONS

Monitoring is a continuous function to inform the programme or project managers and stakeholders of

progress achieved against planned results (outputs, outcome and objectives). Data on specific, pre-

determined indicators is systematically collected and analysed to track actual programme or project

performance for management decision making (IAEA-TC Glossary).

Monitoring generally involves collecting and analysing data on implementation processes, strategies and

results.

Other definitions in the literature

“Monitoring is a continuing function that uses systematic collection of data on specified indicators to provide

management and the main stakeholders of an on-going development intervention with indications of the

extent of progress and achievement of objectives and progress in the use of allocated funds.” (Organisation

for Economic Co-operation and Development - Development Assistance Committee (OECD-DAC) expert

group, 2002-2008)

“Monitoring can be defined as the on-going process by which stakeholders obtain regular feedback on the

progress being made towards achieving their goals and objectives.” (UNDP Handbook on Planning,

Monitoring and Evaluation for Development Results, 2009)

“Monitoring is defined as the systematic and continuous collecting, analysing and using of information for the

purpose of management and decision-making. The purpose of monitoring is to achieve efficient and effective

performance of an operation. Monitoring provides an ‘early warning system’, which allows for timely and

appropriate intervention if a project is not adhering to the plan.” (European Commission, 2008)

Evaluation is an objective, independent and systematic examination of the extent to which a programme or

project has achieved (or is achieving) over time its stated objective and, therefore, is meeting the needs and

priorities of Member States. Evaluation assesses the efficiency, effectiveness, relevance, impact, and

sustainability of a programme or project (IAEA-TC Glossary).

Other definitions in the literature:

“Evaluation is the systematic and objective assessment of an on-going or completed project, programme or

policy, its design, implementation and results.” (OECD–DAC expert group)

“An evaluation is an assessment, as systematic and impartial as possible, of an activity, project, programme,

strategy, policy, topic, theme, sector, operational area, institutional performance, etc. It focuses on expected

Additional definitions are provided for the purpose of comparison.

Purpose of Chapter 2:

 Clarify the conceptual framework related to monitoring and evaluation (M&E) as well the rationale

for undertaking M&E tasks

 Present considerations for M&E within the context of IAEA’s TC programme

and achieved accomplishments, examining the results chain, processes, contextual factors an causality, in

order to understand achievements or the lack thereof. It aims at determining the relevance, impact,

effectiveness, efficiency and sustainability of the interventions and contributions of the organizations of the

UN system”. (UN Norms for Evaluation, 2005)

An independent evaluation uses rigorous standards, and must be conducted by persons or entities

independent of those who designed and implemented the programme or project.

An evaluation can be formative (e.g. midterm evaluation) or summative (e.g. final evaluation and impact

evaluation). Evaluation seeks to provide information that is credible and useful, enabling the incorporation of

lessons to learn into the decision-making process of the organizations of the UN system and those of

Member States.

2.2. DIFFERENCE BETWEEN MONITORING AND EVALUATION

The terms ‘monitoring’ and ‘evaluation’ refer to two different functions.

Table 1 presents a comparison between the two. There are important differences when considering

frequency, purpose, focus, participants and reporting.

TABLE 1. COMPARISON OF MONITORING AND EVALUATION

Monitoring

Evaluation

Frequency

Continuously throughout the project

life time

At a given point in time, e.g. end of project,

mid-term, ex-post or change of phase

Basic

Purpose

Steer the project; provide timely

information on progress made

Assess and provide judgement on the

performance; learn from past to improve

future programming

Focus

Collecting and analysing factual

information about activities, outputs,

(without forgetting outcome) and the

processes

Assess outputs, outcome and impact; and

quality of the design, project implementation

and context

Participants

Project staff, project end users

External evaluators, project staff, end users,

donors and other stakeholders

Reporting

Programme managers, project staff,

primary stakeholders, funding agency

Programme managers, project staff, primary

stakeholders, funding agency and policy-

makers

2.3. MONITORING & EVALUATION CRITERIA

There are five criteria to take into consideration in relation to monitoring and evaluation. These are:

relevance, effectiveness, efficiency, impact and sustainability (OECD-DAC Principles and Standards).

Table 2 presents these criteria, their definitions according to the OECD-DAC glossary, and a sample of

questions in relation to each.

TABLE 2. MONITORING AND EVALUATION CRITERIA

Criteria and definition

Sample of questions that can be asked

Relevance: The extent to which the

objectives of a development intervention are

consistent with beneficiaries’ requirements,

country needs, global priorities and partner

and donor policies.

 Is/was the project the right project given the

situation?

 Does the project address real problems and the

roots/causes?

 Does/did it deal with the right target group?

 Is/was it consistent with existing (donor/government)

policies?

Effectiveness: The extent to which the

development intervention’s objectives were

achieved, or are expected to be achieved,

taking into account their relative importance.

 To what extent are/were outputs and outcome

achieved?

 Is/was the intervention logic (see § 2.1.2) well

designed and feasible?

Efficiency: A measure of how economically

resources/inputs (funds, expertise, time, etc.)

are converted to results.

 Are/were resources used in the best possible way?

 What can/could be done differently to improve the

implementation at an acceptable/lower cost?

Impact: Positive and negative, primary and

secondary long term effects produced by a

development intervention, directly or

indirectly, intended or unintended.

 To what extent has the project contributed towards

the overall (long- term) objective?

 What unexpected positive or negative consequences

did the project produces? And the reasons for that?

Sustainability: The continuation of benefits

from a development intervention after major

development assistance has been

completed.

 To what extent one can expect the change/new state

to exist in the future without external inputs?

2.4. RATIONALE FOR MONITORING AND EVALUATION

Reasons for monitoring and evaluating projects include: accountability, performance improvement, learning,

communication and empowerment of primary stakeholders.

Accountability

Projects are implemented by staff on behalf of different stakeholders, e.g. donors, government, hosting

organization and beneficiaries. Therefore, it is important that project implementers are held responsible for

their actions, i.e. they are accountable to all stakeholders.

Improving performance

Monitoring identifies the extent to which a project is making progress in producing expected outputs and

achieving a desired outcome. Corrective measures can be taken in time to improve project performance.

Ongoing monitoring also allows the assessment of whether inputs and resources are being used efficiently.

Learning

Monitoring and evaluation can provide valuable lessons for other projects within the same country or

organization. These lessons may be used for the ongoing cycle, or can be applied during the next

programming cycle to repeat successes or to avoid failures.

Communication

M&E activities improve the communication between different stakeholders, thus enabling a better

understanding of implementation issues and supporting better achievement reporting. In order to make

communication effective, a favourable environment for exchange and discussion is essential. Clear and

transparent communication mechanisms such as regular meetings, workshops, reporting, and information

sharing via internet or printed media should also be established.

Empowerment of stakeholders

M&E creates opportunities for beneficiaries to provide useful feedback to the implementers. Furthermore, the

involvement of different stakeholders in the process can increase their motivation and skills for planning and

implementing future projects. M&E can thus strengthen the participation of primary and end beneficiaries in

decisions about project performance, and therefore increase stakeholder ownership.

2.5. CONSIDERATIONS FOR M&E WITHIN THE TC PROGRAMME

Taking into consideration the particulars of the TC programme as presented in the introductory section, M&E

activities are implemented within the TC programme as one single function (including monitoring and self-

evaluation), aligned with the structure and responsibilities of the TC Department. The function related to

external independent evaluation is the responsibility of the Office of Internal Oversight Services (OIOS) of the

IAEA

The Office of Internal Oversight Services (OIOS) was established in April 2001 by the Director General to strengthen

the Agency’s internal oversight services, its ability to ensure management efficiency, programme effectiveness and to

enhance accountability. Its functional areas are: Internal Audit, Programme Evaluation, Management Services, and

Investigation.

The following objectives are expected to be achieved through M&E implementation within the TC

programme:

a. To increase the performance and accountability of TC projects;

b. To improve communication and participation of TC stakeholders;

c. To enhance learning and continuous improvement within the organization;

Figure 1 presents the main M&E processes during the TC project cycle.

More information on the scope of M&E functions is presented in Table 3. The involvement of relevant

stakeholders must be ensured at each stage.

Figure 1: M&E in TC project cycle

The TC programming cycle is divided into three main phases: Planning & Design; Implementation; and End

Review

. Each stage implies different M&E functions.

The first phase of the TC programming cycle consists of project identification, planning and design. This

includes upstream work and the preparation of national, regional and interregional programme notes. These

notes give an overview of the national, regional or interregional programme and include project concepts.

This is followed by the preparation and design of project documents using the logical framework approach

(LFA), and the elaboration of a detailed workplan and budget.

During this first phase, it is essential to consider M&E tasks that will be undertaken later on. This requires the

development of an M&E matrix (or plan). It is also necessary at this stage to ensure that TC quality criteria

are embedded in the project design

See the TC Operations Manual for more details.

The quality criteria for TC programme/projects are based on the central criterion of the Technical Cooperation Strategy

and the logical framework approach. They are: relevance, ownership/commitment, sustainability, effectiveness,

efficiency. See PCMF reference desk, http://pcmf.iaea.org/ for more details.

Develop an M&E matrix

Implement M&E tasks

Conduct self-evaluation

Adjustments

Lessons learned

1- PLANNING & DESIGN 2- IMPLEMENTATION 3- END REVIEW

M&E IN TC – PROJECT CYCLE

TABLE 3. SCOPE OF M&E FUNCTION WITHIN THE TC PROGRAMME CONTEXT

Planning and Design

Implementation

Review

 Development of an M&E

Matrix with M&E tasks to be

included in the project

workplan

 Ensure TC quality criteria are

embedded in project design

 Carry out M&E tasks by

applying relevant tools and

methods

 Ensure that quality criteria

are applied during the

implementation of projects

 Conduct end-project M&E

tasks (self-evaluation/

assessment)

 Document and disseminate

lessons identified for

continual improvement

During the implementation phase, intended project results are delivered, resources are managed, and

progress is monitored and reported for decision making purposes. At this point, the M&E tasks (defined at

the design stage) are carried out as planned. These include information gathering and analysis, preparation

of reports and dissemination of information. The need to ensure the application of quality criteria during

implementation must also be considered here.

The end of the programming cycle corresponds to the review phase. At this stage, the M&E function includes

conducting self-evaluations or self-assessments, and the documentation and dissemination of lessons to be

learned

. The findings will be used for continual improvement of the TC programme, but also will serve as

inputs for independent, external evaluations conducted under the responsibility of the OIOS

There is difference between lessons identified and lessons learned (or learnt). The process of M&E implies identifying

lessons which should later be learned, i.e. taking actions (or decisions) toward continual improvement.

The IAEA – OIOS evaluation policy states (§9 and 10): “Agency evaluations fall into two categories: 1) independent

evaluations conducted or coordinated by OIOS; and 2) self-assessments carried out by staff involved in the design or

implementation of the programme. Although self-assessments do not replace OIOS evaluations, the results of self-

assessments will be used as references by OIOS when planning and carrying-out independent evaluations. Management

functions and support activities also come within the scope of an evaluation”. See SEC/DIR/122 - OIOS Evaluation Policy

and updates to Part III of the IAEA Administrative Manual.

3. PLANNING FOR MONITORING AND EVALUATION

This module covers the design stage of a TC project and presents M&E issues that must be addressed at

this stage. The logical framework approach is the methodology applied in the design of a TC project, and is

therefore the starting point. The approach leads to the Logical Framework Matrix (LFM). From the LFM, one

can move forward to the M&E matrix in which details needed for M&E purposes are included. The sections

below present the process for developing the M&E matrix (for TC projects) and explain the related elements.

3.1. STARTING POINT: THE LOGICAL FRAMEWORK APPROACH (LFA)

The LFA is the methodology adopted and used for the design of TC projects. This is aligned with the results-

based management approach applied within the Agency. A capacity building package and related material

has been developed specifically for TC projects.

3.1.1. Logical framework approach in summary

The introductory section of the manual ‘Designing IAEA Technical Cooperation Projects using the Logical

Framework Approach: A Quick Reference’ states (p.7):

“The LFA helps stakeholders to think through and analyse the “logic‟ of a project in a systematic and

structured way, first by conducting a detailed analysis of a number of elements, and secondly by relating the

results of these analyses to each other and to the overall project objective. This ensures a sound project

proposal and a high quality project. The LFA provides a project structure in which major components are

explicitly and clearly interrelated, and interrelationships clarified. The LFA plays a particularly critical role in

project planning and design, but it can also be used throughout the project cycle, including during monitoring

and evaluation.

The LFA is essentially a sequence of analytical steps, comprising a situation analysis that reviews project

context and relevance, a stakeholder analysis that covers counterpart mandate and vision, end-users and

any other organizations or group or institution having an interest or being affected by the project, a problem

analysis that examines the problem in detail from the perspective of different stakeholders, and finally an

objectives analysis where the project team decides on the scope of the project. On the basis of these

analyses, the project team constructs a Logical Framework Matrix (LFM) that summarizes the project, and

shows the logical linkages between the project elements. This is an iterative process of testing, review and

validation that then continues with the preparation of a suitable work plan”.

The typical LFM model used for TC projects is presented in Table 4. Concrete examples are presented in

Annexes A and C.

Purpose of Chapter 3:

 Recall notions of logical framework approach (LFA) and result hierarchy

 Explain the linkage between Logical Framework Matrix (LFM) and M&E Matrix

 Describe elements of the M&E Matrix

TABLE 4. TYPICAL LFM FOR TC PROJECTS

3.1.2. Results Chain/Hierarchy of Results

The primary purpose of M&E is to track the performance of a project, i.e. the degree to which it is

implemented as planned and how successfully it achieves its intended results.

The relationship between each result level is described in the LFM and is also known as the Result Chain (or

Hierarchy of Results). When designing the project, it is effective to start by defining the overall objective to

which the project will contribute and defining the expected outcome (the project specific objective). Then the

outputs needed to achieve the outcome are defined, and thereafter the activities and inputs.

When it comes to project implementation, one starts by putting inputs and resources in place, and these are

converted into activities: activities implemented will produce outputs, and the outputs achieved will, in turn,

result in the desired outcome. This logic is known as a Result Chain or the Hierarchy of Results (see Figure

2 and Figure 3). It is necessary that the logic of the project design is clear and straightforward.

Design

Elements

Narrative

Description

Indicators

Means of

Verification

Assumptions

Overall

Objective

Outcome

(Specific

Project

Objective)

Outputs

...

Activities

1.1

1.2

1.3

2.1

2.2

2.3

3.1

3.2

3.3

…

Figure 2: Example of Result Chain

Figure 3: Example of results hierarchy

A result is a change that can be observed, described and measured in some way, and for which the cause

can be identified. Results may a) appear within a short time or take years to be fully realized; b) be planned

or unforeseen; c) be either positive or negative; d) be reflected at the level of individuals, groups of

individuals, institutions, or society.

Different levels of results exist. At the operational level there are immediate results or ‘Outputs’; at the

developmental level, there are medium term results or ‘Outcomes’ (project specific objective) and long term

result or ‘Impact’ (overall objective).

Questions for checking the Design Logic:

 Are inputs necessary and sufficient for activities to take place?

 Are activities necessary and sufficient to achieve outputs?

 Are all outputs necessary? And are all output assumptions necessary and sufficient to achieve the

outcome?

 Are the outcome plus related assumptions necessary and sufficient to achieve the expected

objective?

Input:

Expert

missions

Activity:

Review of

nuclear safety

regulations

Output:

Latest safety

standards

adopted

Outcome

Effectiveness

of regulatory

system

improved

Objective

Sustainability

of the

regulatory

system

enhanced

3.1.3. From logical framework approach to monitoring and evaluation Matrix

Once the LFM has been prepared and the design logic verified, the monitoring and evaluation matrix

(M&EM) can be developed. This should be discussed and agreed among key stakeholders toward the end of

the planning and design phase. This is essential if M&E is to be systematically included in the project

implementation stage.

The M&E matrix outlines concrete steps for the monitoring plan, by providing the what, how, when and who.

It includes:

a. What is to be monitored? i.e. Indicators;

b. How will information be obtained to monitor them? i.e. Data collection methods (these imply M&E

tasks);

c. Who is responsible for the implementation of M&E tasks? i.e. Responsibility;

d. When are the proposed tasks due? i.e. Schedule or timing;

In addition, relevant risks should be considered seriously, anticipated, and included in the M&EM, as well as

the resources needed (if necessary).

Table 5 is an example of an M&E matrix. This is a simplified result-oriented matrix

that can be applied

easily to TC projects. The starting point is the LFM, with some changes introduced in the columns and rows.

Vertically (in the columns), the core elements are: performance indicators (with baseline and target); data

collection methods/M&E tasks; responsibility for M&E tasks; timeframe and risks

Horizontally (in the rows), the activities and inputs are not included. This does not mean that they should not

be monitored (see §3.1.3 on how to do this). The overall objective row is included with the narrative; but it is

not mandatory to fill in the remaining fields (for IAEA-TC projects). Two other rows are added in relation to

the project Implementation Arrangements & Overall Context.

Implementation Arrangements refer to the internal (micro) dynamics of operations, including: interactions,

service delivery mechanisms, management practices, coordination, and the linkages among these.

The overall Context is related to the external (macro) environment of the project, i.e. national

budget/resource allocation, policy & legal frameworks, technology development patterns, the physical

environment and cultural setting.

The overall objective, as well as other issues is not included due to the small size of TC projects. The format highlights

the results at output and outcome levels. More elements can be found in M&E matrices such as: M&E questions, key

variables, communication mechanism, budget/resource etc.

Please note that in the LFM assumptions are presented in relation to the achievement of output and outcome results.

In the M&EM, risks are presented in relation to the implementation of the M&E tasks.

TABLE 5. STRUCTURE OF M&E MATRIX FOR TC PROJECTS

Narrative

elements

Performance

Indicators

(with baseline

& target)

Data

collection/

M&E tasks

Responsibility

for M&E tasks

Schedule

Timeframe

Risks

Overall

objective

------

-----

Outcome

Output

Implementation

Arrangements

Project Context

Because of their importance, it is essential to monitor the project arrangements and context (see §4.1.4).

Only those elements that can have a real impact on the success or failure of the project should be mentioned

in the M&EM.

The following sections explain the different elements in the columns in detail. Two examples of M&EMs for a

TC project are presented in Annexes B & D.

For most TC projects, the LFM will be the basis for the M&E Matrix. Therefore, key M&E aspects and

elements shall be considered when designing the LFM.

3.2. INDICATORS

Indicators are of great importance for M&E. Thus, indicator development is a core step in building the matrix,

and this drives all subsequent data collection, analysis, and reporting.

An indicator is a measure of a concept, phenomenon or behaviour. It does not necessarily reflect the entire

phenomenon itself, but an aspect of it. An indicator provides evidence that a result has been achieved, or

signals that progress is being made towards the achievement of a result.

Indicators enable the measurement of actual achievement against planned or expected results, in terms of

quality, quantity, and timeliness. They always include at least one variable. Therefore, indicators can be:

 Quantitative, i.e. have a numerical value (number, percentage or ratio), e.g. number of technicians

trained; or

The narrative statement of the overall objective is enough for TC projects. It is not essential to define indicators (with

baseline and target) at this level, which corresponds to the long-term impact.

 Qualitative, i.e. reflect perception, judgements or attitudes (e.g. perception of end-users about the

project).

3.2.1. Types of indicators

Indicators can be directly or indirectly related to the result they are measuring.

Direct indicators

These indicators are directly related to the subject of interest. This is often the case with operational and

more technical subjects. What needs to be monitored can be (and generally is) measured directly.

In the example below (Table 6), it will take some years (even decades) to measure the increase in the life

expectancy of cancer patients treated with radiotherapy. For this reason an indirect indicator can be used,

assuming that patients successfully treated will enjoy better and longer life.

Indirect indicators

Indirect indicators (or proxy-indicators) refer also to the subject of interest, but not directly. There can be

several reasons to formulate indirect indicators:

- The subject of interest cannot be measured directly: this is particularly the case for more qualitative

subjects like behavioural change, living conditions, good governance, etc.;

- The subject of analysis can be measured directly, but it is very complex, cumbersome or can take a

long time to do so.

The use of an indirect indicator can be more cost-effective than the use of a direct one. For instance, in the

TC programme context, scientifically validated monitoring data are not necessary, the purpose being to have

reliable/accurate data for management decision making. Therefore, an indirect indicator may represent the

right balance between level of reliability of information and the efforts needed to obtain the data.

TABLE 6. DIRECT AND INDIRECT INDICATORS

Concept or Phenomenon

Indicator

Improved life expectancy

for cancer patients

Direct

Measurement

Proportion of increase in the life expectancy of cancer

patients

Indirect/Proxy

Proportion of cancer patients treated with radio-

therapy that reported no side effects afterwards

The rationality or plausibility of indirect indicators should be ensured so that the relation between the

indicator and what is to be monitored (phenomenon/variable) remains relevant.

Simple and complex Indicators

Indicators can be simple or complex. Simple indicators are straightforward and require only a single

measurement. For example: number of saliva samples analysed; number of mother-infant participating in the

trials; a database is in place.

Complex indicators imply multiple measurements or require different bits of information. They are also called

multiple indicators. They can be combined to form an index, e.g. the Human Development Index of UNDP

includes: school enrolment, life expectancy and income per capita.

3.2.2. Project performance indicators

A performance indicator is defined as a variable that allows the verification (measurement) of change

resulting from an intervention, compared to the prior situation.

For TC projects, performance indicators are defined at leastat output and outcome levels. They have to

include a baseline and target values to the variable to be measured (see table 7).

TABLE 7. EXAMPLES OF TC PROJECT PERFORMANCE INDICATORS

Narrative

Indicator

Variable

Baseline

Target

Outcome

Enhanced effectiveness of

regulatory functions within

the country

Proportion of compliance to

IAEA safety and security

standards

40%

95%

Improved radiotherapy

services for cancer

treatment

Number of cancer patients

treated using radiotherapy

(latest standard)

100

Proportion of patients that

received optimal radiation

doses

100%

Output

Nuclear Regulatory

infrastructures in place

Regulatory authority

established

Legal regulatory framework

adopted

0=No

1=Yes

Dosimetry and radiation

protection equipment in use

Number of dosimeters in use

Qualified technical staff in

place

Number of technicians trained

A baseline is the value of the indicator before the implementation of the project or activities, while a target is

the specific, planned level of result to be achieved within an explicit timeframe. Baselines help project

managers and implementers to determine progress in achieving outputs and outcome. They also help

identify the extent to which change has happened at each level of result. Lack of baseline data presents

challenges for making a decision on the extent to which results (outputs and outcome) are achieved. It also

hinders evaluation efforts at a later stage. Therefore, it is important to consider gathering baseline data or to

reconstruct the data when designing the M&E matrix.

Targets help project stakeholders to focus on results and motivate counterparts to do their best in ensuring

that set targets are met. Targets also help to establish clear expectations. It is essential that key

stakeholders are involved in setting the targets for output and outcome indicators. Milestones can be set for

achieving targets.

3.2.3. Steps for formulating performance indicators

In order to formulate performance indicators, the following steps are suggested.

 Brainstorm to identify the variable(s) that may provide means to measure the change in line with the

result, objective or phenomena. What and how well?

This stage may lead to a number of different options. When indirect variables are identified, check on

reliability, relevance, sensitivity and feasibility before deciding which one(s) will best serve the

monitoring information needs. Cost-effectiveness also needs to be considered for direct indicators

and may well be a reason to select indirect indicators.

 Define the magnitude of the change that is to be achieved. How much?

 Clarify who or what is affected by the change. Often specific information on who (or what) is

necessary. Who or what?

 Specify the location where the change will take place (if necessary). Where?

 Define the timeframe for the change before it happens. When?

Formulated indicators should be SMART, i.e. Specific, Measurable, Attainable, Relevant and Time-bound.

The criteria CREAM – i.e. Clear, Relevant, Economic, Adequate and Monitor-able – can also be used.

3.3. DATA COLLECTION/M&E TASKS

In the LFM, the column ‘Means of Verification’ (MoV) – or sources of information – specifies documents that

provide evidence for the measurement or verification of specified indicators. These can be reports of different

types, accounting documents, direct observations, laboratory results, etc. In some cases it might be

necessary to gather data in order to verify the achievement of performance indicators (outputs and outcome).

Therefore, identifying the types of data and how to obtain them is essential for M&E tasks to be carried out.

In relation to the type of data, one can distinguish quantitative versus qualitative data. Quantitative data are

generally regarded as being more objective, while qualitative data as more subjective. However, recent

debates have concluded that both types of data have subjective and objective characteristics. The fact is,

qualitative and quantitative data complement each other, and both should be used for M&E purposes.

See Imas & Rist 2009, The Road to Results – Designing and Conducting Effective Development Evaluation, The

World Bank, p.117

Definition of SMART

Specific: Is the indicator specific enough to measure progress towards the results?

Measurable: Is the indicator a reliable and clear measure of results?

Attainable: Are the results for which the indicator seeks to chart progress realistic?

Relevant: Is the indicator relevant to the intended outputs and outcomes?

Time-bound: Are data available at reasonable cost and effort?

See UNDP 2009, Handbook on Planning, Monitoring and Evaluation. p.63

Data can be obtained from either primary or secondary sources. While primary data are those that must be

gathered using quantitative and/or qualitative methods, secondary data are those that have already been

collected and published. Attributes of quantitative and qualitative data collection methods are presented

below. One complements the other. A review of documentation is necessary at the beginning. This is

particularly useful in obtaining baseline information and understanding the context in which the project is

evolving, and can help to explain whether expected changes are occurring or not and the reasons behind

this

3.3.1. Quantitative methods

With quantitative methods, things are either measured or counted, or questions are asked according to a

defined questionnaire so that the answers can be coded and analysed numerically.

Quantitative methods help to answer questions such as “who?”, “how much?”, and “how many?”. Where

probability sampling is used, statistical analysis will provide precise estimates for study variables, such as:

frequencies, averages, ranges, means, and percentages, at a known and quantifiable degree of confidence.

Quantitative methods allow the identification of major differences in the characteristics of (or conditions

affecting) a population. They also determine whether there is a statistical relationship between a problem and

an apparent cause, and allow generalizing to the larger population based on data from samples. However,

they tend to ‘simplify’ the reality.

Examples are:

 Direct measurements

 Observations

 Census/counting

 Questionnaires (quantitative)

3.3.2. Qualitative methods

Qualitative methods help to capture an in-depth picture of a specific issue from a relatively small sample of

people. They help to answer questions such as “how?” and “why?”. The focus is on presenting perceptions,

judgments, and opinions, and also on explaining meanings, processes and reasons.

Due to their relatively simple nature and small size, TC projects do not need complex data gathering tools and

sophisticated analysis.

Quantitative methods

Strengths

Precise estimates backed by statistical theory; highly valuable for decision-making and advocacy

because they are robust and objectively verifiable providing that data are collected and analysed

correctly.

Weaknesses

The greatest weakness of the quantitative approach is that it can take human behaviour out of context

in a way that removes the event from its real world setting. Factors or variables left out of the data

collection instrument are simply not considered in analysis.

Qualitative methods are flexible. Questions are asked in an open-ended way and the findings are analysed

as data are collected. Data are collected through much more flexible tools and techniques. Research guides

and tools are reformulated during the process and new ones may be added. The evaluators/researchers

must adapt, using flexibility to probe, while maintaining enough structure to allow systematic analysis of data.

Qualitative methods are intended to explore issues and allow more in-depth examination, but are less

generalizable to a broader population and include the risk of superficiality, bias, and errors. They use

informal approaches to capture differences and provide a more holistic approach to the reality.

Examples of qualitative methods:

 Focus group discussion

 Strengths, weaknesses, opportunities and threats (SWOT) analysis

 Most Significant Change (MSC)

 Mapping

Some qualitative data gathering methods/tools are described in Annex L.

3.4. FREQUENCY OF AND RESPONSIBILITIES FOR M&E TASKS

3.4.1. Frequency

A number of aspects are important for determining the frequency or intervals of observation:

 Steering needs: How often does a project need information for project steering and accountability?

On a monthly or quarterly basis? Or does it suffice to record annually or at even longer intervals?

 Velocity of change: While rainfall or temperature can be constantly monitored, the growth of plants or

the establishment of a nuclear power plant (NPP) can be monitored in much longer intervals.

 Availability of resources, both financial and human.

For typical TC projects, observation intervals of six months to a year are sufficient to capture changes in the

output indicators. The recording of progress made in relation to outcome indicators may be done

simultaneously, but it is highly likely that the information needed may only be available towards the end of, or

after, the two-year project cycle.

Qualitative Methods

Strengths

The strengths of using qualitative methods are that they generate rich, detailed data that keep the

participants' perspectives intact and provide a context for their behaviour. Respondents provide their

own explanations in a participatory exchange with interviewers.

Weaknesses

The weaknesses of using qualitative methods are that data collection and analysis may be labour

intensive and time-consuming. As a result the number of respondents to which the method is applied is

usually far fewer than for quantitative methods. Another disadvantage is that qualitative methods are

often not objectively verifiable.

3.4.2. Responsibilities for monitoring and evaluation tasks

Responsibilities for M&E specify who will be carrying out M&E tasks during project implementation. In the

context of the TC programme, the main stakeholders concerned are: counterparts (CPs), Designated Team

Members (DTMs), coordinators (for regional projects) and National Liaison Officers (NLOs)/ National Liaison

Assistants (NLAs) on one hand, and the IAEA Secretariat, i.e. the TC Quality Assurance Section (TCQAS) of

the Division for Programme Support and Coordination (TCPC), the regional Divisions (RDs) through the

Programme Management Officers (PMOs), and the technical Departments (TDs) through the Technical

Officers (TOs) on the other hand.

Table 8 presents M&E responsibilities by type of stakeholder. These responsibilities are the same for almost

all TC projects.

TABLE 8. M&E RESPONSIBILITIES FOR TC PROJECTS

Actors

Roles & Responsibilities

CPs, DTMs, or

Coordinators

 Design the project and include M&E considerations;

 Carry out M&E tasks, i.e. develop and maintain a sound information system to

track progress and achievements;

 Prepare the Project Progress Assessment Reports (PPARs) and submit them on

time;

 Organize periodic review meetings on project implementation;

 Ensure the quality of M&E information and reports;

 Conduct final assessment at the end of the project in order to document and

disseminate lessons to be learned.

NLOs/NLAs

 Ensure that M&E considerations are included at planning, design and

implementation stages;

 Ensure that projects are designed in line with TC criteria and standards;

 Organize reviews of the country programme;

 Ensure preparation and timely submission of PPARs by project CPs;

 Oversee the implementation and monitoring of all TC projects within the country.

IAEA RDs/TDs

(PMOs and

TOs)

 Ensure that methods (e.g. LFA) and tools (e.g. PPAR, self-evaluation, etc.) are

applied as appropriate, and provide capacity building opportunities for that

purpose;

 Ensure quality standards at all stages of the project cycle;

 Provide technical and operational support to CPs for the implementation of M&E

tasks, including the usage of appropriate tools;

 Undertake field monitoring missions of TC projects;

 Facilitate the identification of lessons to be learned at all stages of the project

cycle.

Actors

Roles & Responsibilities

TCPC/TCQAS

 Develop tools and methods for M&E of TC projects;

 Provide training on M&E tools and methods for TC stakeholders;

 Provide technical support to RDs for the application of M&E tools and methods;

 Provide assurance of quality standards at all stages of the project cycle;

 Undertake field monitoring missions for samples of TC projects.

3.5. RISKS RELATED TO MONITORING AND EVALUATION IMPLEMENTATION

It is important to anticipate external factors that can hamper the implementation of planned M&E activities

and events, because the success or failure of the M&E system can strongly depend on them.

At the technical level, risks encompass a lack of capabilities/capacities that can be related to skills and/or

human resources. There may be also issues relating to the sensitivity of data to be collected for M&E

purposes: this is particularly relevant in some IAEA fields of activity.

At the institutional level, the multiplicity of donor procedures may be a challenge when harmonizing the M&E

system. In such a situation, a greater involvement of key stakeholders or actors is needed from the beginning

in order to develop the M&E system.

Addressing the challenges of accountability/transparency should be supported by sufficient resource

allocation for M&E activities. In anticipating possible risks at the planning stage (during the design of the

M&EM), strategies to minimize their burden can be identified and addressed at an early stage.

4. IMPLEMENTING MONITORING AND EVALUATION FOR TECHNICAL

COOPERATION PROJECTS: PRINCIPLES AND TOOLS

As explained in the previous module, the entry point for developing an M&EM is the LFM, which contains the

hierarchy of results. Indicators in the matrix are given for each level of results (output and outcome),

implementation arrangements and context. Data gathering methods are provided for each indicator.

When designing the project one moves from the top downwards along the result hierarchy, and when

implementing it is the other way around. Therefore, M&E starts as soon as inputs and resources are in place.

In this module, the principles and methods to observe when conducting M&E at each level of the hierarchy

are discussed. These are followed by the presentation of tools and mechanisms applied for TC projects.

4.1. PRINCIPLES OF MONITORING AND EVALUATION WITHIN THE TC

PROGRAMME

M&E efforts within the TC programme should, at a minimum, include the following aims:

 Assess progress made towards achieving the expected outcome. This entails analysing the extent to

which the intended outcome is going to be achieved (after completion of the project);

 Highlight factors contributing to, or impeding the achievement of, the outcome. This necessitates

monitoring the project context and assumptions;

 Assess whether or not outputs are being achieved as planned and the extent to which they will

contribute to the outcome. This also entails monitoring the implementation arrangements;

 Analyse timeliness and efficiency in the completion of planned activities;

 Highlight lessons to be drawn for knowledge creation and sharing.

4.1.1. Outcome monitoring

An outcome is achieved through the generation of outputs. Therefore, monitoring an outcome requires a

clear understanding of all contributory outputs. Due to their specificity (often of short term duration and little

volume of funds), outcome monitoring of TC projects should take into account the likelihood that the

expected change will happen after the project is closed. In other words, it is generally not expected that the

outcome will be achieved by the end of the TC project.

Monitoring a TC project outcome implies ascertaining if the expected outcome is likely to be achieved or

whether it will evolve into something else. This is done by tracking any sign, indication or evidence in line

with the indicators in the LFM.

Purpose of Chapter 4

 Clarify key principles for M&E within the TC context;

 Present tools and methodologies for M&E of TC projects.

It is also necessary to regularly check whether the assumptions related to the outcome have materialized,

and therefore, to analyse the context of the project in order to identify lessons to be learned

(see below).

4.1.2. Monitoring outputs

Outputs are obtained by converting inputs and resources through activities. TC projects are normally of a

simple nature, thus the achievement of planned outputs is straightforward once the planned activities are

implemented.

The primary responsibility for producing outputs lies with the project’s main CP. Outputs must be achieved

within the project lifetime.

M&E of TC project outputs can be done on a six-monthly basis. However, if there is a delay in launching the

project, the first M&E progress report can be postponed until the end of the first year.

Monitoring of outputs involves:

a. Analysis of the extent to which outputs have been produced. This is done by comparing the ‘actual’

against the ‘target’ of each listed output indicator;

b. Verification if outputs assumptions have been realized;

c. Analysis of the implementation arrangements in order to highlight important issues and lessons that

can be learned.

4.1.3. Monitoring activities and inputs

In relation to the M&E of activities and inputs of TC Projects, the concern is to ascertain whether:

a. Planned activities and tasks are being (or have been) implemented in a timely manner;

b. Resources are being (or have been) used in an optimal manner (efficiently).

For these purposes, the project operational plan (workplan) and budget established at the planning and

design phase, serve as key reference documents (see Annex E).

The M&E of activities and inputs start automatically with the launching of the project. The project workplan

should identify the main project milestones and the critical path of the project. Deviations occur all the time,

but not all deviations are equally important. M&E should focus on these two aspects. In case of deviations

from the critical path it is essential to identify timely corrective or remedial actions.

The follow up of critical paths and meeting of deadlines is the responsibility of the project CP, while the

responsibility for resource monitoring lies primarily with the Secretariat, which uses various IT platforms (e.g.

PCMF, TC-PRIME, AIPS and ATLAS) for this purpose.

4.1.4. Monitoring the project implementation arrangements and overall context

In order to achieve project results specified in the LFM, it is necessary to learn as a team, and from

interactions with partners, beneficiaries, and other stakeholders. While the design of the LFM and M&EM,

and the implementation of strategies are structural elements, the process of learning refers to the ‘soft’

dimension of project management.

The ultimate goal of monitoring the project implementation arrangements and overall context is to enhance

learning within the organization and among its stakeholders. This relates to how experiences are validated

and turned into lessons to be learned. Sometimes lessons will stem from success, and sometimes they will

See footnote 8

originate from failure. All lessons, however, are beneficial to learn from, in terms of what worked well or what

went wrong. The main questions to ask are:

 What are the lessons to be learned and how can they be disseminated within the project team

and

fed back to the organization as a whole?

 Are experiences and knowledge shared within a project team and its stakeholders?

 What can most likely be replicated in another project phase or component, i.e. transferability?

For lessons to be drawn, project teams need to understand what worked – or did not work – where, with

whom, under what circumstances and why. This requires being able to analyse the project implementation

arrangements and context.

It is only possible to learn lessons when there is enough time to reflect on practice, identify lessons and

publicise them to others, and when others have the chance to absorb and apply the lessons. Ideally project

teams should schedule regular meetings for this type of reflection on the project, and to record monitoring

information. In the first instance, monitoring should give the project team and stakeholders the chance to

discuss/exchange, identify some lessons, and learn from lessons identified in order to improve their practice.

4.2. MONITORING AND EVALUATION TOOLS FOR TC PROJECTS

A range of tools can be applied to M&E. A mixture of tools is recommended in order to ensure that M&E is

balanced with other project management functions, and is useful and relevant in achieving its purpose.

The following tools and mechanisms are proposed for the M&E of TC projects: Project Progress Assessment

Report (PPAR), field monitoring mission, and Self-Evaluation.

4.2.1. Project Progress Assessment Report (PPAR)

The PPAR is the main monitoring tool for TC projects. It is used during the lifetime of a project to capture

progress made towards achieving the expected/planned results. It is also used at the end of the project for

the submission of Project Completion Report (PCR). Its submission is mandatory for CPs (see frequency

§2.4) using the PCMF (web-based) platform (http://pcmf.iaea.org). The information contained in the PPARs

submitted is used by the TC Department to compile reports.

The format is presented in Annexes F and G and an example is given in Annex H. The PPAR comprises six

sections: Basic information, Outputs, Equipment, Overall Analysis, Outcome, and Lessons to

learn/Recommendations. Explanations are provided below for each section.

Basic information:

This is an introductory section that provides information on the country, name of the main CP and institution,

project number and title, year of approval and year of effective start up, total approved budget (by donor),

reporting period (e.g. 01/2011 to 06/2011), and report contributors, i.e. other project members.

The section ends with a question to point out any change that has occurred during the period under review

that has affected the implementation of the project.

Outputs achieved

This section addresses output achievements. The CP is requested to report on outputs that have been

achieved fully, partially or not at all, taking into consideration Actual versus Target for each output indicator.

A TC project team includes CP, NLO/NLA, PMO/PMA, TO and other DTMs (in the case of regional and interregional

projects).

For planned but partially achieved outputs, it is useful to analyse the extent to which progress is recorded

and/or to present the status in terms of activities implemented. For planned outputs that have not been

achieved, it is necessary to explain why there is a gap.

Equipment and human resources

This section is related to the equipment and human resources components (inputs provided by the IAEA). It

includes any issue related to equipment (commissioning, reception, installation, testing and functioning),

fellowships, training, scientific visits, or expert missions/visits. Because these are the main inputs through

which TC support is delivered, it is essential to report on their delivery during the reporting period.

Comments and recommendations

This section includes a self-rating, comments in line with the rating, lessons to be learned, and

recommendations.

Firstly, the respondent is expected to express his or her true opinion of project performance and the support

received from IAEA by rating each aspect on a scale from 1 (very poor) to 5 (very good); thereafter he or she

comments on the rating by giving some explanations.

Secondly, it is essential that lessons identified throughout the reporting period be highlighted (as discussed

in §3.1.4) and that these lead to specific recommendations. More general recommendations can also be

made, but it is necessary to specify to whom each one is addressed.

Outcome progress

The report on outcome is mandatory at project closure and optional at any other time. The aim here is to

analyse the likelihood that the expected change will happen after the project ends. For this purpose, the

template includes self-explanatory bullet points, as follows:

a. To what extent the expected outcome is being achieved.

b. Details/ explanations supporting the statement.

c. Any other achievements.

d. Issues encountered (if any) that affected the achievement of the outcome.

A rapid survey/assessment may be necessary to obtain the required information. For this reason, the self-

evaluation methodology is relevant for assessing the progress made toward achieving the project outcome.

4.2.2. Field monitoring missions

Field monitoring missions (FMMs) are essential for a better understanding of the reality on the ground. They

provide the opportunity to assess the performance of on-going projects and to analyse factors of success

and failures during implementation. It is important that field monitoring missions are implemented according

to international M&E standards of OECD-DAC.

The objective of monitoring visits is to facilitate mutual learning and TC programme improvement, together

with the NLO, CP, as well as other project team members. The scope includes the following M&E criteria:

relevance, effectiveness, efficiency, sustainability and ownership.

Further details are provided in the guidelines presented in Annexes I and J.

4.2.3. Self-evaluation

Self-evaluation is the process of self-reflection during which an individual, group of individuals, or an

institution, critically reviews the: quality, relevance, efficiency, effectiveness and sustainability of the work

they have performed against expected results and/or established criteria. In the TC context, a self-evaluation

can be conducted at both project and country programme levels. The main purpose is to assess the extent to

which the intended results have been achieved (outputs) or are likely to be achieved (outcome), and to

highlight lessons to be learned and recommendations for continual improvement.

The scope of self-evaluations covers the evaluation criteria of relevance, efficiency, effectiveness, and

sustainability/ownership. The process is formative rather than summative even if the implementation is

recommended at the end of projects and country programmes.

The findings are used to inform and improve the next TC programming cycle. The independent evaluation is

the responsibility of OIOS as discussed in §1.5. Further details for self-evaluation are provided in Annexes K

and L.

Sample of Questions for TC Project Monitoring

The main questions that monitoring visits seek to answer include the following:

1) To what extent is the project still responding to a gap/need/priority of the country?

2) To what extent are activities being/been implemented as planned?

3) What are the factors delaying the project implementation and how these are addressed?

4) What mechanisms are in place for the project steering/coordination and monitoring and how are

these effective?

5) Which project outputs have been achieved to date?

6) To what extent do the end users or/and beneficiaries have access to the project products/services

so far?

7) Are gender perspectives taken into consideration in the access/use of the project benefits by end-

users or/and beneficiaries (where applicable)?

8) Which strategies are in place in order to maintain the project benefit after the end of IAEA support?

9) To what extent can other partners (national and international) sustain the project benefits?

10) What lesson(s) can be identified so far and what recommendation(s) can be made?

RESOURCE DOCUMENTS

EUROPEAN COMMISSION, Aids delivery methods: project cycle management guidelines. European

Commission (March 2004.).

EUROPEAN COMMISSION, ROM Handbook: Results-Oriented Monitoring of, European Commission

(April 2012.).

IAEA, Designing IAEA Technical Cooperation Projects using the Logical Framework Approach; A

quick reference guide (2010).

IAEA (OIOS), Guides for Programme and Project Evaluation (2003).

IFAD, Managing for Impact in Rural Development, a guide for project M&E; Rome (2002).

OECD, Glossary of Key Terms in Evaluation and Results Based Management (2010).

UNDG, Results-based Management Handbook: Harmonizing RBM concept and approaches for

improved development results at country level (October 2011).

UNDP, Handbook on Planning Monitoring and Evaluating for Development Results (2009).

WFP, Monitoring and Evaluation Guidelines

WORLD BANK, Self-Assessment in Managing for Results: Conducting Self-Assessment for

Development Practitioners (2005).

WORLD BANK, Ten Steps to a Results-Based Monitoring and Evaluation System (2004).

ANNEXES

A. Example 1 of LFM: Improving a Regulatory Framework

B. Example 1 of M&E Matrix: Improving a Regulatory Framework

C. Example 2 of LFM: Radiotherapy Services

D. Example 2 of M&E Matrix: Radiotherapy Services

E. Example of Action/Work Plan

F. Project Progress Assessment Report (PPAR) Template

G. Guidelines for preparation of PPAR

H. Example of PPAR submitted

I. Guidelines for Field Monitoring Missions

J. Check-list of specific Questions for Monitoring

K. Guideline for Self-Evaluation

L. Sample of Information Gathering Tools/Methods

A. EXAMPLE-1 OF LFM: IMPROVING A REGULATORY FRAMEWORK

The project context is presented below

Design Element

Indicator

Means of

Verification

Assumptions

Overall

objective

Increased nuclear safety

and security in the

country…

No. of incidents reduced

by x% by 2020

Self-

assessment,

national, and

expert reports

Outcome

(project

objective)

Effectiveness of the

regulatory system in the

country improved

% of compliance to IAEA

standards:

baseline=45%, target=

90% by end of 2016

Self-

assessment,

national, and

expert reports.

RASIMS

Government

support

sustained over

time

Outputs

1. Law establishing the

Nuclear Regulatory

Authority harmonized

with the IAEA standards

A new bill introduced for

adoption by the

legislative organ by end

of 2014

Bill document

No delay in the

vote of the law

2. Latest safety

standards in line with

IAEA in place/adopted

100% of national

regulatory guides (3

level legislation) aligned

with IAEA standards and

adopted by end of 2013

Approval

document

----

3. Improved

organizational structure

in place

- A new organogram

approved by end of

2012

- # of staff members

deployed to the NRA

(baseline 20, target 40

by end of 2013)

Activity report

New deployed

staff members

join on due time

and remain on

board

4. System of monitoring

and control of personal

and patients exposure is

introduced

A national dose registry

in place by end 2013

% of personal and

patients over exposure

reported to NRA on

annual basis (baseline

0, target 5%)

Activity report

No delay by the

service provider

in setting up the

system

In this example, the indicators are the same in the LFM as in the M&E matrix (i.e. with baseline, target and

timeline). This is important for the TC projects where the LFM is used as a monitoring plan.

Design Element

Indicator

Means of

Verification

Assumptions

Activities

1.1 Drafting the new law

and implementation

regulations

1.2 Workshop for

discussing the draft law

and implementation

regulations with

stakeholders

2.1 Review of national

nuclear waste safety

regulations versus latest

set of IAEA safety

standards

2.2 Upgrade of the

regulatory framework

following IAEA safety

standards

2.3 Organize a workshop

to discuss and prepare

an action plan for 2013-

2015

3.1 Review of human

resources and

organizational structure

of NRA

3.2 Preparation of

proposals for

development of

regulatory system to

reflect current

challenges

4.1 Establishing national

dose registry

4.2 Quality programme

for individual dosimetry

services

Project justification

The Nuclear Regulatory Authority (NRA) is the main central executive body that regulates the safety

of nuclear energy use and radiation safety in Country XXX. Since its creation in 2002, the NRA has

benefited from IAEA support in improving the application of nuclear technology within the country. As

a result of this support, staff have been trained in different fields of nuclear and radiation safety, and

regulatory documents were drafted aligned with recommendations of experts and international good

practices.

Despite this support, work remains to be done to strengthen the existing regulatory system in

addressing the following challenges: i) improving the law establishing the Nuclear Regulatory

Authority to align it with IAEA standards; ii) improving the organizational structure of the Authority; iii)

adopting the latest safety standards; iv) introducing a system for controlling personnel and patient

exposure.

The current project aims to address these challenges through IAEA supports in terms of: workshops,

training courses on regulatory approaches, scientific visits, fellowships, expert missions, as well as

procurement and supply of equipment.

B. EXAMPLE 1 OF M&E MATRIX: IMPROVING THE REGULATORY

FRAMEWORK

Design

Element

Indicator

(with

baseline and

target)

Data

collection

Responsible

Periodicity

Risks

Overall

objective

To enhance the

sustainability of

the regulatory

system

No. of

incidents

reduced by

x% by 2020

Outcome

Improved

effectiveness of

regulatory

functions within

the country

% of

compliance to

IAEA

standards:

baseline=45%,

target= 90%

by end of

2016

Inspection

reports

Annual NRA

report (NRA

self-

assessment)

Peer review

missions

Senior expert

NRA

Once a

year

Outputs

1. Law

establishing the

Nuclear

Regulatory

Authority

harmonized with

IAEA standards

A new bill

introduced for

adoption by

the legislative

organ by end

of 2014

Documentation

review

Interviews with

experts and

officials and

NLO/NLA

Every six

months

Non availability

of legislative

contact

persons

2. Latest safety

standards

(IAEA) in place

within the

country

100% of

national

regulatory

guides (3

level

legislation)

aligned with

IAEA

standards and

adopted by

end of 2013

Documentation

review

Quarterly

------

3. A revised

organizational

structure

implemented

- A new

organogram

approved by

end of 2012

- # of staff

members

Documentation

review

Quarterly

------

Design

Element

Indicator

(with

baseline and

target)

Data

collection

Responsible

Periodicity

Risks

deployed to

the NRA

(baseline 20,

target 40 by

end of 2013)

4. System of

monitoring of

personnel

exposure and

patients

exposure

introduced

A national

dose registry

in place by

end 2013

% of personal

and patients

over exposure

reported to

NRA on

annual basis

(baseline 0,

target 5%)

Direct

observation

and

documentation

review

Interview of

trained staff

and direct

observation

Quarterly

Implementation

Arrangements

Effective project

management

and

coordination

One project

steering

meeting

organized

every quarter

One PPAR

prepared and

submitted on

time every six

months

Review of

minutes of

meetings

Analysis of

feedbacks

received from

the secretariat

Quarterly

June and

December

each year

None

No feedback

from the

Secretariat

Overall

Context

Government

commitment

A high level

meeting

organized with

officials and

experts once a

year

Documentation

review (both

project and

policies docs)

Director NRA

NLO/NLA

Annually

Non availability

of government

representatives

C. EXAMPLE-2 OF LFM: RADIOTHERAPY SERVICES

Design Element

Indicator

MoV

Assumptions

Overall

objective

To improve the quality

of life for cancer

patients through

radiotherapy

techniques and

nuclear medicine-

based diagnosis

% of increase in life

expectancy of cancer

patients by 2025

Report DHS

National health

database

Outcome

Improved radiotherapy

services within the

country

# of cancer patients

treated using

radiotherapy increased

from N0 to N1.

100% of patients

receiving optimal

radiation doses, by end

of 2016

Clinic patient

Quality Control

report

No change in the

clinic technical

staff and

managers

Output

1. Physical

infrastructure for

radiotherapy in place

and operational

3D Conformal

equipment fully

functional by end of

2014

Progress report

No technical

problem and no

delay

2. Qualified staff

available in

radiotherapy and

radiation safety

6# of trained technicians

on board by mid-2013

List of staff

Trained staff

remained on

board

3. Dosimetry and

radiation protection

equipment in place

and operational

# of dosimeters in used:

baseline=0, target=10 by

end of 2014

Progress report

No technical

problem

encountered

4. Guidelines and

protocols (according

to latest standards)

adopted

100% of technicians

oriented on guidelines

and protocols by June

2014

Progress report

----

See footnote No.17.

Design Element

Indicator

MoV

Assumptions

Activities

1.1- Implement prior

actions required for

the installation of the

equipment

1.2- Delivery and

initial testing of

radiotherapy

equipment

2.1- Participation in

the training on

radiotherapy

2-2- Participation in

training in radiation

safety

3.1- Commissioning of

dosimetry equipment

3.2- Installation and

initial testing

4.1- Develop

guidelines and

protocol

4.2- Organize a

consultation on

guidelines and

protocols

D. EXAMPLE 2 OF M&E MATRIX: RADIOTHERAPY SERVICES

Indicators

(+ baseline and target)

Data collection/

analysis

Responsible

Periodicity/

Timeframe

Risks

Overall objective:

To improve the quality

of lives for cancer

patients

% of increase in life expectancy

of cancer patients by 2025

--------

Outcome:

Improved radio therapy

services within the

country (latest

standards)

# of cancer patients treated

using radiotherapy increased

from N

to N

100% of patients receiving

optimal radiation doses, by end

of 2016

Hospital/clinic

registry data retrieval

& analysis

Quality control report

analysis

Annually (from

2014)

End 2014, 2015

and 2016

None

Output 1:

Physical infrastructure

for Radiotherapy in

place and operational

3D Conformal equipment fully

functional by end of 2014

Activity and

technical reports

analysis

Lead technician

Quarterly (from

mid of 2014)

None

Output 2:

Qualified staff

available in

radiotherapy and

radiation safety

6# of trained technicians on

board by mid-2013

Nominative list of

staff

Interview with HR

Lead technician

Mid and end 2013

and 2014

None

Indicators

(+ baseline and target)

Data collection/

analysis

Responsible

Periodicity/

Timeframe

Risks

Output 3:

Dosimeters and

radiation protection

equipment in place

and operational

# of dosimeters in used:

baseline=0, target=10 by end of

2014

Activity reports

Content analysis

Lead technician

Quarterly (from

mid of 2014)

None

Output 4:

Guidelines and

protocols (according to

latest standards)

adopted

100% of technicians oriented

on guidelines and protocols by

June 2014

Activity and mission

reports

Content analysis

Lead technician

Mid and end of

2014

None

Implementation

Arrangements

Effective project

management

At least one quarterly review

meeting

Meeting minutes

Content analysis

Quarterly

None

Overall Context

Interaction with the

regulatory body

At least one meeting in a year

with experts of the regulatory

body

Meeting minutes

Content analysis

Annually

Non availability of

regulatory body

experts

E. EXAMPLE OF WORK/ACTION PLAN

(OUTPUT /) Activities

Responsibility

(MS, IAEA, Others)

Inputs

(e.g. FE, SV, EX, PR, TRC,

meeting, cash)

Funding Source

(IAEA, Govt.

Cost-Sharing,

MS, Other)

Quantity

(Q)

Rate (R)

(see table in next

page for IAEA

inputs)

Budget

(=QxR)

Start

End

Output 1: (From the LFM)

1.1 …

1.2 …

…(add lines as needed)

Output 2: (From the LFM)

2.1…

…(add lines as needed)

Output 3: (From the LFM)

3.1…

3.2…

…(add lines as needed)

40 40 40 40 40 40 40

F. PROJECT PROGRESS ASSESSMENT REPORT (PPAR)

TEMPLATE

Explanations

SECTION-1: BASIC INFORMATION

Country

This information will come from

the system when filled in the

PCMF.

Please complete manually when

filling in this template

Counterpart Name &

Institution :

Project Number and

Title:

Year of Approval:

i.e. first approval year

Effective Starting Date:

Month and year

Expected End Date:

Month and year

Total Project Budget:

IAEA TCF:

Other funding :

Please specify the currency

Reporting Period

Specify: from month/year to

month/year

Report Contributors

Other contributors to the report

besides counterpart

Has there been any

change that negatively

affected the project

implementation? If yes,

please explain.

☐ Change of project team member:

(☐ CP, ☐ NLO ☐ PMO ☐ TO);

Explanation …………

☐ Change in budget/funding;

Explanation ………

☐ Other change;

Explanation ………

SECTION-2: OUTPUTS ACHIEVEMENT (mandatory for PPAR and project closure report)

Please refer to the project LFM and provide the following information

Outputs achieved as the

results of activities

implemented

Fully achieved

Present what has been achieved

against planned target for each

output and its indicator of the

LFM. Attach relevant

documentation as needed.

Outputs partially achieved

or in progress and status

Partially achieved or in progress:

For each partially achieved output,

explain status of progress made

and related implementation issues

(if any)

Outputs not achieved and

reasons

Not achieved:

For each non-achieved output,

explain why.

SECTION-3: EQUIPMENT & HUMAN RESOURCES (mandatory for PPAR and project closure report)

Please explain issues

related to the equipment

component.

This can be related to request,

reception, commissioning,

installation, testing or

functioning.

Please explain issues

related to the human

resource (HR) component.

In relation with fellowship,

training, experts, and scientific

visits.

SECTION-4: COMMENT AND RECOMMENDATIONS BY CP (mandatory for PPAR and project closure report)

Rating by CP: So far, how

would you rate on a scale

of 1 (very poor) to 5 (very

good)?

1) Your project performance:

1 ☐ 2 ☐ 3 ☐ 4 ☐ 5 ☐

1= very poor

2= poor

3= fair

4= good

5= very good

2) The support received from the Agency:

1 ☐ 2 ☐ 3 ☐ 4 ☐ 5 ☐

Comment(s) by CP

Comment(s) supporting the

previous ratings

Lessons learned

Highlight factors of successes

and failures

Recommendation(s) by

Indicate to whom the

recommendation is addressed

e.g. IAEA (TO, PMO or other), the

NLO, the Government…

SECTION-5: OUTCOME PROGRESS: (mandatory for PROJECT CLOSURE REPORT (PCR), Optional for PPAR)

Outcome statement

To be selected from the project

LFM

Indicator(s)

1) Please state to what

extent the expected

outcome is being

achieved.

Progress in relation to the

likelihood that the expected

outcome will be achieved or not

2) Please provide details/

explanations

supporting the

statement.

Provide examples, (field)

observations, or signs.

Attach any document supporting

your statement

3) Please state any other

achievements.

Spin-offs, unexpected/unplanned

benefits or negative effect(s)

4) Please explain issues

encountered (if any)

that affected the

achievement of the

outcome.

Issues can be related to the

overall project context

SECTION-6: CLEARANCE BY NLO (mandatory for PCR and PPAR)

Clearance by NLO

Date:

Remarks:

Kindly provide remarks or

comments, if any

SECTION-7: FEEDBACK BY IAEA ON THE REPORT

Comments by TO(s)

Feedback from the TO(s) on the

report

Comments by PMO

Feedback from the PMO on the

report

G. GUIDELINES FOR PREPARATION OF PPAR

SECTION-1: BASIC INFORMATION

Country

(1)

Counterpart Name &

Institution :

(1)

Project Number and

Title:

(1)

Year of approval:

(1)

Effective starting date:

Effective starting month and year after the project was first approved

Expected end date

(year):

Expected end date (month and year)

Total Project budget:

IAEA TCF: i.e approved fund by IAEA TC at the beginning

Other fundings: fund(s) from other source(s)

Reporting Period

From month/year to month/year

Report Contributors

Names of other project team members that contributed to the report

Has there been any

change that negatively

affected the project

implementation? If

yes, explain

Tick appropriate box and explain the issue

☐ Change of project team member:

(☐ CP, ☐ NLO ☐ PMO ☐ TO); Explanation_________________

☐ Change in budget/funding; Explanation________________________

☐ Other change;

Explanation___________________

(1)

Uploaded automatically when using PCMF. To be completed manually if using the Word template.

SECTION-2: OUTPUTS ACHIEVEMENT (mandatory for PPAR and project closure report)

Kindly refer to the project LFM and assess achievements in line with each planned output. This shall be done by

comparing the “actual” against the “target” for each output indicator. Be advised that outputs are neither

activities nor inputs. Outputs are direct results of activities completed by converting inputs.

Examples: Fellowship is the Input; Participation in a training course or capacity building workshop is the

Activity and the resulting Output is the availability of qualified staff member(s) within the institution.

Outputs achieved as the

results of activities

implemented

Fully achieved

i.e. 100% or more realized compared to target.

Outputs partially

achieved or in progress

and status

Partially achieved or in progress:

i.e. less than 100% realized compared to target or achievement is underway (e.g. of output

in relation to the approval or adoption)

Outputs not achieved

and reasons

Non achieved:

Neither achieved (fully or partially) nor in progress; Reasons for this can include activities

not yet planned, impediments, change in project context etc.

When re-submitting the PPAR at any time of the project cycle, it is highly recommended to take the last

version submitted and update this.

SECTION-3: EQUIPMENT & HUMAN RESOURCES (mandatory for PPAR and project closure report)

Please explain issues

related to the equipment

component.

Issues can be related to the: request, delivery, commissioning, installation, testing,

operation or functioning of equipment.

Please explain issues

related to the human

resource (HR) component

This can be related to fellowship, training, scientific visits, or expert visits.

SECTION-4: COMMENTS AND RECOMMENDATIONS (mandatory for PPAR and project closure report)

This section includes: self-assessment (or rating), comments in line with the rating, lessons learned and

recommendations.

Rating by CP: So far, how

would you rate on a scale

of 1 (very poor) to 5 (very

good)?

The respondent (CP) is expected to express his/her true opinion on the project

performance and the support received from IAEA.

1) Your project performance:

1 ☐ 2 ☐ 3 ☐ 4 ☐ 5 ☐

2) The support received from the Agency:

1 ☐ 2 ☐ 3 ☐ 4 ☐ 5 ☐

Comments by CP

Provide comment / explanation that supports your previous rating

Lessons learned

Identification of factors of successes (what went well) or / and failures (what went

wrong) in terms of: how, why, with whom, under what circumstances and so what.

Lessons learned are mainly related to the implementation arrangements and the

overall project context.

Recommendation by CP

Recommendations shall derive from lessons learned. It is essential to indicate to

whom a specific recommendation is addressed.

SECTION-5: OUTCOME PROGRESS (mandatory for PROJECT CLOSURE REPORT (PCR), Optional for PPAR)

1) Please state to what

extent the expected

outcome is being

achieved.

An outcome is achieved after planned outputs are realized. The point here is to

ascertain if the expected outcome is likely to be achieved. Thus, explain any progress

already recorded in line with each outcome indicator.

2) Please provide details/

explanations

supporting the

statement.

Any example, sign, or (field) observation in line with your previous response shall be

reported. This will be useful for identification of cases for success stories. Attach any

documentation supporting your statement.

3) Please state any other

achievements.

Report spin-offs, unexpected/unplanned benefits or negative effect

4) Please explain issues

encountered (if any)

that affected the

achievement of the

outcome.

Issues can be related to the assumptions of the LFM (outcome and outputs levels)

and also to the project context and implementation issues. Report bottlenecks or

problems encountered not already mentioned above.

SECTION-6: CLEARANCE BY NLO (mandatory for PCR and PPAR)

Clearance by NLO

Date of clearance by the NLO and feedback, if any.

SECTION-7: FEEDBACK BY IAEA ON THE REPORT

Comments by TO(s)

Feedback from the TO(s), after the report is submitted by the CP

Comments by PMO

Feedback from the PMO, after the report is submitted by the CP

H. EXAMPLE OF PROJECT PROGRESS ASSESSMENT

REPORT (PPAR)

Explanations

SECTION-1: BASIC INFORMATION

Country

Country-M

This information will come from

the system when filled in the

PCMF.

Please fill it manually when

filling in this template

Counterpart Name &

Institution :

Mrs. A. E. N.

Agriculture Research Institute of Country-M, Directorate of

Animal Sciences , Ministry of Agriculture

Project Number and

Title:

M/5/002, Promoting Sustainable Animal Health, Reproduction

and Productivity Through the Use of Nuclear and Related

Techniques

Year of approval:

2009

i.e. first approval year

Effective starting date:

01/2009

Month / year

Expected end date:

12/2013

Month / Year

Total Project budget:

IAEA TCF: USD 324,265

Other fundings: None

Please specify the currency

Reporting Period

January 2010 to December 2011

Specify: from month/year to

month/year

Report Contributors

1- S. A.

2- L. M.

3- P. D.

Other contributors to the report

other than counterpart

Has there been any

change that negatively

affected the project

implementation? If yes,

explain.

☒ Change of project team member

(☒ CP, ☐ NLO ☐ PMO ☐ TO);

Explanation: Two local team members were changed due

respectively post graduate studies and other commitments

☐ Budget/funding; Explanation________________________

☐ Other; Explanation___________________

SECTION-2: OUTPUTS ACHIEVEMENT (mandatory for PPAR and project closure report)

Please refer to the project LFM and provide the following information

Outputs achieved as the

results of activities

implemented

Fully achieved

1. The Central Veterinary Laboratory (CVL) was upgraded

as planned, due to new equipment received, installed

and in use, to accommodate serological and molecular

techniques;

2. Capability of CVL staff members enhanced due to

trainings received: 6# in serological and molecular

techniques (ELISA and PCR) as planned (100%

achievement); 2# in AI and cryo preservation techniques

(100% achievement); 1# in quality assurance (100%

achievement)

Present what has been achieved

against planned target for each

output and its indicator of the

LFM. Attach relevant

documentation as needed.

Outputs partially achieved

or in progress and status

Partially achieved or in progress:

1. Capacity on disease diagnostic laboratory established on

performing nuclear and related techniques partially

achieved. The laboratory personnel were trained,

equipment and conditions to perform nuclear and related

techniques such as ELISA and PCR, some test kits were

provided by the agency. However, due to procurement and

other administrative issues at the Agency the planned

diagnostics reagents and consumables are delayed. While

the planned techniques have been established there is a

need to validate the established diagnostic tests and to

consolidate the all work, performing the diagnosis and

epidemiological studies of the most important diseases.

At the animal production side, the changes on the team

members of this component and delay on some

consumables partially affected the planned activities.

For each partially achieved output,

explain status of progress made

and related implementation issues

(if any)

Outputs not achieved and

reasons

Non-achieved:

1. Validation of some planned diagnostic tests was not

achieved due to delay on implementation of Lab activities

and acquisition of reagents and consumables.

2. Characterization of Indigenous livestock not performed due

to changes of project staff and other organizational issues

For each non- achieved output,

explain why

SECTION-3: EQUIPMENT & HUMAN RESOURCES (mandatory for PPAR and project closure report)

Please explain issues

related to the equipment

component.

Planned equipment and other needs for 2011 were not procured

due to less availability of funds , this request was revised,

harmonized and planned for 2012

Issues can be related to request,

reception, commissioning,

installation, testing or

functioning.

Please explain issues

related to the human

resource (HR) component.

The project team and other Lab technicians were trained on basic

nuclear and related techniques; however there is a need of an

intensive and periodic in service training for better familiarization

of introduced nuclear techniques and on the use and maintenance

of equipment. This can be done by specific experts missions

In relation with fellowship,

training, experts, and scientific

visits.

SECTION-4: COMMENT AND RECOMMENDATIONS BY CP (mandatory for PPAR and project closure report)

Rating by CP: So far, how

would you rate on a scale

of 1 (very poor) to 5 (very

good)?

3) Your project performance:

1 ☐ 2 ☐ 3 ☐ 4 ☒ 5 ☐

1= very poor

2= poor

3= fair

4= good

5= very good

4) The support received from the Agency:

1 ☐ 2 ☐ 3 ☐ 4 ☒ 5 ☐

Comments by CP

We welcome the support provided by the Agency to meet our

goal, however there is a need to improve our performance,

especially trying to find solutions to the financing of planned local

activities under the project, however the Agency should also make

efforts to allocate the planned resources on time

Comment supporting the

previous ratings

Lessons learned

Improved team working; Established networking in all

components of the project and within national counterparts

Highlight factors of successes

and failures

Recommendation by CP

To IAEA: Improvement of procurement and the process of

equipment and reagents delivery; Provision of more technical

expert missions for identification of real gaps and provision of

recommendations.

To all: improve of communication among actors; Establishment of

an efficient M&E of project activities

Indicate to whom the

recommendation is addressed

e.g. IAEA (TO, PMO or other), the

NLO, the Government…

SECTION-5: OUTCOME PROGRESS (mandatory for PROJECT CLOSURE REPORT (PCR), Optional for PPAR)

Outcome statement

Enhanced diagnosis and control of trans boundary animal

diseases is improving the livelihoods of rural communities and

farmers; Breeding strategies and animal reproduction improved

through better characterization of indigenous/local livestock

From the project LFM

Indicator(s)

Diagnostic techniques on the most important diseases

established (FMD, RFV, Brucellosis, TB and TBD), and a package of

recommendations on diseases status produced by the end of the

project ; Indigenous livestock characterized

5) Please state to what

extent the expected

outcome is being

achieved.

The impact of introduced veterinary diagnostic tools and AI

techniques is being felt throughout the country. It has an effect on

the livelihoods of many communities through the rapid

identification and prevention of the most important and strategic

animal diseases and on the improvement of animal breeding

management

Progress in relation to the

likelihood that the expected

outcome will be achieved or not

6) Please provide details/

explanations

supporting the

statement.

As example, the most recent FMD outbreak in 2010 that occurred

in the South part of the country was timely detected by the ELISA

technique performed at CVL, which helped to control the spread

of the disease to other animals and to other areas. In a

strategized and focused sampling frame from one Province, 31

samples were confirmed as FMD positives out of a total of 189. In

addition, a survey carried out in the same period in one district of

the Province revealed a sero prevalence of about 0.6%. The

extension of the immunological platforms with the molecular

platforms will facilitate the characterization of the circulating

FMD virus at the time, which will help with the matching of

outbreak FMD virus with vaccine FMD virus

Provide examples, (field)

observation or signs.

Attach any document supporting

your statement

7) Please state any other

achievements.

Spin-offs, unexpected/unplanned

benefits or negative effect

8) Please explain issues

encountered (if any)

that affected the

achievement of the

outcome.

Delay on the provision of some Lab reagents and consumables

and on animal production activities; problems of availability of

national funds for field activities could affect the outcome

Issues can be related to the

overall project context

SECTION-6: CLEARANCE BY THE NLO (mandatory for PCR and PPAR)

Clearance by NLO

Date: ______________________________

Remark: ____________________________

Kindly provide remark or

comment, if any

I. GUIDELINES FOR FIELD MONITORING MISSIONS

Field monitoring missions (FMMs) are essential for better understanding the reality on the ground.

They provide the opportunity to assess the performance of on-going projects and analyse factors of

success and failures during implementation.

It is important that field monitoring missions are implemented according to international M&E

standards of OECD-DAC.

Objectives

The objective of monitoring visits is to facilitate mutual learning and TC programme improvement

through the assessment of the performance of on-going projects, together with the NLO, CP, as well

as other project team members.

Expected output

The expected output or deliverable of a FMM is the report presenting findings and conclusions on the

assessment of on-going projects with regard to the following aspects:

a. Relevance of the need(s)/gap(s) being addressed;

b. Progress made in achieving the expected outputs and outcome;

c. Efficiency of implementation arrangements and mechanisms;

d. Incidence of the overall context with regard to sustainability and ownership;

e. Lessons to be learned.

Methodology

The data gathering methods to apply during the FMMs shall be qualitative and participatory. They

include: desk review of documentation (project design and other reports), semi structured individual

and group interviews with relevant stakeholders (e.g. officials, project team members, end users and

beneficiaries), direct observations (of experiences, events and facts), and gathering of evidence (e.g.

pictures, press release, testimonies).

Specific questions to be discussed/covered are presented in the table below as well as suggested

data gathering methods. It might be necessary to discuss and agree on relevant questions/topics

depending on the type of project .

It is essential to start the desk review of available project documentation (e.g. CPF or Regional

Agreement document, project document, progress reports, previous duty travel and expert reports) at

the Secretariat and to meet with the relevant PMO, Programme Management Assistant (PMA) and

TO(s) before the mission. It will help to better understand the context of projects and to clarify

questions and issues that need specific consideration.

Before the visit, a short questionnaire will be sent to CPs receiving the mission and this helps them to

be prepared. The mission will start with a briefing meeting with the NLO/NLA and relevant

stakeholders in order to explain the purpose and strategy of the mission and agree on the agenda and

This tool is intended to be used by Agency staff members and external resource persons (e.g. consultants,

experts) undertaking independent monitoring visits of TC projects. It can also be used for routine project

monitoring conducted by a project stakeholder.

sites to be visited. In the same line, a debriefing meeting shall be organised at the end of the mission

to present key findings and conclusions.

Sample of tasks to be undertaken include: Field project documentation review, meeting and

discussion with project CPs and team members at their respective institutions (2 projects per day),

observation of realizations/achievements and other evidences, discussion with end-users or/and

beneficiaries (if necessary).

Reporting

After the FMM, a report shall be produced (within 2 weeks, if possible) and shared with key

stakeholders. The fields below are proposed:

 City and country visited

 Dates

 Experts

 Projects monitored (no. and title)

 Institutions/sites visited

 Objective

 Activities undertaken

 Findings

 Conclusions, lessons and recommendations

 Appendices

o List of persons met

o Rating table

J. CHECKLIST OF SPECIFIC QUESTIONS FOR MONITORING

The questions are only indicative and shall be selected/adjusted/adapted to each context. It is

recommended to apply more than one data gathering method in order to triangulate and ensure

evidence based monitoring.

Possible data gathering methods

(not exhaustive)

Relevance

 Does the project still respond to a need/priority within the country?

 Are the IAEA role and contribution still relevant to address the gap

identified at the beginning of the project?

 Is the result hierarchy (especially outcome) clear to the project key

stakeholders?

Desk review of documents

Semi structured Interviews with

CPs, NLO and relevant

resource persons...

Focus group discussion

Efficiency

 Have all financial contributions been provided on time?

 To what extent are inputs (equipment and HR component)

available/ put in place on time?

 To what extent are activities implemented as planned and

according to the set deadlines?

 To what extent is the project workplan updated and documented?

 What are the delay factors and how corrective measures are taken

to address these?

 How is the project monitored/steered or coordinated?

Desk review of TC financial and

implementation reports

Semi structured interviews

Direct observation

Focus group discussion

Effectiveness

 To what extent have planned outputs been delivered to date?

 What is the quality of the outputs already delivered?

 Are the outputs achieved (or being achieved) likely to contribute

achieving the expected outcome?

 To what extent do the end-users or/and beneficiaries have access

to the project products/services so far?

 To what extend gender perspectives are taken into consideration in

the access to products or services (where applicable)?

 Is there any unplanned effect – whether positive or negative – that

occurred (or is likely to occur)?

 To what extent did/can the CP/Institution take appropriate

corrective measures?

Desk review of documents

Semi structured Interviews

Review of field documentation

(Press release, official

reports…)

Focus group discussion

Direct observations (facts,

pictures, testimonies)

Sustainability & Ownership

 Is the CP institution able to afford the maintenance and operational

costs of the equipment/technology introduced?

 Are the Human Resources in the CP institution trained and

retained in order to continue the delivering of services?

 To what extent are different local stakeholders involved in the

project implementation?

 To what extent is the project anchored within a programme or/and

strategy of the CP institution?

 What is the likelihood that relevant achieved results (or being

achieved) will be maintained even if a contextual change occurs

(e.g. management, government)?

 Is there strong/good partnership(s) developed in order to sustain

(technically, financially and managerially) the project benefits?

Desk review of documents

Semi structured interviews

Direct observations

Review of field documentation

(Official reports, policies and

plans…)

K. GUIDELINES

FOR SELF-EVALUATION

Definition

Self-evaluation is the process of self-reflection during which an individual, a group of individuals, or an

institution, critically reviews the quality, relevance, efficiency and effectiveness of the work and its

performance against expected results or/and established standards/criteria. When conducted on

projects, self-evaluation highlights achievements as well as areas for improvement, and supports

progress towards project outcome.

Objectives

The objectives of self-evaluation can include:

1. Assess the project achievements;

2. Assess progress made towards achieving the expected outcome;

3. Analyse the implementation approaches, project arrangements and context in order to identify

lessons to be learned;

4. Make specific recommendations.

Scope

Self-evaluation can be conducted at the mid- or end-term of a project or country programme. The

scope covers the evaluation criteria of relevance, efficiency, effectiveness and sustainability.

The monitoring questions presented above are also applicable here. Moreover, self-evaluations are

more analytical in terms of making inferences on the successes and failures of the project by

answering the following questions.

 What has succeeded and/or failed in the project?

 Why did the successes and failures happen?

 Is it necessary to do things differently or utilize different approaches?

 What are the implications for the future in terms of actions and improvements?

Steps to conduct a self-evaluation

A simple methodology is proposed below, aligned with the small size of most TC projects. It includes

the following steps: preparation of terms of references (ToRs), data gathering and analysis, reporting

and usage of findings.

a. Preparation of ToRs: This consists of:

 Clarifying the scope of the self-evaluation (i.e. questions to be answered) and

deliverables;

 Agreeing on tools to be applied (in order to get the right answer to the self-evaluation

questions);

The self-evaluation tools are intended to be applied by NLOs and TC project CPs for ending projects.

A more detailed guidelines on self-evaluation will be developed separately for TC projects and programme.

 Defining stakeholders that should be involved in the process;

 Setting the timeframe for completion of tasks.

b. Data gathering and analysis: for this purpose, quantitative and qualitative tools can be used.

A combination of tools is needed depending of the nature of project, resources and time

available. A documentation review may also be necessary. The following tools are proposed

(not restrictive):

 Direct observation/measurement;

 Survey (formal and quantitative);

 Interviews (semi-structured and informal);

 Focus group discussion (FGD);

 Critical Reflection & Analysis Workshops;

 Strengths, Weaknesses, Opportunity and Threats (SWOT) analysis;

 Successes, Failures, Potentialities and Obstacles (SEPO) analysis;

 Most Significant Change (MSC) technique.

c. Reporting and using findings, this step includes:

 Writing a report. There is no specific format for TC project reports. It is necessary to

have a concise report that presents clearly the methodology (stakeholders, data

collection/analysis methods), findings, conclusions and recommendations;

 Disseminate the report to key stakeholders. In relation to the Secretariat, the findings

of the self-evaluation are incorporated in the PPAR, and the report itself should be

sent as an attachment. The report should also be sent to all other involved partners.

 Implementation of the recommendations made for improvement.

L. SAMPLE OF INFORMATION GATHERING TOOLS/METHODS

Below are presented in summary some information gathering tools/methods that are simple and

easily to apply in the TC context. Some of the suggested tools/methods serve the purposes of

information gathering and analysis simultaneously.

1. Interviews

Interviews aim to collect information and/or views on a specific subject matter. Interviews can be

informal (unstructured), semi-structured, and formal (standardized open-ended). Each type serves a

different purpose and has different preparation and instrumentation requirements.

The informal interview relies primarily on the spontaneous generation of questions in the natural

flow of an interaction. This type of interview is appropriate when the evaluator wants to maintain

maximum flexibility to be able to pursue questioning in whatever direction appears to be appropriate,

depending on the information that emerges from observing a particular setting, or from talking to one

or more individuals in that setting.

Semi-structured interviews involve the preparation of an interview guide that lists a pre-determined

set of questions or issues that are to be explored during an interview. This guide serves as a checklist

during the interview and ensures that the same basic information is obtained from a number of

people. Yet there is a great deal of flexibility. The order and the actual wording of the questions are

not determined in advance. Moreover, within the list of topic or subject areas, the interviewer is free to

pursue certain topics in greater depth.

The formal interview (standardized open-ended) consists of a set of open-ended questions carefully

worded and arranged in advance. The interviewer asks the same questions to each respondent with

essentially the same words and in the same sequence. This type of interview may be particularly

appropriate when there are several interviewers and it is necessary to minimize variations in the

questions they pose. It is also useful when it is desirable to have the same information from each

interviewee at several points in time or when there are time constraints for data collection and

analysis. Standardized open-ended interviews allow the systematic collection of detailed data and

facilitate comparability among all respondents.

2. Focus Group Discussion (FGD)

A focus group discussion is a qualitative and participatory evaluation tool to be used by a trained and

experienced moderator/facilitator with a group of six to twelve people. The discussion – interview is

conducted through a check-list of questions. Participants are asked to reflect on the questions asked

by the interviewer, provide their own comments, listen to what the rest of the group have to say and

react to their observations. The main purpose is to elicit ideas, insights and experiences in a social

context where people stimulate each other and consider their own views along with the views of

others. The interviewer acts as facilitator: introducing the subject, guiding the discussion, cross-

checking each other comments and encouraging all members to express their opinions. It can take

one to one and a half hours.

Generally the group is homogenous in composition so that people with the same social status feel

comfortable enough to give their point of view on a specific topic. The information can be directly

recorded by a tape-recorder, or typed by somebody taking notes.

Focus groups can be used in the monitoring and evaluation of complex projects with a variety of

counterparts. Focus group meetings can help to achieve the following:

 Validate observations or findings on results achieved;

 Qualify the project arrangements and overall context, in particular how things went;

 Validate conclusions and recommendations for improvement.

3. Critical Review Meetings/Workshops

A critical review is a monitoring mechanism that provides an opportunity to project stakeholders to

reflect on “how things are going” or “how things are progressing”. Regular project reviews are

recognized to be part of good management practice in terms of tracking progress, obtaining and

discussing feedback, and mutual support and learning among the project team members.

Critical reviews should include all stakeholders who play important roles in the project. The main

purpose is to increase project performance and mutual learning. The main questions for critical

reviews are:

How are we progressing?

What went well? What went wrong?

What to do differently in future? Ideas?

What can be learned so far from both successes, and challenges?

What future actions might be taken?

During critical reviews, special attention should be given to the ‘assumptions’ and ‘risks’ identified in

the project LFM to ensure that there is no change to this level which could have a negative effect or

impair the implementation or success of the project. Beyond formal meetings, the process of critical

reflection should also encourage informal exchanges of experience between stakeholders.

4. SEPO Analysis

SEPO stands for the French abbreviations of successes (succès), failures (échecs), potentials

(potentialités), and obstacles (obstacles). SEPO analysis is similar to the well-known SWOT analysis.

But while SWOT analysis divides the field of analysis in an internal (strengths, weaknesses) and an

external dimension (opportunities, threats), SEPO analysis focuses on the timeframe.

The SEPO analysis allows assessing the project considering i) looking backward (the past) with

Successes and Failures and looking forward (the future) with Potentials and Obstacles.

Past

Future

Success

 What went well

 Results achieved

 Successful process/events

Potentials

 Assets

 Possible successes

 Unused capabilities

 New challenges

Failures

 What went wrong

 Difficulties/constraints

 Blockages and excesses

 Negative effects

Obstacles

 Handicaps/resistance

 Opposition

 Unfavourable context

 Possible excess

The tool is useful when one intends to proceed in the same direction without major changes (e.g.

continuation of the same project). But if the intention is to change the direction (e.g. a new project) it is

better to use the well-known SWOT analysis method.

5. Example of SEPO application to evaluate a workshop

Past

Future

Success

 What went well

o Organization

o Interaction

o Implementation

o Attendance

o Group activities

o Social programme

 Results achieved

o Refreshed knowledge about LFA

o Better understanding of LFA

o Better understanding of M&E

o New approach on self-evaluation

o Harmonization of LFM-clearer

 Successful process/events

o Interaction was good

o Learning process/discussions-gradual

o Case studies for group discussions

Potentials

 Assets

o Acquired knowledge will help participants

improve current projects, including

implementation, achievement and

evaluation.

o The knowledge will help formulate projects

in future

 Possible successes

o Better implementation of the projects and

evaluation for current and future

 Unused capabilities

o Technical knowledge related to nuclear

techniques and technologies

 New challenges

o Implementation and sustainability of M&E

and also integrating it in projects

Failures

 What went wrong

o Nothing

 Difficulties/constraints

o Too much information to master within

one week

 Blockages and excesses

o None

 Negative effects

o None

Obstacles

 Handicaps /resistance

o Heterogeneity of approach from colleagues

who did not attend the course

 Opposition

o Same as above

 Unfavourable context

o local constraints regarding funding, brain

drain, infrastructure and human resources

 Possible excess

o local abilities do not match with the

availabilities of resources

This was used by participants in a March 2012 workshop to evaluate whether it was successful or not.

6. Most Significant Change (MSC) technique

The Most Significant Change technique is a form of participatory monitoring and evaluation. It is

participatory because many project stakeholders are involved both in deciding the sorts of change to

be recorded and in the analysis. It is a form of monitoring because it occurs throughout the project

cycle and provides information to help manage the project. It contributes to evaluation because it

provides data on impact and outcome that can be used to help assess the performance of the project

as a whole.

Essentially, the process involves the collection of Significant Change (SC) stories emanating from the

field level, and the systematic selection of the most significant of these stories by panels of

designated stakeholders or staff. The designated staff and stakeholders are initially involved by

‘searching’ for project impact. Once changes have been captured, various people sit down together,

read the stories aloud and have regular and often in-depth discussions about the value of these

reported changes.

When the technique is implemented successfully, whole teams of people begin to focus their attention

on project outcome and impact. The technique is especially helpful in identifying and analysing the

unexpected positive and negative outputs and outcome of our project.

It should perhaps be noted that MSC is also a very time-consuming exercise involving trained

facilitators capable of eliciting and drawing out information across various cultures. Furthermore, it is

not clear how easily this may be applied in the context of nuclear technology.

It is also possible to adapt the tool to a specific context. But this can only be done by somebody who

knows the tool and has applied it at least once.

For more details, see http://mande.co.uk/special-issues/most-significant-change-msc/

May 2013

Quality Assurance Section

Department of Technical Cooperation

International Atomic Energy Agency

PO Box 100

Vienna International Centre

1400 Vienna

Austria

Tel.: (+43-1) 2600-0

Fax: (+43-1) 2600-7

Email: Official.Mail@iaea.org

PCMF web site: http://pcmf.iaea.org/

TC web site: www.iaea.org/technicalcooperation

13-XXXX