1. No category

IDEAS Global Assembly
11 – 15 April 2011
Amman, Jordan
Amman, Jordan
Session 6.7
Complexity of Assessing
Evaluation Approaches
Developing Diagnostic Tools
for National Monitoring
g and Evaluation Systems:
y
A Pilot Experience in Latin America
Idania Fernández, Jacques Lefevre, Nadia Croteau, Frederic Martin
Pour un secteur public performant
For public sector performance
Por la excelencia del sector público
© 2008 IDEA International
Context in Latin America
 Also times of change and turbulence!
 Growth
G
h andd overallll emerging
i economies:
i
B il
Brazil,
Chile, but also other countries such as Panama
 Latin America as a region
g
went through
g the
financial crisis relatively well
 At the same time, old issues of structural poverty
andd inequality
i
li still
ill there
h
andd even bigger
bi
b
because
benefits of growth were not shared among all
social categories
 Faced with these persistent and even growing
problems and pressure from citizens and civil
society governments felt the need for new
society,
solutions
© IDEA International 2011
© 2008 IDEA International
Governments’ response to this situation in the region:
New Integrated social protection programs
Main poverty reduction strategy program: Conditional monetary
transfers.
Started in Mexico (Progreso in 1982 which became
Oportunidades in 2000).
2000) Adopted later by:
 Brasil (Bolsa familia en el programa Fome zero en 2000)
 Chile (Chile Solidario en 2004)
 Honduras (Bono 10000, PRAF)
 El Salvador (Programa de Apoyo Temporal
al Ingreso Familiar)
 Panamá
P
á (Red
(R d de
d Oportunidades
O
id d in
i 2006)
© IDEA International 2011
© 2008 IDEA International
Challenges faced by goverments
Very complex programs:
o Many actors, many partnerships
o Multiple activities to conduct, coordinate, and
monitor
o Issues of identification and follow up
p of
beneficiaries
o Cost-effectiveness
issue
of
delivery
mechanism
o Need to demonstrate outcomes and impacts
o Subject to external and internal shocks
Need for a solid results-based M&E
y
system!
© IDEA International 2011
© 2008 IDEA International
Challenges for setting up and implementing a results-based M&E system
SStarting
i or young M&E
M E systems:
o No baseline; systems only designed, but not fully implemented
o Weak logical
g
frameworks linkingg outputs
p to outcomes to impacts,
p
o Data quality issues
o Limited use of results
o Limited human capacities
o Limited culture of results
o Multiple, ad hoc, and donor driven M&E systems for some time; now
i
improving
i with
i h move towards
d a national
i l M&E system
In spite of all those challenges, M&E systems were
put in place. Now the challenge is: to know where
are we in terms of national M&E systems; and how
p
them?
to improve
© IDEA International 2011
© 2008 IDEA International
National M&E system diagnostic tool
Project of SISCA with support from World Bank
Concept paper by Ray Rist
Objectives of the tool:
 Guide a structured and
participatory diagnostic of
the National M&E system
 Guide the formulation of
a priority action plan to
improve the performance
of the system
y
Herramienta de Diagnóstico del
Sistema de Monitoreo y Evaluación de
Programa de Protección Social (MESOC)
Dimensión del
diagnostico
D1
Version 1
04 d
de ffebrero
b
d
de 2011
Estructura del MESOC
Definición
D2
Pre-condiciones para un sistema de M&E basado en
resultados
Mecanismos institucionales de M&E del p
programa
g
D3
Planificación y presupuesto del M&E del programa
D4
Desempeño real del componente de monitoreo del
programa
Desempeño real del componente de evaluación del
programa
D5
Repeat this exercise every 2-3 years for a stepwise
improvement in the M&E system (example of
PEFA approach in public finance)
© IDEA International 2011
© 2008 IDEA International
Diagnostic tool structure - Part I: Diagnostic of the OVC
National M&E system
Key dimensions:
o RBM prerequisites
o Legal and institutional frame
plan and budget
g
o M&E p
o Monitoring implementation
o Evaluation implementation
Key questions: existence, quality,
and use
Quantitative assessment on
an
ordinal
scale
and
qualitative
evaluation
to
justify
the
quantitative
assessment
© IDEA International 2011
© 2008 IDEA International
Diagnostic tool structure - Part II: Action Plan to improve the
OVC national M&E system
 Identification of existing relevant
actions
 Identification of additional
actions
 Prioritization of actions
 Characterization of actions
(Tasks, Roles and responsibilities,
Ti
Timetable,
bl Budget
B d andd funding
f di
needs)
© IDEA International 2011
© 2008 IDEA International
Overall Process
1. Documentation review (international,
national)
2. M&E group work sessions
3. Individual stakeholder sessions
4 Field visit
4.
isit
5. Workshop(s) for diagnostic and
formulation of elements of an action
plan
6. Analysis and report writing on
diagnostic
g
and action p
plan
7. Discussion
and
validation
of
diagnostic and action plan with
stakeholders
8. Action plan implementation
© IDEA International 2011
© 2008 IDEA International
Issues based on experience so far
 Generic vs. tailor-made
 Optimal
p
degree
g of disaggregation
gg g
(number of indicators)
 Choice of scales
 Identification and involvement
of stakeholders (national/local)
 Ownership of the tool and of the
process
 Indicators on risk in turbulent
times
© IDEA International 2011
© 2008 IDEA International
Conclusion
This tool is not an end
in itself; it facilitates a
systematic, stepwise,
evidence-based
process
off
improvement of the
national M&E system.
system
Pilot test in El
Salvador, Honduras,
and Panama.
© IDEA International 2011
© 2008 IDEA International
Thank You!
© IDEA International 2011
© 2008 IDEA International