1. No category

Evaluating Impact:
From Promise to Evidence
Theory of Change
How to Get from Program to Impact?
Impact Evaluation Workshop
Debre Zeit, Ethiopia
• Christian Borja-Vega
(GWASP)
December 9 – 12, 2014
Not only ask:
“Did the program work?”
…but also ask:
“Why did the program work?”
Results Chain
Inputs
Activities
Implementation (SUPPLY SIDE)
Outputs
Intermediate
Outcomes
Outcomes
Results (DEMAND + SUPPLY)
Simplified Results Chain
Impact Evaluations
identifies these links
PROGRAM
INTERMEDIATE
OUTCOMES
Theory of Change
OUTCOMES
Theory of Change
• What is the program?
Ask the
right
questions
• What impact does this
program hope to achieve?
• How does it expect to achieve
this impact?
Sanitation Program in Azania
Providing Clean Water
Question 1: what is the program?
PROGRAM
INTERMEDIATE
OUTCOME
OUTCOME
PROGRAM
INPUTS
People, money,
supplies
ACTIVITIES
Construct
facilities
OUTPUTS
Handwashing
facilities
Question 2: what impact does this
program hope to achieve?
PROGRAM
INTERMEDIATE
OUTCOME
OUTCOME
OUTCOMES
LOWER DIARRHEA
LOWER ANEMIA
LOWER INFANT
MORTALITY
IMPROVE CHILD
DEVELOPMENT
Question 3: how does this program
expect to achieve this impact?
PROGRAM
INTERMEDIATE
OUTCOMES
OUTCOMES
INTERMEDIATE OUTCOMES
People use and benefit from program
WASH HANDS
LESS
GERMS ON
HANDS
Full Results Chain
PROGRAM
• Inputs: people,
money supplies
• Activities:
Construction
• Output:
Handwashing
facilities
INTERM.
OUTCOME
• People wash
hands
• Less germs
on hands
OUTCOME
• Lower
diarrhea and
anemia
• Lower infant
mortality
• Improved
child
development
If we only measured infant mortality and
there was no change, what does this tell us?
Program not implemented properly?
People didn’t use it properly?
Or, did not reduce diarrhea, anemia, etc.?
Bottom Line
• Ask not only what, but why.
• And, measure it!
Other Examples
School Textbooks
PROGRAM?
INTERMED
IATE
OUTCOME
OUTCOME
PROGRAM
INTERMEDIATE
OUTCOME
OUTCOME
?
Which one of the following is
an impact of the program?
A. Pupils are doing their
homework.
B. Textbooks are being
delivered to the school.
C. Pupils are performing
better at school.
D. Children attend school
more days per month.
Which one of the following is an
intermediate outcomes of the program?
A. Textbooks are ordered
at the publisher.
B. Textbooks are being
delivered to the school.
C. Pupils are using the
textbooks
D. Pupils are performing
better at school.
PROGRAM
• Activities:
Distribute
Textbooks
• Output:
Textbooks at
school
INTERMEDIATE
OUTCOME
• Teachers and pupils
use textbooks
OUTCOME
• Improved
learning
Type of
Program
Activities
Outputs
o
Education
o
o
Health
Social
Protection
and Labor
o
o
o
o
Teacher
training
Doctors
hired
Birth
attendants
trained
CCTs
delivered
Targeting
system
MIS
o
o
o
o
Teachers
trained in new
methods
Textbooks
delivered
New doctors
practicing
Attendants
applying
methods
CCTs delivered
to target
households in
accordance
with
conditions
Intermediate
Outcomes
o
o
New methods
used in
classrooms
Increased use
of textbooks
o
Increased use
of health
clinics for
deliveries
o
Increased
food
consumption
Increased
child health
visits
o
Outcomes
o
Increased test
scores
o
Improved
maternal
mortality
o
Decreased
poverty
Lower child
mortality
o
Other Examples
Results chain
–Programs tackle specific outcomes of complex issues;
– Certain programs combine multiple interventions;
–Results Chains can help identifying how and why program
components can lead, when combined, to certain outcomes
– Facilitates understanding of complex programs
Example: The Fecal Peril
Fecal contamination is the main source of diarrhea infections…
Water supply
Sanitation
Fluids
Hygiene
Fingers
Faeces
Food
Flies
Fields/Floor
s
PROGRAM?
Water Supply + Sanitation+ Hygiene
INTERMEDI
ATE
OUTCOME
OUTCOME
PROGRAM
INTERMEDIATE
OUTCOME
OUTCOME
What is the impact?
What does this program hope
to achieve?
•
•
•
Would the program improve health by delivering water quality only?
 If behavior change does not take place and sanitation infrastructure is poorly
delivered, people may still show poor health outcomes
Would the program improve health by delivering sanitation infrastructure only?
 If water quality is not delivered and behavior change does not take place, people
may still show poor health outcomes
Would the program improve health by delivering behavioral change triggering?
 If water quality and sanitation infrastructure are poorly delivered, people may still
show poor health outcomes
 Results chain may help us understand program component’s contribution
leading to the desired outcomes
Example: Combined intervention of education, health and clean water
Impact Evaluations help understanding
how and why components (individually
and collectively) contribute to the desired impacts