Challenges in evaluating ANTI-POVERTY PROGRAMMES
S&T partner Ross Jennings has considerable experience in evaluating small development projects and large development programmes. In this article he outlines some of the key challenges he has encountered.
Tackling poverty and measuring impact
South Africa faced massive challenges in the 1990s; tackling the poverty
in which many citizens live was among the most important. Using a basic
needs approach, the African National Congress provided a national blueprint
(The Reconstruction and Development Programme) for integrated development.
Since 1994, significant policy and regulatory reviews have been undertaken.
The RDP was superceded by the Growth, Employment and Redistribution (GEAR)
financial strategy; commentators now tell us GEAR is dead, and we wait
for the next acronym to arrive.
We have a battery of tools and terms: integrated development and sustainable
development, poverty that is chronic or transitory, verifiable and key
performance indicators, and toolkits that claim to measure everything.
How effectively and efficiently has government tackled poverty? How
useful have these tools been? This article tries to shed some light on
these questions by looking at some of the challenges of evaluating government's
What is poverty?
One of the major challenges in evaluating government's anti-poverty
programmes relates to definitional issues. A recent study of the monitoring
and evaluation practises of national departments involved in anti-poverty
programmes (commissioned by the Department of Provincial and Local Government)
found multiple and varying definitions of poverty. Different interpretations
of the "poorest of the poor" resulted in programmes targeting
different geographical areas and groups of people. What do anti-poverty
The problem of definition is further compounded by the differing objectives
and goals of anti-poverty programmes. These objectives and goals commonly
mix deliverables and concepts - capacity building, empowerment, community
participation, sustainability, high social impact, economic growth, job
creation and so on. Many are differently understood and defined across
Definitional differences:do they matter?
So what is the problem with these different definitions, in the context
of evaluation? We need to remember that evaluations are meant to measure
programmes against their objectives, identify strengths and weaknesses,
and produce recommendations that are realistic and lead to an improvement
in delivery. One could argue that there is no problem with differing definitions
as long as each individual programme clearly states its objectives at
the outset against which it is then evaluated.
What to measure?
The problem, however, manifests at two levels. At the level of each
individual programme, the absence of shared definitions has hampered the
development of measurable indicators. This, in turn, restricts the ability
of any evaluation to comprehensively measure the programme against its
The second level is to be found in government as a whole. The different
definitions across programmes do not allow for consolidation of information
and findings. It becomes methodologically problematic (if not impossible)
to evaluate delivery by government as a whole.
What to do?
Given South Africa's history, the pressure on government to deliver
(especially through its anti-poverty programmes) is considerable. Recognising
this pressure, the government is looking to develop a comprehensive system
for performance and progress monitoring across its departments, directorates
and programmes. In the poverty arena, the Integrated Sustainable Rural
Development Programme (ISRDP) has been designed to provide an integrated
response to poverty.
The ISRDP, however, is not a programme but a strategy for co-ordinating
delivery of existing programmes around priorities set at local level through
Integrated Development Plans (IDPs). It is imperative that goals and objectives
are clearly defined and measurable indicators developed for each of the
key areas within these definitions. The various anti-poverty programmes
then need to ensure that their own goals and objectives are in line with
government's overall policy thrust.
It is important to remember that integrated delivery demands integrated
evaluation. Evaluations of individual programmes remain important. However,
given that many of the programmes work in the same areas often delivering
very similar services, some form of cross-over in evaluating delivery
will limit duplication and be more cost-effective.
This cross-over will also help address the fact that programmes do not
operate in a vacuum - methodologically, evaluations of individual programmes
are unable to conclusively tie any noted effect/impact to the actual programme
Furthermore, cross-over should allow resources to be allocated to longitudinal
studies, rather than the standard approach of only conducting summative
evaluations at the end of each funding cycle. This would allow richer,
more accurate data to be gathered on issues such as empowerment, sustainability
and impact, which can only be effectively measured over time.