Evaluating social programmes: a key component of S&T's workIn the last newsletter we explored the M (monitoring) in M&E, and outlined the numerous projects S&T has been involved in. In this newsletter we explore the E (evaluation) in M&E and explain why we think evaluation is the cornerstone of any project.
More and more South Africans tasked with the development and delivery of
policy and programmes are being asked to plan more carefully, reflect more
critically and justify reasons for selected courses of action. This is what
evaluation is all about.
S&T, currently involved in evaluations of social programmes in health, education
and welfare, sees its role as one in which the decision makers are provided
with knowledge about their projects to ensure that the correct decisions are
made. In a context of limited resources, there is an understandable pressure
to use new knowledge wisely. We at S&T believe that well-conducted, focused
investigations, can provide a major contribution to decisions about policies
and social programmes.
What is evaluation?
Evaluations - good evaluations - are critical tools for programme and project
managers. They offer insights from a range of perspectives not readily available
to implementers. They should also be able to offer recommendations across
a wide range of issues and areas, when devised in close consultation with
participants, experts, beneficiaries and others.
Michael Scriven, a leading evaluation theorist of the 20th Century has identified
6 key questions which a sound evaluation should answer:
- What is needed?
- What are the components of this programme and how do they relate to each
- What is happening on this programme?
- How is the programme performing on a continuous basis?
- How could we improve this programme?
- How could the success of this programme be repeated elsewhere?
In answering these questions S&T utilises a methodology designed in conjunction
with our clients. A common understanding of the objectives of an evaluation
are a vital aspect of any successful evaluation. Hence, S&T ensures that there
is agreement with the client on the following aspects of the methodology before
the evaluation begins:
In answering these questions, and thereby providing an evaluation of a project
or social programme, S&T provides an evaluative judgement. Typically this "judgement"
would be in the form of key recommendations and is usually drawn up in conjunction
with the client. For more on this topic see the article below on summative versus
- What is the underlying basis for selecting criteria for judging the worth
of a project?
- What evidence will be used and on the basis of what criteria will the
judgement of worth be made?
- What standards will be applied and how will the conclusions be reached
Not surprisingly, the evaluation process can take many forms. Considerable
literature exists on evaluations and there is much debate over how best to
do an evaluation. On page two we present an example of an evaluation process,
a process that should be embedded in every social programme that aims to ameliorate
the socio-economic problems that beset our country.
Read all about it!
For more on evaluation:
Chelimsky, E.C. and Shadish, W.R. (Eds). (1997). Evaluation for the 21
Century: A Handbook. Sage: California.
Owen, J.M. & Rogers, P.J. (1999). Program Evaluation: forms and approaches.
2nd Edition. Allen & Unwin: Australia.
Shadish, W.R. Cook, T.D., & Leviton, L.C. (1991). Foundations of Program
Evaluation: Theories of Practice. Sage: California.
How does the evaluation process work?
Evaluations can take many different forms, a needs assessment would be carried
out very differently from an accreditation, which in turn would be done very
differently from a performance audit. The evaluation process is usually determined
by the needs of the client, and an agreed understanding between the client
and S&T as to what the goals of the evaluation will be. Nevertheless, it may
be useful to outline an example of an evaluation:
- Talk to client, and determine the goal of the evaluation (e.g. when should
the evaluation be done, and for what purpose).
- Identify the scope of the programme or project (might include, for example,
the extent and distribution of the target population).
- Draw up an overview of the activities of the programme (e.g. the design
of the programme and the administrative structure).
- Discover the goals, objectives/purposes/concerns of the programme.
- Conceptualise the themes and debates within the programme.
- Identify the needs and issues within and without the programme (e.g. list
the questions that should be asked in the evaluation).
- Validate, confirm or attempt to disconfirm findings (e.g. identify strengths
and weaknesses of programme).
- Observe and analyse the programme's process and outcomes (could include
description, impact assessment, measurement of KPIs etc.; might also assess
attitudes to the programme).
- Select observers/instruments for data collection (e.g. reference group,
surveys, focus groups, etc).
- Match the issues to the audiences (e.g. development of a draft report).
- Discuss the findings with the client.
- Meta-evaluation (evaluate the evaluation).
When to evaluate
Formative & summative
Evaluations take various forms. Commonly known as formative and summative,
evaluations most often happen before large-scale implementation begins to
assess whether the programme has been well-designed, costings are accurate,
conditions are favourable, outputs appropriate and so on (formative).
Alternatively, they take place to assess what happened and judge whether
goals were met (summative).
This division is often rigidly applied. The most common problem faced by
S&T evaluators - and many other professionals - is when we win evaluation
tenders only to find the programme is drawing to a close, and we have to provide
a final judgement of performance. The result: evaluators have to reconstruct
what happened, from design stage through piloting, costing and implementation,
to be able to understand why outputs may be different from those initially
expected; or why key components were changed; or new suppliers brought in;
and so on. The further result: too much time (and money) is spent trying to
catch up with implementors, often leaving too little time to assess the findings.
Judging performance is what evaluators do. However, it is extremely difficult
to do so when brought in at the end of the project or programme.
Management not judgement
Evaluations come in all shapes and sizes. Their terms of reference should
be drawn up by clients to meet their immediate and/or long-term needs regardless
of text-book divisions of 'formative', 'summative', 'process' or other types.
Evaluators should be regarded as important partners in management. Their job
is to identify problems as well as their solutions; to place these solutions
in appropriate and workable contexts; and to work with colleagues in piloting
and assessing their efficacy.
Even if evaluators are to undertake summative work, their appointment must
take place at the outset of the programme. They should be able to attend meetings
(irregularly), to be privy to policy shifts and the reasons for them, and
to meetings which assess problems and strengths in the programme. It is only
when evaluators are treated as colleagues - even where they will be delivering
an 'external' judgement - that one can be confident that flip judgements based
on an inadequate understanding will not be delivered.
Framework of self evaluation
Evaluations do not necessarily always have to be performed by outsiders.
Internal evaluations or self-evaluations should be conducted within a project
or programme on an ongoing basis. Sound strategic planning is based upon thoughtful
Self-evaluation the search for truth, and a search for self-knowledge, is
based on the perceptions of the participants. It is like looking in the mirror
and seeing our own reality. Individuals, groups, institutions, projects or
programmes may undertake self-evaluation. For each of these participants motivation,
objectives and purpose are the guiding principles of self-evaluation.
Objectives of self-evaluation involve looking and learning from experience.
To use the mirror analogy when one looks into the mirror you see what you
would like to see. This is because human perception is limited. However, the
more you look the greater the chance that you will see things previously overlooked
or new things which are worth looking at again. Through looking one may find
new experiences, experiences from which one can learn.
Each evaluation needs methods and instruments if it is to reach the target
it aims at. Methods used in self-evaluation should improve awareness and recognition,
and make discoveries and results more understandable. Methods should guide
the process and establish activities to be undertaken. Developing instruments
or a tool for self-evaluation takes place against a background of experience.
A common tool is the SWOT analysis and another is the "quality circle" approach.
It is useful to choose self-evaluation tools that can be easily handled by
participants. In the SWOT analysis, participants identify strengths, weaknesses,
obstacles and threats to the project or programme. In quality circles, all
those who are involved in the project or programme come together on a regular
basis to evaluate their activities and interactions.
The key in self-evaluation is to get those responsible for the functioning
of a project or programme to solve and analyse the problems related to their
Targeting to maximise impactWhat is poverty? How is it defined? Who are the poor? Where are they living? How can they be reached? These are questions frequently asked by those involved in development. The approach or means one adopts to address poverty largely depends on how one defines poverty.
Our understanding of poverty in South Africa is partly conditioned by international indices, such as the Human Development Index produced by the United Nations Development Programme, or the competitiveness index produced by the World Economic Forum. These reports use particular indicators and assess countries in their totality. What these reports often fail to reflect (because they are working at a national level) is the deep inequality that South Africa inherited from its past. South Africa is rated as a middle-income country: the abject poverty of hundreds of thousands of citizens is obscured by the wealth of others.
It is against this background that targeting has become a buzz word in development circles in South Africa. What exactly is targeting and how useful is it? These are some of the questions we focus on in this article.
The basic problem of targeting is how to maximise the gains to the poor for a given cost. Crudely put, without targeting, a programme could just do no more than spread income, which may lead to widespread leakage (i.e. the benefits may go to those not necessarily in need). The issue therefore becomes how to efficiently deliver programme benefits to the poor.
Targeting obviously requires a mechanism to distinguish the poor from the non-poor. Since there is no universal definition of poverty, identifying this mechanism becomes one of the important tasks of targeting. A second important task of targeting is to ensure more poverty reduction benefit per transferred amount of money. In the light of government's budgetary constraints, targeting is therefore a vital exercise.
There are certain conditions that dictate when one should and when one cannot or should not target.
Firstly, information on the poor or vulnerable group needs to be readily available and reliable for targeting to occur. More importantly, the cost of targeting must not outweigh the benefits. The system that needs to be put in place to administer and manage the programme should be as simple as possible.
There are a number of costs and constraints associated with targeting that need to be borne in mind.
Firstly, it is often difficult to define, and subsequently target a particular target group. Having identified the poor, targeting may miss the intended beneficiaries (i.e. exclusion errors) and/or include the non-poor (i.e. inclusion errors).
Secondly, there are administrative costs associated with targeting. Generally, the higher the degree of targeting the higher the administrative costs. A further constraint is that any targeting exercise and subsequent intervention creates losers and winners. Targeting may be unpopular if politicians' constituencies are ignored.
A final factor that needs to be recognised is that of incentive compatibility which may give rise to undesirable programme effects. For example, a food nutrition programme at a school may result in the school children getting little or no food from home as their care-givers ration scarce food resources to those members of the household that are not attending the school. In addition, there may be a stigma attached to receiving benefits which recipients may be wary of.
Implications for targeting
Questions around whether to target, how to target and which programmes to be used should result from an empirical analysis of the country or programme situation. A number of issues need to be borne in mind. It should be noted that targeting is not a panacea for resolving poverty. Secondly, appropriate targeting methods help but are constrained by factors outlined above. Therefore, the design of a programme is extremely important and must be monitored and evaluated at all steps in the process.
At the end of the day, there is no single, perfect solution to targeting. The poor are not a homogenous group and therefore perfection can never be attained in any targeted programme. However, targeting remains an important exercise given the challenges the government has to tackle and the limited resources that they have at their disposal.
Sport and the South African nationMuch has been written about the role sport can and sometimes does play in the reconstruction and development of South Africa. Indeed worldwide, much is written of the role sport plays in a divided society or one in transition.
There seems little doubt that South Africa is, on the whole, a sports-mad country. It would also appear, at a first glance, that sport in South Africa can play an important role in nation building. One had only to drive down Jan Smuts Avenue after the 1995 Rugby World Cup Final or be at the First National Bank stadium in 1996 after Bafana Bafana had won the Africa Cup of Nations, to witness the potential that sport has to overcome the racial divides that characterise much of South African society.
And yet the potential for sport in South Africa to further divide our society also exists. The recent furores around the apparent lack of transformation of two of the traditionally white sports, namely cricket and rugby, have prompted politicians, civil servants and sport administrators to engage in a series of attacks and rebuttals.
Allegations of racism and rugby have been frequent bed-partners over the last few years. The recent adoption of the Transformation Charter by the United Cricket Board further highlights the pressure on South African sport to broaden their constituencies.
Other sporting codes will undoubtedly be under pressure to adopt similar strategies and processes. From the debate that has raged on, mainly in the press, what is seemingly important is that sport in South Africa must be seen to be addressing the inequalities and inequities of the past.
What does the average citizen of this country think about sport and its role in society? A national survey of 3000 respondents, undertaken for the Kaiser Family Foundation, sought to address this question. Respondents were asked about their level of agreement with two statements:
As one would expect, the vast majority of respondents (85%) agree with the first statement. This is true across the different races where the majority were in agreement that sport is important for building unity. While Africans were more likely to strongly agree than their counterparts, at least three quarters of respondents in each race were in general agreement with the statement.
The findings around the second statement reveal differences. Two thirds of all respondents (66%) agreed with the statement. While there is still agreement among approximately three quarters of Africans, coloureds and Indians with the statement, the level of agreement amongst whites drops dramatically to 23%, with only 6% strongly agreeing with the statement.
What is also worth noting is that sport is one of the top three areas where all South Africans think that government should cut its spending in order to focus limited resources elsewhere. Sport mad South Africans may be, but there is, fortunately, enough pragmatism left in this country to see that our resources can be better spent elsewhere.
Where does this leave one in terms of the debate around the role of sport in South Africa? It would appear that sport can and should play an important part in the nation building process in South Africa. This potential for sport to aid in the broader nation building process needs to be harnessed and well directed so that it can be realised.
However, it would also seem that, given the level of agreement among the vast majority of South Africans, that serious attention needs to be paid to the composition of sporting teams and their administrations. Whether this involves legislation, quotas or coherent transformation programmes is yet another matter for debate.
Investment survey for the Office of the PresidentStrategy & Tactics has been commissioned by the Office of the President to help design, implement and analyse a survey of 1100 Small, Medium and Micro Enterprises (SMMEs) around the country. The sample is drawn from a massive data-set of established SMMEs, which itself had to be crafted from various sources. The sample will focus on three major metropolitan centres and six peri-urban towns.
The focus of the study - which has a parallel survey of large enterprises - is on factors that help and hinder investment. S&T has sub-contracted Citizen Surveys, our frequent fieldwork partner, to apply the questionnaires and code and capture the data. S&T will undertake initial analysis, focusing on attitudinal and socio-political issues; the Office of the President will manage an intensive process of analysing the data by a team of economists.