Evaluating social programmes: a key component of S&T's workIn the last newsletter we explored the M (monitoring) in M&E, and outlined the numerous projects S&T has been involved in. In this newsletter we explore the E (evaluation) in M&E and explain why we think evaluation is the cornerstone of any project.
More and more South Africans tasked with the development and delivery of
policy and programmes are being asked to plan more carefully, reflect more
critically and justify reasons for selected courses of action. This is what
evaluation is all about.
S&T, currently involved in evaluations of social programmes in health, education
and welfare, sees its role as one in which the decision makers are provided
with knowledge about their projects to ensure that the correct decisions are
made. In a context of limited resources, there is an understandable pressure
to use new knowledge wisely. We at S&T believe that well-conducted, focused
investigations, can provide a major contribution to decisions about policies
and social programmes.
What is evaluation?
Evaluations - good evaluations - are critical tools for programme and project
managers. They offer insights from a range of perspectives not readily available
to implementers. They should also be able to offer recommendations across
a wide range of issues and areas, when devised in close consultation with
participants, experts, beneficiaries and others.
Michael Scriven, a leading evaluation theorist of the 20th Century has identified
6 key questions which a sound evaluation should answer:
- What is needed?
- What are the components of this programme and how do they relate to each
- What is happening on this programme?
- How is the programme performing on a continuous basis?
- How could we improve this programme?
- How could the success of this programme be repeated elsewhere?
In answering these questions S&T utilises a methodology designed in conjunction
with our clients. A common understanding of the objectives of an evaluation
are a vital aspect of any successful evaluation. Hence, S&T ensures that there
is agreement with the client on the following aspects of the methodology before
the evaluation begins:
In answering these questions, and thereby providing an evaluation of a project
or social programme, S&T provides an evaluative judgement. Typically this "judgement"
would be in the form of key recommendations and is usually drawn up in conjunction
with the client. For more on this topic see the article below on summative versus
- What is the underlying basis for selecting criteria for judging the worth
of a project?
- What evidence will be used and on the basis of what criteria will the
judgement of worth be made?
- What standards will be applied and how will the conclusions be reached
Not surprisingly, the evaluation process can take many forms. Considerable
literature exists on evaluations and there is much debate over how best to
do an evaluation. On page two we present an example of an evaluation process,
a process that should be embedded in every social programme that aims to ameliorate
the socio-economic problems that beset our country.
Read all about it!
For more on evaluation:
Chelimsky, E.C. and Shadish, W.R. (Eds). (1997). Evaluation for the 21
Century: A Handbook. Sage: California.
Owen, J.M. & Rogers, P.J. (1999). Program Evaluation: forms and approaches.
2nd Edition. Allen & Unwin: Australia.
Shadish, W.R. Cook, T.D., & Leviton, L.C. (1991). Foundations of Program
Evaluation: Theories of Practice. Sage: California.
How does the evaluation process work?
Evaluations can take many different forms, a needs assessment would be carried
out very differently from an accreditation, which in turn would be done very
differently from a performance audit. The evaluation process is usually determined
by the needs of the client, and an agreed understanding between the client
and S&T as to what the goals of the evaluation will be. Nevertheless, it may
be useful to outline an example of an evaluation:
- Talk to client, and determine the goal of the evaluation (e.g. when should
the evaluation be done, and for what purpose).
- Identify the scope of the programme or project (might include, for example,
the extent and distribution of the target population).
- Draw up an overview of the activities of the programme (e.g. the design
of the programme and the administrative structure).
- Discover the goals, objectives/purposes/concerns of the programme.
- Conceptualise the themes and debates within the programme.
- Identify the needs and issues within and without the programme (e.g. list
the questions that should be asked in the evaluation).
- Validate, confirm or attempt to disconfirm findings (e.g. identify strengths
and weaknesses of programme).
- Observe and analyse the programme's process and outcomes (could include
description, impact assessment, measurement of KPIs etc.; might also assess
attitudes to the programme).
- Select observers/instruments for data collection (e.g. reference group,
surveys, focus groups, etc).
- Match the issues to the audiences (e.g. development of a draft report).
- Discuss the findings with the client.
- Meta-evaluation (evaluate the evaluation).
When to evaluate
Formative & summative
Evaluations take various forms. Commonly known as formative and summative,
evaluations most often happen before large-scale implementation begins to
assess whether the programme has been well-designed, costings are accurate,
conditions are favourable, outputs appropriate and so on (formative).
Alternatively, they take place to assess what happened and judge whether
goals were met (summative).
This division is often rigidly applied. The most common problem faced by
S&T evaluators - and many other professionals - is when we win evaluation
tenders only to find the programme is drawing to a close, and we have to provide
a final judgement of performance. The result: evaluators have to reconstruct
what happened, from design stage through piloting, costing and implementation,
to be able to understand why outputs may be different from those initially
expected; or why key components were changed; or new suppliers brought in;
and so on. The further result: too much time (and money) is spent trying to
catch up with implementors, often leaving too little time to assess the findings.
Judging performance is what evaluators do. However, it is extremely difficult
to do so when brought in at the end of the project or programme.
Management not judgement
Evaluations come in all shapes and sizes. Their terms of reference should
be drawn up by clients to meet their immediate and/or long-term needs regardless
of text-book divisions of 'formative', 'summative', 'process' or other types.
Evaluators should be regarded as important partners in management. Their job
is to identify problems as well as their solutions; to place these solutions
in appropriate and workable contexts; and to work with colleagues in piloting
and assessing their efficacy.
Even if evaluators are to undertake summative work, their appointment must
take place at the outset of the programme. They should be able to attend meetings
(irregularly), to be privy to policy shifts and the reasons for them, and
to meetings which assess problems and strengths in the programme. It is only
when evaluators are treated as colleagues - even where they will be delivering
an 'external' judgement - that one can be confident that flip judgements based
on an inadequate understanding will not be delivered.
Framework of self evaluation
Evaluations do not necessarily always have to be performed by outsiders.
Internal evaluations or self-evaluations should be conducted within a project
or programme on an ongoing basis. Sound strategic planning is based upon thoughtful
Self-evaluation the search for truth, and a search for self-knowledge, is
based on the perceptions of the participants. It is like looking in the mirror
and seeing our own reality. Individuals, groups, institutions, projects or
programmes may undertake self-evaluation. For each of these participants motivation,
objectives and purpose are the guiding principles of self-evaluation.
Objectives of self-evaluation involve looking and learning from experience.
To use the mirror analogy when one looks into the mirror you see what you
would like to see. This is because human perception is limited. However, the
more you look the greater the chance that you will see things previously overlooked
or new things which are worth looking at again. Through looking one may find
new experiences, experiences from which one can learn.
Each evaluation needs methods and instruments if it is to reach the target
it aims at. Methods used in self-evaluation should improve awareness and recognition,
and make discoveries and results more understandable. Methods should guide
the process and establish activities to be undertaken. Developing instruments
or a tool for self-evaluation takes place against a background of experience.
A common tool is the SWOT analysis and another is the "quality circle" approach.
It is useful to choose self-evaluation tools that can be easily handled by
participants. In the SWOT analysis, participants identify strengths, weaknesses,
obstacles and threats to the project or programme. In quality circles, all
those who are involved in the project or programme come together on a regular
basis to evaluate their activities and interactions.
The key in self-evaluation is to get those responsible for the functioning
of a project or programme to solve and analyse the problems related to their