home
the team
newsletters
about
cape town
stnews
downloads
links
 
   

This article is taken from the September 1999 Phatlalatsa newsletter

 

Evaluating social programmes: a key component of S&T's work

In the last newsletter we explored the M (monitoring) in M&E, and outlined the numerous projects S&T has been involved in. In this newsletter we explore the E (evaluation) in M&E and explain why we think evaluation is the cornerstone of any project.

More and more South Africans tasked with the development and delivery of policy and programmes are being asked to plan more carefully, reflect more critically and justify reasons for selected courses of action. This is what evaluation is all about.

S&T, currently involved in evaluations of social programmes in health, education and welfare, sees its role as one in which the decision makers are provided with knowledge about their projects to ensure that the correct decisions are made. In a context of limited resources, there is an understandable pressure to use new knowledge wisely. We at S&T believe that well-conducted, focused investigations, can provide a major contribution to decisions about policies and social programmes.

What is evaluation?

Evaluations - good evaluations - are critical tools for programme and project managers. They offer insights from a range of perspectives not readily available to implementers. They should also be able to offer recommendations across a wide range of issues and areas, when devised in close consultation with participants, experts, beneficiaries and others.

Michael Scriven, a leading evaluation theorist of the 20th Century has identified 6 key questions which a sound evaluation should answer:

  • What is needed?
  • What are the components of this programme and how do they relate to each other?
  • What is happening on this programme?
  • How is the programme performing on a continuous basis?
  • How could we improve this programme?
  • How could the success of this programme be repeated elsewhere?

In answering these questions S&T utilises a methodology designed in conjunction with our clients. A common understanding of the objectives of an evaluation are a vital aspect of any successful evaluation. Hence, S&T ensures that there is agreement with the client on the following aspects of the methodology before the evaluation begins:

  • What is the underlying basis for selecting criteria for judging the worth of a project?
  • What evidence will be used and on the basis of what criteria will the judgement of worth be made?
  • What standards will be applied and how will the conclusions be reached and presented?
In answering these questions, and thereby providing an evaluation of a project or social programme, S&T provides an evaluative judgement. Typically this "judgement" would be in the form of key recommendations and is usually drawn up in conjunction with the client. For more on this topic see the article below on summative versus formative evaluation.

Not surprisingly, the evaluation process can take many forms. Considerable literature exists on evaluations and there is much debate over how best to do an evaluation. On page two we present an example of an evaluation process, a process that should be embedded in every social programme that aims to ameliorate the socio-economic problems that beset our country.

Read all about it!

For more on evaluation:

Chelimsky, E.C. and Shadish, W.R. (Eds). (1997). Evaluation for the 21 Century: A Handbook. Sage: California.

Owen, J.M. & Rogers, P.J. (1999). Program Evaluation: forms and approaches. 2nd Edition. Allen & Unwin: Australia.

Shadish, W.R. Cook, T.D., & Leviton, L.C. (1991). Foundations of Program Evaluation: Theories of Practice. Sage: California.

How does the evaluation process work?

Evaluations can take many different forms, a needs assessment would be carried out very differently from an accreditation, which in turn would be done very differently from a performance audit. The evaluation process is usually determined by the needs of the client, and an agreed understanding between the client and S&T as to what the goals of the evaluation will be. Nevertheless, it may be useful to outline an example of an evaluation:

  1. Talk to client, and determine the goal of the evaluation (e.g. when should the evaluation be done, and for what purpose).
  2. Identify the scope of the programme or project (might include, for example, the extent and distribution of the target population).
  3. Draw up an overview of the activities of the programme (e.g. the design of the programme and the administrative structure).
  4. Discover the goals, objectives/purposes/concerns of the programme.
  5. Conceptualise the themes and debates within the programme.
  6. Identify the needs and issues within and without the programme (e.g. list the questions that should be asked in the evaluation).
  7. Validate, confirm or attempt to disconfirm findings (e.g. identify strengths and weaknesses of programme).
  8. Observe and analyse the programme's process and outcomes (could include description, impact assessment, measurement of KPIs etc.; might also assess attitudes to the programme).
  9. Select observers/instruments for data collection (e.g. reference group, surveys, focus groups, etc).
  10. Match the issues to the audiences (e.g. development of a draft report).
  11. Discuss the findings with the client.
  12. Meta-evaluation (evaluate the evaluation).

When to evaluate

Formative & summative

Evaluations take various forms. Commonly known as formative and summative, evaluations most often happen before large-scale implementation begins to assess whether the programme has been well-designed, costings are accurate, conditions are favourable, outputs appropriate and so on (formative).

Alternatively, they take place to assess what happened and judge whether goals were met (summative).

This division is often rigidly applied. The most common problem faced by S&T evaluators - and many other professionals - is when we win evaluation tenders only to find the programme is drawing to a close, and we have to provide a final judgement of performance. The result: evaluators have to reconstruct what happened, from design stage through piloting, costing and implementation, to be able to understand why outputs may be different from those initially expected; or why key components were changed; or new suppliers brought in; and so on. The further result: too much time (and money) is spent trying to catch up with implementors, often leaving too little time to assess the findings. Judging performance is what evaluators do. However, it is extremely difficult to do so when brought in at the end of the project or programme.

Management not judgement

Evaluations come in all shapes and sizes. Their terms of reference should be drawn up by clients to meet their immediate and/or long-term needs regardless of text-book divisions of 'formative', 'summative', 'process' or other types. Evaluators should be regarded as important partners in management. Their job is to identify problems as well as their solutions; to place these solutions in appropriate and workable contexts; and to work with colleagues in piloting and assessing their efficacy.

Even if evaluators are to undertake summative work, their appointment must take place at the outset of the programme. They should be able to attend meetings (irregularly), to be privy to policy shifts and the reasons for them, and to meetings which assess problems and strengths in the programme. It is only when evaluators are treated as colleagues - even where they will be delivering an 'external' judgement - that one can be confident that flip judgements based on an inadequate understanding will not be delivered.

Framework of self evaluation

Evaluations do not necessarily always have to be performed by outsiders. Internal evaluations or self-evaluations should be conducted within a project or programme on an ongoing basis. Sound strategic planning is based upon thoughtful self-evaluation.

Self-evaluation the search for truth, and a search for self-knowledge, is based on the perceptions of the participants. It is like looking in the mirror and seeing our own reality. Individuals, groups, institutions, projects or programmes may undertake self-evaluation. For each of these participants motivation, objectives and purpose are the guiding principles of self-evaluation.

Objectives of self-evaluation involve looking and learning from experience. To use the mirror analogy when one looks into the mirror you see what you would like to see. This is because human perception is limited. However, the more you look the greater the chance that you will see things previously overlooked or new things which are worth looking at again. Through looking one may find new experiences, experiences from which one can learn.

Each evaluation needs methods and instruments if it is to reach the target it aims at. Methods used in self-evaluation should improve awareness and recognition, and make discoveries and results more understandable. Methods should guide the process and establish activities to be undertaken. Developing instruments or a tool for self-evaluation takes place against a background of experience. A common tool is the SWOT analysis and another is the "quality circle" approach. It is useful to choose self-evaluation tools that can be easily handled by participants. In the SWOT analysis, participants identify strengths, weaknesses, obstacles and threats to the project or programme. In quality circles, all those who are involved in the project or programme come together on a regular basis to evaluate their activities and interactions.

The key in self-evaluation is to get those responsible for the functioning of a project or programme to solve and analyse the problems related to their own work.

 

[top] [to ] [Previous page]
     
home
the team
newsletters
about
contact
stnews
downloads
links