home
the team
newsletters
about
contact
stnews
links
 
Strategy & Tactics logo    
 

September 1999

Evaluating social programmes: a key component of S&T's work

In the last newsletter we explored the M (monitoring) in M&E, and outlined the numerous projects S&T has been involved in. In this newsletter we explore the E (evaluation) in M&E and explain why we think evaluation is the cornerstone of any project.

More and more South Africans tasked with the development and delivery of policy and programmes are being asked to plan more carefully, reflect more critically and justify reasons for selected courses of action. This is what evaluation is all about.

S&T, currently involved in evaluations of social programmes in health, education and welfare, sees its role as one in which the decision makers are provided with knowledge about their projects to ensure that the correct decisions are made. In a context of limited resources, there is an understandable pressure to use new knowledge wisely. We at S&T believe that well-conducted, focused investigations, can provide a major contribution to decisions about policies and social programmes.

What is evaluation?

Evaluations - good evaluations - are critical tools for programme and project managers. They offer insights from a range of perspectives not readily available to implementers. They should also be able to offer recommendations across a wide range of issues and areas, when devised in close consultation with participants, experts, beneficiaries and others.

Michael Scriven, a leading evaluation theorist of the 20th Century has identified 6 key questions which a sound evaluation should answer:

  • What is needed?
  • What are the components of this programme and how do they relate to each other?
  • What is happening on this programme?
  • How is the programme performing on a continuous basis?
  • How could we improve this programme?
  • How could the success of this programme be repeated elsewhere?

In answering these questions S&T utilises a methodology designed in conjunction with our clients. A common understanding of the objectives of an evaluation are a vital aspect of any successful evaluation. Hence, S&T ensures that there is agreement with the client on the following aspects of the methodology before the evaluation begins:

  • What is the underlying basis for selecting criteria for judging the worth of a project?
  • What evidence will be used and on the basis of what criteria will the judgement of worth be made?
  • What standards will be applied and how will the conclusions be reached and presented?
In answering these questions, and thereby providing an evaluation of a project or social programme, S&T provides an evaluative judgement. Typically this "judgement" would be in the form of key recommendations and is usually drawn up in conjunction with the client. For more on this topic see the article below on summative versus formative evaluation.

Not surprisingly, the evaluation process can take many forms. Considerable literature exists on evaluations and there is much debate over how best to do an evaluation. On page two we present an example of an evaluation process, a process that should be embedded in every social programme that aims to ameliorate the socio-economic problems that beset our country.

Read all about it!

For more on evaluation:

Chelimsky, E.C. and Shadish, W.R. (Eds). (1997). Evaluation for the 21 Century: A Handbook. Sage: California.

Owen, J.M. & Rogers, P.J. (1999). Program Evaluation: forms and approaches. 2nd Edition. Allen & Unwin: Australia.

Shadish, W.R. Cook, T.D., & Leviton, L.C. (1991). Foundations of Program Evaluation: Theories of Practice. Sage: California.

How does the evaluation process work?

Evaluations can take many different forms, a needs assessment would be carried out very differently from an accreditation, which in turn would be done very differently from a performance audit. The evaluation process is usually determined by the needs of the client, and an agreed understanding between the client and S&T as to what the goals of the evaluation will be. Nevertheless, it may be useful to outline an example of an evaluation:

  1. Talk to client, and determine the goal of the evaluation (e.g. when should the evaluation be done, and for what purpose).
  2. Identify the scope of the programme or project (might include, for example, the extent and distribution of the target population).
  3. Draw up an overview of the activities of the programme (e.g. the design of the programme and the administrative structure).
  4. Discover the goals, objectives/purposes/concerns of the programme.
  5. Conceptualise the themes and debates within the programme.
  6. Identify the needs and issues within and without the programme (e.g. list the questions that should be asked in the evaluation).
  7. Validate, confirm or attempt to disconfirm findings (e.g. identify strengths and weaknesses of programme).
  8. Observe and analyse the programme's process and outcomes (could include description, impact assessment, measurement of KPIs etc.; might also assess attitudes to the programme).
  9. Select observers/instruments for data collection (e.g. reference group, surveys, focus groups, etc).
  10. Match the issues to the audiences (e.g. development of a draft report).
  11. Discuss the findings with the client.
  12. Meta-evaluation (evaluate the evaluation).

When to evaluate

Formative & summative

Evaluations take various forms. Commonly known as formative and summative, evaluations most often happen before large-scale implementation begins to assess whether the programme has been well-designed, costings are accurate, conditions are favourable, outputs appropriate and so on (formative).

Alternatively, they take place to assess what happened and judge whether goals were met (summative).

This division is often rigidly applied. The most common problem faced by S&T evaluators - and many other professionals - is when we win evaluation tenders only to find the programme is drawing to a close, and we have to provide a final judgement of performance. The result: evaluators have to reconstruct what happened, from design stage through piloting, costing and implementation, to be able to understand why outputs may be different from those initially expected; or why key components were changed; or new suppliers brought in; and so on. The further result: too much time (and money) is spent trying to catch up with implementors, often leaving too little time to assess the findings. Judging performance is what evaluators do. However, it is extremely difficult to do so when brought in at the end of the project or programme.

Management not judgement

Evaluations come in all shapes and sizes. Their terms of reference should be drawn up by clients to meet their immediate and/or long-term needs regardless of text-book divisions of 'formative', 'summative', 'process' or other types. Evaluators should be regarded as important partners in management. Their job is to identify problems as well as their solutions; to place these solutions in appropriate and workable contexts; and to work with colleagues in piloting and assessing their efficacy.

Even if evaluators are to undertake summative work, their appointment must take place at the outset of the programme. They should be able to attend meetings (irregularly), to be privy to policy shifts and the reasons for them, and to meetings which assess problems and strengths in the programme. It is only when evaluators are treated as colleagues - even where they will be delivering an 'external' judgement - that one can be confident that flip judgements based on an inadequate understanding will not be delivered.

Framework of self evaluation

Evaluations do not necessarily always have to be performed by outsiders. Internal evaluations or self-evaluations should be conducted within a project or programme on an ongoing basis. Sound strategic planning is based upon thoughtful self-evaluation.

Self-evaluation the search for truth, and a search for self-knowledge, is based on the perceptions of the participants. It is like looking in the mirror and seeing our own reality. Individuals, groups, institutions, projects or programmes may undertake self-evaluation. For each of these participants motivation, objectives and purpose are the guiding principles of self-evaluation.

Objectives of self-evaluation involve looking and learning from experience. To use the mirror analogy when one looks into the mirror you see what you would like to see. This is because human perception is limited. However, the more you look the greater the chance that you will see things previously overlooked or new things which are worth looking at again. Through looking one may find new experiences, experiences from which one can learn.

Each evaluation needs methods and instruments if it is to reach the target it aims at. Methods used in self-evaluation should improve awareness and recognition, and make discoveries and results more understandable. Methods should guide the process and establish activities to be undertaken. Developing instruments or a tool for self-evaluation takes place against a background of experience. A common tool is the SWOT analysis and another is the "quality circle" approach. It is useful to choose self-evaluation tools that can be easily handled by participants. In the SWOT analysis, participants identify strengths, weaknesses, obstacles and threats to the project or programme. In quality circles, all those who are involved in the project or programme come together on a regular basis to evaluate their activities and interactions.

The key in self-evaluation is to get those responsible for the functioning of a project or programme to solve and analyse the problems related to their own work.

 

Targeting to maximise impact

What is poverty? How is it defined? Who are the poor? Where are they living? How can they be reached? These are questions frequently asked by those involved in development. The approach or means one adopts to address poverty largely depends on how one defines poverty.

Our understanding of poverty in South Africa is partly conditioned by international indices, such as the Human Development Index produced by the United Nations Development Programme, or the competitiveness index produced by the World Economic Forum. These reports use particular indicators and assess countries in their totality. What these reports often fail to reflect (because they are working at a national level) is the deep inequality that South Africa inherited from its past. South Africa is rated as a middle-income country: the abject poverty of hundreds of thousands of citizens is obscured by the wealth of others.

It is against this background that targeting has become a buzz word in development circles in South Africa. What exactly is targeting and how useful is it? These are some of the questions we focus on in this article.

Targeting

The basic problem of targeting is how to maximise the gains to the poor for a given cost. Crudely put, without targeting, a programme could just do no more than spread income, which may lead to widespread leakage (i.e. the benefits may go to those not necessarily in need). The issue therefore becomes how to efficiently deliver programme benefits to the poor.

Targeting obviously requires a mechanism to distinguish the poor from the non-poor. Since there is no universal definition of poverty, identifying this mechanism becomes one of the important tasks of targeting. A second important task of targeting is to ensure more poverty reduction benefit per transferred amount of money. In the light of government's budgetary constraints, targeting is therefore a vital exercise.

There are certain conditions that dictate when one should and when one cannot or should not target.

Firstly, information on the poor or vulnerable group needs to be readily available and reliable for targeting to occur. More importantly, the cost of targeting must not outweigh the benefits. The system that needs to be put in place to administer and manage the programme should be as simple as possible.

There are a number of costs and constraints associated with targeting that need to be borne in mind.

Firstly, it is often difficult to define, and subsequently target a particular target group. Having identified the poor, targeting may miss the intended beneficiaries (i.e. exclusion errors) and/or include the non-poor (i.e. inclusion errors).

Secondly, there are administrative costs associated with targeting. Generally, the higher the degree of targeting the higher the administrative costs. A further constraint is that any targeting exercise and subsequent intervention creates losers and winners. Targeting may be unpopular if politicians' constituencies are ignored.

A final factor that needs to be recognised is that of incentive compatibility which may give rise to undesirable programme effects. For example, a food nutrition programme at a school may result in the school children getting little or no food from home as their care-givers ration scarce food resources to those members of the household that are not attending the school. In addition, there may be a stigma attached to receiving benefits which recipients may be wary of.

Implications for targeting

Questions around whether to target, how to target and which programmes to be used should result from an empirical analysis of the country or programme situation. A number of issues need to be borne in mind. It should be noted that targeting is not a panacea for resolving poverty. Secondly, appropriate targeting methods help but are constrained by factors outlined above. Therefore, the design of a programme is extremely important and must be monitored and evaluated at all steps in the process.

At the end of the day, there is no single, perfect solution to targeting. The poor are not a homogenous group and therefore perfection can never be attained in any targeted programme. However, targeting remains an important exercise given the challenges the government has to tackle and the limited resources that they have at their disposal.

 

Sport and the South African nation

Much has been written about the role sport can and sometimes does play in the reconstruction and development of South Africa. Indeed worldwide, much is written of the role sport plays in a divided society or one in transition.

There seems little doubt that South Africa is, on the whole, a sports-mad country. It would also appear, at a first glance, that sport in South Africa can play an important role in nation building. One had only to drive down Jan Smuts Avenue after the 1995 Rugby World Cup Final or be at the First National Bank stadium in 1996 after Bafana Bafana had won the Africa Cup of Nations, to witness the potential that sport has to overcome the racial divides that characterise much of South African society.

And yet the potential for sport in South Africa to further divide our society also exists. The recent furores around the apparent lack of transformation of two of the traditionally white sports, namely cricket and rugby, have prompted politicians, civil servants and sport administrators to engage in a series of attacks and rebuttals.

Allegations of racism and rugby have been frequent bed-partners over the last few years. The recent adoption of the Transformation Charter by the United Cricket Board further highlights the pressure on South African sport to broaden their constituencies.

Other sporting codes will undoubtedly be under pressure to adopt similar strategies and processes. From the debate that has raged on, mainly in the press, what is seemingly important is that sport in South Africa must be seen to be addressing the inequalities and inequities of the past.

What does the average citizen of this country think about sport and its role in society? A national survey of 3000 respondents, undertaken for the Kaiser Family Foundation, sought to address this question. Respondents were asked about their level of agreement with two statements:

  • Sport is important for building unity in South Africa; and
  • All sporting teams should have to reflect the racial make-up of South Africa.

As one would expect, the vast majority of respondents (85%) agree with the first statement. This is true across the different races where the majority were in agreement that sport is important for building unity. While Africans were more likely to strongly agree than their counterparts, at least three quarters of respondents in each race were in general agreement with the statement.

The findings around the second statement reveal differences. Two thirds of all respondents (66%) agreed with the statement. While there is still agreement among approximately three quarters of Africans, coloureds and Indians with the statement, the level of agreement amongst whites drops dramatically to 23%, with only 6% strongly agreeing with the statement.

What is also worth noting is that sport is one of the top three areas where all South Africans think that government should cut its spending in order to focus limited resources elsewhere. Sport mad South Africans may be, but there is, fortunately, enough pragmatism left in this country to see that our resources can be better spent elsewhere.

Where does this leave one in terms of the debate around the role of sport in South Africa? It would appear that sport can and should play an important part in the nation building process in South Africa. This potential for sport to aid in the broader nation building process needs to be harnessed and well directed so that it can be realised.

However, it would also seem that, given the level of agreement among the vast majority of South Africans, that serious attention needs to be paid to the composition of sporting teams and their administrations. Whether this involves legislation, quotas or coherent transformation programmes is yet another matter for debate.

 

Investment survey for the Office of the President

Strategy & Tactics has been commissioned by the Office of the President to help design, implement and analyse a survey of 1100 Small, Medium and Micro Enterprises (SMMEs) around the country. The sample is drawn from a massive data-set of established SMMEs, which itself had to be crafted from various sources. The sample will focus on three major metropolitan centres and six peri-urban towns.

The focus of the study - which has a parallel survey of large enterprises - is on factors that help and hinder investment. S&T has sub-contracted Citizen Surveys, our frequent fieldwork partner, to apply the questionnaires and code and capture the data. S&T will undertake initial analysis, focusing on attitudinal and socio-political issues; the Office of the President will manage an intensive process of analysing the data by a team of economists.

 

[top] [to newsletters] [Previous page]
 
     
home
the team
newsletters
about
contact
stnews
downloads
links