THE EVALUATION OF SOCIO-ECONOMIC DEVELOPMENT The GUIDE December 2003 Tavistock Institute in association with: GHK IRS CONTENTS INTRODUCTION1 PART 1 THE CONTRIBUTION OF EVALUATION TO SOCIO-ECONOMIC DEVELOPMENT5 1. 1 THE BENEFITS OF EVALUATION5 1. 2 INTRODUCING EVALUATION: HISTORY AND PURPOSE10 1. 3 METHODS AND THEIR ROOTS17 1. 4 EVALUATION TO STRENGTHEN SOCIO-ECONOMIC DEVELOPMENT24 1. 5 GOLDEN RULES35 PART 2 DESIGNING AND IMPLEMENTING EVALUATION FOR SOCIO-ECONOMIC DEVELOPMENT38 2. 1 DESIGNING AND PLANNING YOUR EVALUATION38 2. 2 IMPLEMENTING AND MANAGING EVALUATIONS56 2. 4 GOLDEN RULES81

PART 3 DEVELOPING CAPACITY FOR SOCIO-ECONOMIC EVALUATIONS84 3. 1 DEVELOPING INSTITUTIONAL CAPACITY84 3. 4 GOLDEN RULES100 PART 4 CHOOSING METHODS, TECHNIQUES AND INDICATORS AND USING EVIDENCE IN EVALUATION103 4. 1 FACTORS INFLUENCING THE CHOICE OF METHOD, TECHNIQUES, DATA AND EVIDENCE103 4. 2 METHODS AND TECHNIQUES FOR EVALUATING DIFFERENT TYPES OF SOCIO-ECONOMIC INTERVENTIONS109 4. 3 METHODS AND TECHNIQUES FOR DIFFERENT EVALUATION PURPOSES115 4. 4 METHODS AND TECHNIQUES APPLICABLE AT DIFFERENT PROGRAMMES/POLICY STAGES117 4. 5 METHODS AND TECHNIQUES APPLICABLE TO DIFFERENT STAGES IN THE EVALUATION PROCESS120 4. ACQUIRING AND USING DATA IN EVALUATION123 4. 7 CREATING INDICATORS AND INDICATOR SYSTEMS127 4. 8 USING INDICATORS TO IMPROVE MANAGEMENT140 4. 9 GOLDEN RULES146 Annexes: Annex AThe Main Stages of Evaluation Annex BChanges in Structural Fund regulations INTRODUCTION This GUIDE, is intended for those involved in the evaluation of socio-economic development in Europe. It is a successor to the MEANS collection (methods for evaluating structural policies). Whilst the GUIDE has a specific focus on evaluation within European Structural Funds, it is not confined to the evaluation of these interventions.

Socio economic development is, after all, strongly featured in many national and regional programmes that are not funded by the EU. In most countries in Europe, as elsewhere in the world, improving regional and local economies that have fallen behind, re-integrating marginalized groups, and adjusting to the challenges of global competition and technical change are priorities. Given the scarcity of resources and the often innovative strategies that socio-economic development requires, the demand for evaluation has expanded alongside the development of policies and complex interventions themselves.

Evaluating socio-economic development; policies, programmes, themes and projects The GUIDE is concerned with the evaluation of socio economic development in Europe which gives it a particular focus on European Structural Funds. The funds are organised in programmes and evaluation takes place at ex ante, mid term and ex post stages. The programmes are one means of achieving wider policy goals and programme evaluation contributes to policy evaluation. The programmes comprise many interventions and projects.

Evaluation at the level of the measure/intervention/project forms a part of programme evaluation. Many different programmes and their elements contribute to thematic objectives and thematic evaluation builds upon project and programme evaluation. The principles stressed in this GUIDE generally apply in socio economic programme, policy, project and thematic evaluation. Thus the GUIDE will be of use to those who have to plan, commission and manage thematic, policy and project evaluations as well as programme evaluation. Who is this GUIDE for?

The readers of this GUIDE will come from many of the different communities active in the evaluation of socio-economic development programmes. These will include: ? Policy makers who have an interest in what evaluation can do for them including what are the strengths and limitations of evaluation and the resources and capacities they will need, ? Public sector managers and civil servants who may commission evaluations and would like an overview of what it available including the choices of approach and methods that they should be drawing on, ?

Programme managers who will wish to incorporate evaluation results into the way they manage and plan their programmes, ? Programme partners who are increasingly involved as stakeholders in evaluations, consulted about evaluation agendas and expected to use evaluation findings, ? Evaluators, many of whom will have detailed knowledge of specific areas of evaluation but will benefit from an overview of a wider range of methods and approaches to support collaborative work with other members of an evaluation team.

Although the GUIDE itself is intended for general users and readers, rather than specialists, we have also taken account of more specialist needs by preparing a number of sourcebooks to back up the content of the GUIDE. This sourcebook material is available via the Internet at: http://www. evalsed. info. Why another evaluation GUIDE? These days we are not short of evaluation guides, textbooks and source material! As the profession and practice of evaluation has grown, a considerable library of evaluation books has been published.

Whilst this literature mainly originated from North America, the expansion of evaluation in Europe – often in response to Structural Fund requirements – has spurred many new publications in Europe. The European Commission has published detailed Methodological Guidance – on indicators, ex-ante evaluation, macro-economic evaluation etc – that is specific and closely aligned with the Structural Fund Regulations. There is also a Financial Regulation that requires ex ante and ex post evaluation of all EU funded programmes that has to be adhered to[1].

Public authorities at member state level also publish guidance for those who evaluate national and European socio-economic development programmes and policies. The obligations to evaluate and the Guidance published by those who share responsibility for the socio economic development programmes are bound to change. Evaluation needs to be closely aligned to the circumstances in which the socio economic development is taking place and the key policy choices that need to be informed. We need to be clear that this GUIDE is not a substitute for other sources and indeed it draws on and cross-refers where relevant to such sources.

This GUIDE is intended to speak to a wider audience – and to present evaluation approaches and practice in these kinds of programme and policy areas ‘in the round’. Very often other sources are very specialised, addressing narrow areas of evaluation at an expert level. This GUIDE intends to fill a gap in the market to broaden understandings of sound methods and good practice in an accessible form. Updating MEANS Of course the main source of such generic guidance up to now has been the MEANS collection – a valuable and comprehensive set of handbooks published by the European Commission in 1999.

The MEANS collection has become a standard text for European evaluators and has enjoyed a justifiable reputation for its scope, ambition and coverage. Indeed many aspects of that collection have stood the test of time and have been incorporated into this new GUIDE. However, times have also moved on since 1999. In particular: ? There have been major changes in the world of evaluation practice, with the emergence of new evaluation tools, a heightened role for theory, new participatory and qualitative approaches (especially relevant to socio economic development) and an emphasis on the institutionalisation of evaluation). European policy has moved on, especially following the Lisbon Agenda. The role of human and social capital, the importance of information society and the knowledge economy as a means of achieving greater competitiveness and priorities of sustainable development and equal opportunities have all been brought to the fore. ? The accession of ten new member states in 2004 also poses challenges for evaluation. Structural Funds are being introduced into public administrations with a relatively short experience of evaluation and consequently without a well developed evaluation culture.

The authors of this GUIDE have had in mind throughout its preparation, the capacity building needs of many new member states. In practical terms we are concerned to maximize what can be achieved pragmatically with available resources, skills, institutional arrangements and data sources. Together these changes are substantial and this GUIDE has been written to take account of their consequences. It has also been planned and written to anticipate future change. There is no reason to believe that the future pace of change of evaluation and of socio economic policy will slow down.

For this reason and to enable the ready updating of material the GUIDE, Sourcebook material and Glossary are accessible and searchable via the Internet. In future it is intended that the material will be further developed so that users will be able to structure and configure their own ‘personal’ GUIDE that matches their needs, the evaluation role that they have and the particular programmes and policies that they are evaluating or managing. Content and structure The new GUIDE is published as single volume.

It is supported by a series of Sourcebooks that provide more specialised and in depth material and which can be accessed and downloaded via the internet. The GUIDE itself is in four parts. Part 1 provides an introduction to evaluation and its benefits. This begins with a general overview of what evaluation can do to improve policies and programmes and ultimately to strengthen socio-economic development. This is followed by an introduction to some basic ideas in evaluation: its history; some of the different traditions which evaluators draw on; and, the different purposes of evaluation.

Finally, the specifics of socio-economic development as an object of evaluation are discussed. This includes unpicking the specific characteristics of this socio economic development policy and its implications for evaluation as well as the main theories and ideas on which policies and programmes are built and which evaluators need to take into account. Part 2 takes readers through practical issues in designing and implementing evaluations. It begins by considering design issues including how to plan an evaluation, defining evaluation questions and choosing methods, as well as launching and commissioning evaluation work.

It then goes on to consider the management issues once an evaluation has been designed including the choice of evaluators and the role of Steering Committees, managing communications to ensure influence and managing quality assurance in evaluation. Part 3 discusses how to develop evaluation capacity and strategies for capacity development are discussed. The argument is structures around an ‘idealised’ model that suggests four stages in capacity development. This part includes discussion of internal capacity within administrations, as well as external capacity within professional networks and partnerships.

Part 4 introduces the methods and techniques of evaluation, in terms of their strengths and weaknesses – and appropriateness. Methods and techniques are discussed within a number of frameworks: of different types of socio-economic programmes, different programme stages, different stages in the evaluation process, and different evaluation purposes. Finally, types of data (quantitative and qualitative), indicator systems and data sources are introduced. Each section of the GUIDE ends with some ‘golden rules’ highlighting both good practice and rules of thumb that can be recommended to those who manage, commission, undertake and use evaluations.

However, in general this GUIDE avoids being too prescriptive. This is partly because there is often no single right way in evaluation and different approaches each have their strengths and weaknesses in different settings. Pragmatically also, the ideal preconditions for evaluation often do not exist – whether because of lack of data, problems of timing or availability of skills. Doing the best we can whilst still trying to improve evaluation capacity in the future is a theme that runs through this GUIDE. To support the GUIDE a series of Sourcebooks has also been prepared, which will be of particular interest to specialists and practitioners.

Sourcebook 1 is entitled ‘Evaluation approaches for particular themes and policy areas’. It considers important priorities such as sustainable development, the information society, social inclusion and equal opportunities and the range of policy areas within which interventions to promote socio economic development take place. The types of evaluation methods, data and indicators that are appropriate to these themes and policy areas are elaborated and examples of their application provided. Sourcebook 2 is entitled ‘Evaluation methods and techniques’.

This includes the elaboration of a wide range of tools and techniques both quantitative and qualitative that are useful at different stages of an evaluation. Sourcebook 3 is entitled ‘Resource material on evaluation capacity building’. This includes case histories of the development of evaluation capacity in the EU, Italy, Netherlands and Ireland; and, references to other regional, national and international experience – including the accession countries. It illustrates the advice provided in the GUIDE and is intended to stimulate the development of evaluation capacity.

Finally, there is a Glossary that contains definitions of the terms used in the GUIDE and Sourcebooks. PART 1 THE CONTRIBUTION OF EVALUATION TO SOCIO-ECONOMIC DEVELOPMENT This first part of the GUIDE begins with a reminder of why evaluation is so important in socio-economic development programmes. It introduces some basic evaluation ideas, their origins and purposes. It then highlights some of the characteristics of socio economic policies and interventions and the implications of these for how they can be evaluated. |1. THE BENEFITS OF EVALUATION | | | |Providing answers to | |Evaluations that make a difference |worthwhile questions | | | | |Investing time, money and effort in evaluation has to be justified in terms of the difference it makes to | | |policy and programme success.

Evaluation is not an end in itself. In socio-economic development the policy | | |concern is to enhance the social and economic prospects of individuals, territories or sectors. Of course | | |each socio-economic programme has its own more specific rationale.

Some may emphasise regeneration of inner | | |cities, some the modernisation of obsolete or declining industrial sectors, some the integration of | | |disadvantaged groups and some the diversification of rural areas. All of these priorities and many more can | | |be found in European Structural Funds programmes. However the justification for evaluation in all these | | |cases is the same: can we apply evaluation rocedures and methods in ways that will improve the quality of | | |life, prosperity and opportunities available to citizens? To make a difference in this way requires that | | |evaluation asks and answers questions that are useful to programme stakeholders – whether they are managers,| | |policy makers or beneficiaries. | |The contribution of programme evaluation is potentially greatest in innovative policy areas where achieving |Evaluation can reduce | |success cannot be taken for granted and where implementation is not always straightforward. There is a need |uncertainty and improve | |for sophisticated management and planning. When properly applied, evaluation can help make manageable some |planning and | |of the unavoidable uncertainties of complex situations.

Socio-economic development, as will be discussed |implementation | |further in this GUIDE, is certainly complex and often faces many uncertainties: it is not a precise science. | | |Choosing goals and measures, designing programmes, implementing and sustaining a development dynamic, all | | |require analysis, anticipation, establishing feedback systems and mobilising different institutions, | | |agencies and population groups. | | | | |It is because evaluation know-how and practice has been shown to make a contribution to these processes that| | |it has become such a key component in so many socio-economic development initiatives. | |There are two important implications if we justify evaluation in these terms: | | |First, if evaluation is to be useful and usable, it needs to be seen as an integral part of decision making |Evaluation capacity and | |and management – and indeed the entire process of democratic accountability. So a well-functioning |integration into the | |evaluation system must be integrated into the policy/programme cycle.

This is why this GUIDE gives so much |policy cycle | |attention to the design of evaluation systems and the development of evaluation capacity inside public | | |agencies and within professional and knowledge networks. | | |Second, evaluators and those who commission and use evaluation findings always need to balance best |Balancing the pragmatic | |available methods with the demands of pragmatism.

In the real world of socio-economic development we rarely |with the ‘state of the | |have the time or resources – or even the data -to implement a comprehensive ‘State of the Art’ evaluation. |art’ | |This is why this GUIDE places such a strong emphasis on the kinds of strategic choices that have to be made | | |about evaluation, for example: when are greater investments in evaluation justified? Under what | | |circumstances are sophisticated methods needed?

How can evaluation fill gaps in knowledge that in a more | | |perfect world would have been covered before an intervention was even planned? | | |Improving programmes over time | | | |Different stages of a | |One important organising principle that runs through the GUIDE is the time-line of policy.

It is common to |policy and programming | |speak of the ‘policy cycle’ that begins when policy (and associated programmes) are formulated and continues|cycle | |through planning and resource allocation, programme design, implementation and the delivery of programme | | |outputs and results. Evaluation language often follows this cycle as we can see from terms such as ex ante, | | |mid-term and ex post evaluation commonly used in European Structural Funds. These terms are elaborated in | | |Annex A.

A similar logic is present in the distinction often made between ‘outputs’, ‘outcomes’, ‘results’ | | |and ‘impacts’. | | | | | |Before returning to the specific characteristics of socio-economic development, the contribution that | | |evaluation can make at each stage of the policy cycle is first described. | | |As the diagram in Box1. 1 suggests, there are three different time ‘cycles’ that are important for those | | |involved in evaluation.

First, the evaluation cycle that occurs at different ‘moments’ and at different | | |stages within a second cycle, then the programme cycle which itself generates ‘demand’ for these different | | |evaluation ‘moments’. There is also a third ‘cycle’, the policy cycle which both shapes and influences | | |programmes and inevitably also, evaluation requirements. Typically, the policy cycle is longer than the | | |programme cycle. | | | | | |Box 1. Policy, programme and evaluation cycles | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |Box 1. 1 is illustrative only – of course there are many more ‘stages’ in each cycle and they can be | | |described in different ways. But the diagram does illustrate some of the main timing problems familiar in | | |evaluation. The inner circle moves from ex ante evaluation that documents starting needs and the | | |feasibility of planned programmes, through to ongoing or mid term evaluation that documents progress and |Keeping these cycles | |implementation and finally to ex post evaluation that focuses on results.

However, ex ante evaluations |aligned | |should feed into programme design and to policy formulation, just as mid-term evaluations should help shape | | |programme implementation and policy about delivery of this and similar programmes. At the end of the | | |evaluation cycle, ex post evaluations should contribute to policy reviews. Getting these cycles to align is| | |desirable but does not always happen. Ex-ante evaluations may be undertaken too late to inform programme | | |design – let alone policy formulation. The results of ex-post evaluations may come in too late to inform | | |policy reviews.

Changes in policy and programming can also occur when an evaluation is already underway – | | |not unusual in national and European programmes of socio-economic development. This can, for example, lead | | |to changes in objectives or priorities after systems have been set up to measure results and even to the | | |close-down of certain ‘projects’ or interventions that have been the ‘objects’ of evaluation. One of the | | |advantages of involving policy makers and planners in evaluation design is to improve the alignment of all | | |of these linked activities. | |In general we are interested in this GUIDE in the evaluation of programmes rather than the evaluation of | | |policies. However these are sometimes not easy to separate. This is especially so in an age of what is | | |sometimes called evidence based policy. There is a strong imperative to use resources wisely and to learn | | |from previous experience: to base policies on evidence. This is indeed a key feature of the way ex ante | | |evaluation is understood in European Structural Funds. Evaluation can be a powerful tool for extracting |Learning for policy | |lessons learned from programmes so as to improve policies in the future. purposes | |There are a number of particular contributions that evaluation can make throughout the programme cycle: | | |Designing programmes | | | | | |One of the core competencies of evaluation is to gather information from different stakeholders or publics. | | |This is especially important at the programme design stage. Ensuring the relevance of programmes to the | | |needs of users is essential at this stage – and evaluation can contribute to this. This input on relevance |Ensuring the relevance of | |is not confined to programme design at one point in time.

In many instances of socio-economic development, |programmes | |programmes are kept on line by a continuous process of feedback from (potential and actual) users and from | | |other stakeholders. Indeed an explicit ‘re-programming’ moment is common. | | |Choosing between instruments | | | | | |A well-designed evaluation system, and particularly ex ante evaluation will also contribute to the selection| | |of specific instruments or interventions within the general scope of a programme.

Evaluation can make |Assessing the strength and| |various inputs into selecting instruments. This may take the form of an economic appraisal that assesses the|eligibility of | |likely costs and benefits of a number of alternative instruments or perhaps large projects. It may also |interventions and | |involve an assessment of eligibility that matches specific interventions with criteria to ensure relevance |instruments | |within an overall programme or regulations. Alternatively there may be an assessment of the clarity and | | |credibility of the proposed intervention to assess the likelihood of success. | |Improving management and delivery | | |A fully integrated evaluation process can make a key contribution to the way programmes are managed and | | |delivered. By analysing monitoring data and investigating underlying causes of difficulties encountered, | | |evaluation can provide feedback to programme management and support ‘mid-course correction’. Even at an | | |early stage there are usually early outputs, especially if there is a well-specified implementation chain | | |and logic-model. So evaluation of implementation brings results to the fore from the very beginning. | |However, many of the issues encountered at the early stages of implementation concern processes: how the | | |parties interact, how decisions and plans are made and how new organisational arrangements and partnerships | | |are settling down. Evaluation of such processes – even straightforward descriptions – can be helpful to all |Early ‘results’ and | |partners as well as to the main sponsoring bodies and managers. |processes | |Identifying outputs, outcomes and impacts | | | | | |From an early stage, ocio-economic development programmes need to demonstrate results. At the earliest | | |stages this is likely to take the form of ‘outputs’, eg numbers of firms taking up a subsidy for updating | | |their equipment or numbers of unemployed people receiving training. However policy makers are soon | | |interested in more substantial results: firms becoming more competitive or unemployed individuals getting |Substantial results and | |jobs. Such ‘outcomes’ are expected to have a bigger impact that relates to policy and programme goals – at |outcomes | |least in the longer term.

This allows evaluation to ask such questions as: has the growth of regional firms| | |been sustained? Or have the employment prospects of the long-term unemployed improved in sustainable ways? | | | | | |For many policy makers, identifying, describing and quantifying such outputs, outcomes and impacts is a | | |major benefit of evaluation. However, to make this process really useful policy makers need to ensure that | | |there are clear ‘objectives’ and a sensible relationship between interventions and programme goals. | |Programme managers, for their part, – if necessary working with evaluators – need to ensure that monitoring | | |and indicator systems are in place and that there are clear links between the indicators chosen and | | |underlying objectives and goals. | | |Identifying unintended consequences and ‘perverse’ effects | | | | | |Even when programmes and instruments fulfil their stated objectives these will also often be unintended | | |consequences. These can be positive or negative.

For example, support for rural entrepreneurs may have | | |spin-off benefits for urban entrepreneurs in a neighbouring city who are in the same sector or market. | | |Sometimes such unintentional consequences can have a negative effect. For example, an instrument designed | | |to improve the employment prospects of one group in the labour market may have negative consequences for | | |another group. In extremis, interventions can even have a ‘perverse’ effect: leading in a precisely | | |opposite direction to that intended.

For example, an intervention to promote tourism may, by | | |misunderstanding the basis for tourist ‘demand’ in a region, undermine the existing tourist trade, without | | |creating an additional market. | | | | | |To capture the results of socio-economic interventions including unanticipated consequences and ‘perverse’ | | |effects is essential. This is also a way in which evaluation can contribute to learning lessons – in this | | |case mainly learning interventions, how to design programmes better and how to avoid wasteful interventions | | |and ‘perverse’ effects. | | | | | |Capturing unintended | | |consequences is important | | |for learning | |Levels of evaluation: policies, themes, programmes and projects | | | | | |The problem of linking policies, programmes and specific interventions or projects is a perennial one in | | |evaluation. Many good programmes do not always add up to a good policy, good programme documents do not | | |necessarily translate into good projects and good projects do not always ensure the success of a programme. | | |However, programme evaluation is necessarily one input into policy evaluation just as project evaluation is |Linking interventions | |one input into programme evaluation. programmes and policies is| | |often a problem | |Thematic evaluations and criteria derived from policies when applied to programme material, are common ways | | |of introducing a policy level dimension into evaluation. | | | | | |There is now a tendency for evaluation to move ‘upstream’ and pay increasing attention to the policy level. | | |This reflects a willingness of policy makers to take on board evaluation results. At the same time it | | |presents challenges for evaluators who need to view the results of their work in a wider context. …especially as there is a | |Considering the policy level can also strengthen programme evaluation, for example by identifying results |tendencies to move | |oriented criteria for programme success. |evaluation upstream to the| | |policy level | |For the most part project evaluation, at least when the projects are of relatively small scale, is devolved | | |to project promoters and other intermediaries. The main exception to this is large-scale projects (e. g. | | |infrastructure projects) that have many of the characteristics of programmes in terms of their complexity as| | |well as size. | | | | |A common requirement for project managers and promoters is that they conduct some kind of self-evaluation. | | |Whilst such evaluations may lack the independence considered important for external credibility they can | | |still make an important contribution in a programme context. For example, if a well-structured framework has| | |been designed for project self-evaluation there can be some assurance that the outcomes will be systematic |Self evaluation can help | |and would merit further analysis at programme level. In addition, the requirements for self valuation can |at the intervention | |encourage a feedback and learning culture within and amongst projects that will benefit the programme as a |/project level | |whole. | | | | | |Those planning and undertaking evaluation work need to be clear of the links between policy, programme, | | |project and thematic evaluation. The principles elaborated in this GUIDE are generally applicable to each | | |type of evaluation. | | |1. INTRODUCING EVALUATION: HISTORY AND PURPOSE | | | | | |A short history of evaluation | | | | | |Evaluation emerged as a distinct area of professional practice in the post-war years in North America. Three| | |strands that were most important in that early period were the evaluation of educational innovations (eg the| | |effectiveness of new curricula in schools); linking evaluation with resource allocation (eg through a | | |Planning, Programming and Budgeting system) and the evaluation of anti-poverty programmes (eg the Great | | |Society experiments of the 1960s).

These different strands already defined some of the main evaluation | | |traditions that continue to this day and included quantitative and experimental studies using control groups| | |– the basis for many educational testing experiments; cost benefit and economic appraisal methods; and |Different strands of | |participatory and qualitative methods involving the intended beneficiaries of programmes in the evaluation |evaluation tradition | |process. | | |Underpinning these different traditions are four main groups whose interests sometimes compete with each | | |other in defining evaluation priorities.

These include: | | | | | |policy makers, eg elected officials and politicians; | | |professional and specialist interests, eg teachers in education or scientists in research; | | |managers and administrators, eg civil servants and managers of local public agencies; | | |citizens and those affected by public action, eg the presumed beneficiaries of planned interventions. | | | | | |Each of these groups makes assumptions about how evaluation can help them.

For example, policy makers tend | | |to see evaluation as a tool to ensure the accountability and justification for policy decisions; citizens | | |are more likely to regard evaluation as a tool for democratic accountability and an opportunity to shape | | |public interventions to their needs; managers and administrators are often concerned with the delivery of | | |policies and programmes – how well they are managed and organised; while professionals often regard | | |evaluation as an opportunity to improve the quality of their work or even the autonomy of their own | | |professional group. Different stakeholders | | |want evaluation to help | |This does not mean that evaluation in the broadest sense – the application of systematic social and economic|them in different ways | |research – was entirely absent from Europe or other parts of the world. However, it was probably strongest | | |in Northern Europe and in those parts of Europe, in particular, that had close links with the United States | | |and Canada. From the 1970s onwards evaluation began to take root in different European countries but often | | |with distinctive traditions and emphases. In Scandinavia for example, where there is a strong commitment to | | |democratic governance, evaluation followed in that tradition.

In France evaluation has, until recently, | | |mirrored the characteristics of the French state with a formal structured approach at a central government | | |level and a more diverse and dynamic practice at regional and local levels. However, evaluation has not been| | |static in any of these countries. For example, French evaluation practice has evolved considerably with the | | |requirements of budgetary reform after 2000. In many countries the focus and scale of evaluative activity | | |has reflected the changing policies of the different governments. For example, in the UK evaluation expanded| | |considerably with the change of government in 1997. | | | | | |There are different | | |national traditions | |European Structural Funds have been a major driver for spreading the practice of evaluation throughout the |Structural Fund approach | |EU. At every stage of the programming cycle (ex-ante, mid-term, ex-post), there are clearly stated aims and |to evaluation has evolved | |responsibilities. It is commonly acknowledged that the introduction of evaluation into many countries in |over time. |Southern Europe occurred as a result of the requirements of Structural Fund regulations. From modest | | |beginnings in 1988, there is now an elaborated Structural Fund evaluation approach. | | | | | |This approach includes: | | | | | |a legal obligation for programme sponsors and managers to evaluate; | |shared responsibility between different tiers of government for the overall evaluation process; | | |a linked multi-stage evaluation process (ex-ante, mid-term, ex-post); | | |the involvement of many partners in programmes and in their evaluation; | | |clear links between evaluation on the one hand and programming and resource allocation on the other. | | | | | |Over recent years there has been an evolution in Structural Fund regulations concerning evaluation (see Box | | |1. 2). | | | | |Some of the main transitions have been: | | | | | |from externally imposed evaluation obligations to internally driven demand for evaluation coming from |There have also been | |programme managers and policy makers themselves; |transitions in the | |from evaluation that is bolted on to programmes at the end of a programme cycle to evaluation that is fully |development of evaluation | |integrated into programmes from the beginning; |regulations | |from the expectation that evaluation results need to be disseminated largely for accountability purposes to | | |a concern for the systematic use of evaluation throughout the implementation of a programme; | | |from a view that the management of evaluation was essentially a matter of contract administration to an | | |interest in the way evaluation can contribute to knowledge management. | | | | | |These changes have been accompanied by shifts in responsibility between the different actors at European, | | |national and regional levels and with the extension of the partnership principal.

The range of potential | | |stakeholders in evaluation has therefore expanded to include, for example, local authorities, social | | |partners and civil society groups. | | | | | |The reform of the Structural Funds Regulations (see Annex B) for the third generation of programmes | | |(2000-2006) whilst devolving many obligations for evaluation to responsible authorities in Member States, | | |requires that these evaluations are used both at the ex-ante stage, and again at the mid-term. The revision | | |of the mid term evaluations are termed final evaluations.

This combination of devolved responsibilities with| | |external scrutiny by higher tiers of government is also typical of national evaluations of socio-economic | | |development. | | | | | | |Shifting responsibilities | | |for evaluation | | | | Box 1. 2 The Evolution of Structural Fund Regulations 1994 I 1999 |2000 I 2006 | |EX-ANTE EVALUATION | |An ex-ante evaluation must be carried out in partnership by the |The Member State has primary responsibility for the ex-ante | |Commission and the Member State; it must include the environmental |evaluation; | |impact; |The aim of the ex-ante evaluation is defined, special attention | |The Commission assesses the plans taking into account the ex-ante |must e given to the impact on the environment, on the labour | |evaluation |market and on equality between the sexes; | | |The ex-ante evaluation is incorporated in the plans | |MID-TERM EVALUATION | |There is no stipulation requiring information to be collected and |The Member State is responsible for the mid-term evaluation in | |communicated; |partnership with the Commission; the Commission assesses the | |There are no provisions defining an evaluative role for the |evaluation’s relevance; | |Monitoring Committee (in practice, the main role of the Monitoring |The objective of the mid-term evaluation is defined; | |Committees has been to appoint evaluators). |The mid-term evaluation is organised by the relevant programme’s| | |managing authority and is carried out by an independent | | |evaluator; it must be completed by 31 December 2003. | |To facilitate the various evaluations, the managing authority | | |must create a system for collecting the financial and | | |statistical data needed for the mid-term and ex-post | | |evaluations, and must provide the information required for the | | |ex-post evaluation; | | |The Monitoring Committee examines the findings of the mid-term | | |evaluation and may propose changes to the programme on that | | |basis; | | |An update of the mid-term evaluation is carried out by the end | | |of 2005 by way of preparing the ground for operations | | |thereafter. (The update of the Mid Term evaluation is also known| | |as the final evaluation). |EX-POST EVALUATION | |An ex-post evaluation must be carried out in partnership by the |The Commission has primary responsibility for the ex-post | |Commission and the Member State, assessing the impact of the |evaluation in partnership with the Member State; | |measures taken in terms of the intended objectives. |The objective of the ex-post evaluation is defined; | | |The ex-post evaluation is carried out by an independent | | |evaluator within three years of the end of the programming | | |period. |PERFORMANCE RESERVE | | |Before 31 March 2004, the 4% of the indicative allocation for each | | |Member State held back at the beginning of the period is allocated | | |as a ‘performance reserve’ the programming documents whose | | |performance the Commission, on the basis of proposals from the | | |Member State, considers to be successful; | | |Performance is assessed on the basis of indicators, effectiveness, | | |management and financial implementation which are defined by the | | |Member State in consultation with the Commission. | |In recent years there has also been a strong move towards public management reform and the introduction of |The link with performance | |performance management concepts in many European countries as well as in the European Commission itself (see |management | |Sound and Efficient Management – SEM 2000; and Spending more Wisely – Implementation of the Commission’s | | |Evaluation Policy). This has been taken further most recently by linking financial and budgetary decisions to| | |performance that therefore needs to be measured and described.

Performance management in the European | | |Commission (see Implementing Activity-Based Management in the Commission) as well as in Member States is now | | |reflected in a further element within the evaluation obligations of the current regulations, those that | | |concern the Performance Reserve. Indicators of effectiveness management and financial implementation have to | | |be gathered in order to justify the release of funds held back at the beginning of the programming period. | | |Different traditions and sources | | |Evaluation has varied roots: it is not a unified practice, or derived from a single set of traditions. This | | |is in part the result of the historical evolution of evaluation, both in Europe and in North America.

As | | |already noted, it is common to highlight three important sources of evaluation thinking. The 1960s Great | | |Society initiatives in the United States; educational innovation and in particular curriculum innovation in |Many roots and different | |schools; and budgetary control and efficiency systems such as Planning, Programming and Budgetary Systems |origins | |(PPBS). In reality these are only three particular sources and one could add management by objectives, | | |participative research in community and rural development, results based management and many more. | | | | |One way of distinguishing some of these different origins is to stand back from particular examples and look | | |at the core ideas or theories that lie behind these different evaluation traditions. | | | | | |We can distinguish between four main sets of ideas: | | | | | |Scientific research and methods. Many of the basic ideas and methods used in evaluation are shared with the | | |wider research community especially in the social sciences and economics.

Within the logic that combines | | |hypotheses testing, observation, data collection and data analysis, explanations are sought for what is |Four sources of ideas and | |observed. In complex socio-economic programmes explanations are rarely straightforward. Much of the work of |theory | |evaluators is an attempt to attribute observed outcomes with known inputs – and vice versa. | | | | | |Economic theory and public choices. Economic thinking is present within evaluation at several different | | |levels.

These include notions of efficiency and resource allocation in the face of scarcity; institutional | | |(mainly micro-economic) incentives and behaviours; and macro-economic studies that seek to identify aggregate| | |effects (e. g. in terms of GDP or competitiveness) of policy interventions. | | | | | |Organisation and management theory. This has begun to feature more prominently in evaluation in recent years | | |as the focus has shifted increasingly to implementation and delivery of programmes and policies. This body of| | |thinking highlights issues of organisational design, inter-organisational co-ordination (e. g. hrough | | |partnerships and consortia), and issues of motivation, ownership and participation. | | | | | |Political and administrative sciences. As public programmes and their managers address issues of the policy | | |process and public sector reform they have increasingly drawn on ideas concerned with governance, | | |accountability and citizenship. Many of the core ideas in public sector reform and the new public management’| | |such as transparency and accountability have been influenced by these perspectives.

In addition in | | |contemporary political perspectives highlights the importance of consensus building in order to strengthen | | |legitimacy of policy action. | | |It follows from the above that evaluators are similarly diverse. They may be economists concerned with |Bridging diversity within | |efficiency and costs; or management consultants interested in the smooth running of the organisation; policy |an evaluation team | |analysts with a commitment to public sector reform and transparency; or scientists (of various disciplines) | | |concerned to establish truth, generate new knowledge and confirm/disconfirm hypotheses. | | | | |One of the biggest problems that those who manage or commission evaluation face is how to put together a | | |suitable ‘team’ or mix of competencies that may properly come from all these traditions (this is taken | | |further in Part 2 of the GUIDE when we discuss the ‘profile’ of the evaluation team and choosing the right | | |evaluators, page 55). | | | | | |At a systemic level (eg nationally or in Europe as a whole) one of the key tasks of evaluation ‘capacity | | |building’ is to build bridges between these different parts of the professional evaluation communities. | |Conferences, networks and professional societies that bring evaluators together are a way of increasing | | |familiarity between those who come from different traditions as well as a way of transferring and sharing | | |know-how, knowledge and expertise (this is also taken further in Part 2 of the GUIDE). | | |Despite these differences in evaluation origins and traditions it is possible to distinguish some of the main| | |types of evaluation. These tend to cohere around two main axes. The first axis is about evaluation purposes | | |and the second concerns evaluation methodologies. | |The main purposes of evaluation | | |Evaluations always serve a broader purpose, which is to make a particular contribution to an area of public | | |policy and its programmes. The most commonly recognised purposes of evaluation are: | | | | | |Planning/efficiency – ensuring that there is a justification for a policy/programme and that resources are | | |efficiently deployed. | | |Five different purposes | |Accountability – demonstrating how far a programme has achieved its objectives and how well it has used its | | |resources. | | | | | |Implementation – improving the performance of programmes and the effectiveness of how they are delivered and | | |managed. | | | | | |Knowledge production – increasing our understanding of what works in what circumstances and how different | | |measures and interventions can be made more effective. | | | | |Institutional strengthening – improving and developing capacity among programme participants and their | | |networks and institutions. | | |These various evaluation purposes are of interest to different stakeholders and also tend to be associated |Different purposes | |with different kinds of evaluation questions. For example: |interest different | | |stakeholders | |If the purpose is planning/efficiency, it will mainly meet the needs of planners and policy makers – as well |