Overview (Part I)
Posted by Teresa Dabán
The establishment of a framework for monitoring and evaluating (M&E) public spending programs and policies is an integral part of the entire process of policy-decision making and public accountability, and therefore a key element of a country’s public financial management system. This post presents a brief overview of the purposes and challenges related to the implementation of sound M&E systems of public spending programs.
The post, whose targeted audience are the “users of evaluations” of public spending programs instead of the “evaluators”, concludes that
- there is an increasing pressure on countries to step up their efforts to strengthen, or establish if they do not have one already, their government M&E systems;
- M&E systems are useful, and needed, at all of the stages of the policy-decision making process;
- the strength of a M&E system depends on the institutional arrangements underpinning its implementation; and,
- there is a large diversity of methods, techniques and analytical tools that can support the implementation of a M&E system, the choice of which will depend of a range of considerations.
What are the reasons behind current pressures on governments to develop and improve their M&E systems?
There are many reasons for the current increasing efforts that governments are making to strengthen, or establish in case they have not developed one yet, their M&E frameworks.
- In OECD countries, fiscal pressures and rising expectations and competing demands from citizens and groups in society provide a continued pressure on government to increase the quality of public spending, identify the most efficient way of making use of taxpayers’ monies and develop accountability mechanisms.
- In developing countries, civil society and parliaments are also increasingly putting accountability pressures on governments to improve and explain their performance. Strong external pressures, linked to the availability of significant financial resources, have sometimes worked as a catalyst in launching M&E systems.
- Many European Union countries, countries that have recently joined, or are candidates to join, are also under the pressure to develop a sound government M&E system to get access to EU’s Structural Funds.
- Finally, international donors have also contributed to the current impetus to strengthen government M&E systems as they are pressed to demonstrate to their constituencies the results achieved with the large volume of funds for which they are responsible.
How does M&E systems support the policy-decision making process?
The adoption of a M&E system for public spending programs can be highly useful at all of the stages of the policy-decision making process.
- At the policy formulation stage, a M&E approach can clarify and identify the key elements of alternative spending programs, such as the objectives, priorities, expected use of resources, envisaged outcomes and outputs and opportunity costs.
- M&E systems are also valuable at the policy design stage, during which they could help increase public managers’ understanding of pros and cons of alternative modalities of interventions and different organization and management arrangements.
- During the policy implementation stage, a M&E system could help strengthen the performance and effectiveness of programs by providing useful feed-backs on the program’s intermediate outcomes and outputs, detecting implementation difficulties, and in case of deviations from the originally-envisaged objectives, identifying remedial actions.
- Finally, a M&E system is also useful at the policy accountability stage for the government to demonstrate (to the Parliament or Congress, to the civil society, donors, etc) how far a program has achieved its objectives and how well has used its resources.
What are the institutional arrangements that could better help the implementation of a M&E system?
Countries approaches for the establishment of M&E systems are highly diverse.
- Most OECD have firmly institutionalized and sophisticated M&E systems whose development has been motivated by the need to improve policy-making during the 1960-70s, curb public spending under the fiscal constraints of the 1980s, and most recently, promote public sector’s legitimacy.
- In middle-income countries in Latin America, Chile has a very strong, disciplined and comprehensive M&E system. Other countries, such as Colombia, Brazil, a Mexico and Peru are also working on developing their M&E systems, although with an emphasis on the assessment of investment projects.
- Low-income countries may not have the human and technical capacity to conduct evaluations and usually have M&E systems that are principally donor-driven and designed to meet donors’ data requirements.
Nonetheless, in spite of these differences, successful M&E frameworks seem to share a set of common characteristics, including:
- a substantive use of the M&E system to support core government activities, such as budget process, national planning, management of public agencies, etc;
- strong leadership, by a powerful ministry of finance, evaluation agency, or line ministries at the sectoral level, to push for the institutionalization of the M&E system and its use on a sustained basis;
- mechanisms to ensure the objectivity, credibility, rigor of the evaluations by building human and technical capacity to implement the M&E system; and
- regular evaluations of the M&E system itself with the view to making it more effective.
What are the information requirements of a good M&E system?
Ideally, a good M&E system of public spending programs (e.g. to evaluate and monitor the construction of a highway between two cities by the ministry of public works) should be based on a comprehensive dataset. Program indicators must be as informative as possible and include data on
- the financial, human, material, organizational and regulatory resources needed for the program’s implementation (e.g. budget appropriation, skills of the staff of the ministry of public work, etc);
- the program’s output or final product (e.g. km of road built);
- the program’s immediate advantages or outcomes (e.g. a safer and faster road); (iv) the program’s impact beyond its direct product or immediate advantages (e.g. saved time for commuters); and
- performance management indicators (e.g. to measure public administration’s productivity versus alternative modalities of provision such as private-public partnerships).
A good set of indicators should also aim at measuring the program’s consequences either on direct beneficiaries, (e.g. reduction in the number of traffic accidents) and or on non-direct beneficiaries, and through market mechanisms (e.g. increase in price of adjacent real states) and non-market mechanisms (e.g. increase in pollution due to increased traffic).
What are the attributes that a sound set of indicators must comply with?
Program indicators must comply with a set of minimum conditions. They must be:
- quantifiable and/or observable without the intervention of the program’s operators,
- sensitive, i.e. with a direct relation with the program’s actions;
- reliable and credible, or in other words, the indicator must give rise to equal results if measured by two different persons;
- comparable, i.e. it should be possible to have the same indicator for alternative programs; and (iv) easy to understand, i.e. the indicator must provide a clear standard or benchmark.
In addition, program indicators must comply with certain minimum conditions as a group: the set must offer a sound coverage of the program’s key features, include a balanced combination of qualitative and quantitative indicators, and be relevant, consistent, and parsimonious.
What are the analytic tools that could be used to support the implementation of government M&E systems?
There exists a great diversity of methods, techniques and tools that could be used to support the implementation of a M&E system of public spending (see Table 1). Moreover, there is no such a thing as the “perfect analytic approach”: all methods, techniques and tools have strengths and weaknesses and in most cases they are complementary and not substitutes. The choice of a certain method, technique or tool would depend on the purpose of the analysis to be undertaken, the point in the program’ life span in which the assessment is conducted, the nature of the program, and other cost, skill, data, and time constraints. For instance, while the Logic Model technique is an inexpensive tool that can be useful to clarify the objectives of a program or policy at the formulation stage, the Cost-Benefit Analysis, which is based on more sophisticated indicators and calculations, could be a more powerful and rigorous tool to inform policy decisions and assess the economic efficiency of alternative programs, but at a much higher cost.
Table. 1 Examples of Methods, Techniques and Tools to support the implementation of M&E systems
Design of the Evaluation | Obtaining of Data | Analysis of Information | Provision of an Evaluative Judgment | |
Low-Medium Cost | e.g. Logic models, availability assessment, etc. | e.g. Statistics; interviews; case studies, focus groups, rapid appraisal methods, etc. | e.g. SWOT (Strengths, Weaknesses, and Opportunity Tool). | e.g. Impact Evaluation, benchmarking, etc. |
Medium Cost | e.g. Mapping of key concept and ideas. | e.g. Formal surveys, participatory approach, etc. | e.g. Delphi Survey. | e.g. Panel of Experts. |
Medium-High Cost | e.g. Stakeholder consultation. | e.g. Observatory techniques. | e.g. Input-Output, Econometric Models, etc. | e.g. Cost-benefit and Cost-Effectiveness Analyses. |
Source: World Bank, European Union, and the author’s elaboration.
Where one could find more information on government M&E systems and the analytic tools available for their implementation?
The bibliography on M&E systems is very extensive.
- The book of Boardman, Greenberg, Vining, and Weimer, Cost Benefit Analysis : Concepts and Practice, offers an excellent and rigorous overview.
- Another excellent, and very detailed and practical, overview can be found at the web page of the European Commission, which provides the EU member countries with guidance on how to evaluate spending programs financed with European Structural Funds.
- The publications of the World Bank’s Independent Evaluation Group (IEG) are also an excellent source of information. For instance, the recent book of Keith Mackay, How to build M&E Systems to Support Better Government, edited by the World Bank’s IEG, includes interesting and didactical material on M&E systems, available analytical tools, and country experiences, among other things. The advantages and disadvantages, costs, skills, and time needed to use alternative monitoring and evaluations tools are also discussed in the IEG’s publication Monitoring and Evaluation: Some Tools, Methods and Approaches.
- The OECD Secretariat has also published numerous surveys and analysis that review OECD’s countries practices as regards M&E systems.
