PEMPAL Meeting on Results-Based Monitoring and Evaluation
From March 3-6, 2014, 20 countries representing the PEMPAL Budget Community of Practice (BCOP) in cooperation with the Ministry of Finance of Turkey gathered in Antalya, Turkey for BCOP’s annual plenary meeting.2 The agenda focused on country experiences with results-based monitoring and evaluation (RBME). David Shand, an international expert in this area, acted as the facilitator for the meeting.
The meeting was formally opened by Mr. Ilhan Hatipoglu, the Director General of Budget and Fiscal Control in the Ministry of Finance of Turkey, and Mr. Martin Raiser, Director of the World Bank in Turkey.
Ms. Prodani (Chair of the BCOP Executive Committee, and General Secretary of the Ministry of Finance in Albania) provided an update on BCOP activities that took place between February 2013 and February 2014. These activities included a general meeting on program budgeting, and study visits on related topics such as IT systems for budget planning (Georgia); education financing reforms (UK); spending reviews (Ireland), program budgeting at the local level (Poland); and the role of parliament (Austria). Thirteen PEMPAL member countries participated in the OECD Budget Practices and Procedures survey.3 Several video conferences took place to consider issues related to wage bill management. The executive of BCOP also met with the executive of the other two COPs (treasury and internal audit), to start preparations for the upcoming meeting in Russia on fiscal transparency and accountability, which all three COPs will attend, later in 2014.
After Ms Prodani’s update, key concepts, terminology, and issues were presented by World Bank staff, and country case studies from Turkey, Russia, South Korea, and the Netherlands were presented by government representatives from these countries.
The meeting focused on four key issues in developing and implementing RBME:
- The importance of line ministry strategic plans;
- Using performance information in budgetary decision making;
- Developing appropriate institutional arrangements;
- Ensuring that the respective roles of the finance ministry and other government agencies are clear and complementary; and
- Moving beyond monitoring to (in depth) evaluation.
Strategic planning by line ministries covering a multi-year period is now a feature in nearly all PEMPAL countries. In several countries the cost of these strategic plans is calculated. The challenge is to prepare better strategic plans. Such plans should be developed from the “bottom up” reflecting line ministry’s knowledge of issues in both policy development and service delivery. Plans should be “owned” by the line ministry concerned, and reflect the way in which it manages program implementation. At the same time, the plans must be realistic and affordable, and linked to the “top down” priorities established in the budget and the country’s national development plan. This requires a dialogue between the finance ministry and other central agencies, on the one hand, and the line ministries, on the other hand. From the discussions at the meeting, however, it became apparent that some countries produce too many unrelated strategic documents, which makes it difficult for line ministries to articulate their own strategic plans. Governments should attempt to rationalise the number of strategic documents at the national level.
Monitoring progress of the budget through performance indicators is now general practice in PEMPAL countries. In several PEMPAL countries, a program classification of the budget has been developed, and performance indicators are part of the budget dialogue with the finance ministry. In some cases, however, performance reports are mainly used in reporting on budget implementation rather than in budget formulation, and the dialog may be more with the ministry of economy than the ministry of finance. It is also clear that problems arise in many countries both with respect to the excessive number of indicators and their mixed quality. The indicator information should be reliable, timely and cost-effective to produce. The design of indicators should limit any possible “gaming” and not have dysfunctional or perverse consequences. Indicators should be “owned” by the line ministries concerned and accepted by the finance ministry and other central agencies. “Not everything that counts can be measured and not everything that can be measured counts”. The practical limitations of performance indicators should be taken into account: they are only a tool for improved decision making, not an end in themselves.
The discussions at the meeting and the pre-meeting survey indicate that not much in-depth evaluation work is yet done in the PEMPAL countries. In some countries, however, the national audit institution has shown interest in evaluation studies as part of its work on performance auditing. The meeting spent considerable time discussing various unresolved conceptual and terminological issues. Put simply, evaluation goes beyond performance indicators by attempting to analyze the fundamental determinants of performance. Why is the delivery of primary healthcare services in a particular country worse than in its neighbouring countries? Is this attributable to the design of the services concerned, or their implementation?
The concept of “evaluation” covers a wide range of activities, and can be applied to policies, programs or individual investment projects. Evaluation work can be carried out at different levels of breadth and depth. If applied at a detailed level, evaluations may require considerable amounts of data and staffing resources. Most advanced countries therefore conduct evaluations on a selective basis, e.g., by choosing areas that are important from a policy perspective, or have high fiscal risk. Evaluation studies, however, may be politically sensitive, by highlighting areas where decisions were wrongly taken, or policies and programs poorly designed. Because of data and methodological issues (e.g., determining cause-effect relationships) evaluation results are seldom conclusive. Rather they become the basis of a dialogue to determine how performance can be best improved.
The meeting concluded that there are likely to be a number of barriers to extending the application of evaluation in PEMPAL countries, namely:
- Lack of understanding of the concept;
- Confusion over different levels and types of evaluation;
- Lack of skills and training in evaluation methodology; and
- Lack of demand for evaluation studies, particularly from the political level.
A key issue for PEMPAL countries in developing the evaluation function is how and where to start. Drawing from OECD recommendations and country experiences, member countries were advised that a possible approach might include the following:
- Establish a small evaluation unit within the ministry of finance with clear terms of reference and legal authority;
- Train staff in evaluation techniques, using the considerable amount of training material that are available;
- Select a small number of pilot projects where the return is likely to be greatest;
- Proceed cautiously and consult closely with the managers of the organization, program or project being evaluated;
- Consider developing some linkage or partnership with the National Audit Institution where it has already developed an evaluation function.
All materials and presentations from the meeting are accessible at the PEMPAL website in the official languages of the network (English, Russian and Bosnian-Croatian-Serbian) at http://www.pempal.org/event/read/105
1Respectively, Ministry of Finance, Albania, and Ministry of Finance, Turkey.
2The countries represented include Albania, Azerbaijan, Armenia, Belarus, Bosnia and Herzegovina, Bulgaria Croatia, Georgia, Kosovo, Kyrgyz Republic, Macedonia, Moldova, Montenegro, Romania, Russian Federation, Serbia, Tajikistan, Turkey, Ukraine and Uzbekistan.
3See presentation by Joung-Jin Jang of the OECD on the preliminary results at http://www.pempal.org/event/read/105
Note: The posts on the IMF PFM Blog should not be reported as representing the views of the IMF. The views expressed are those of the authors and do not necessarily represent those of the IMF or IMF policy.