The work was documented in several joint IMF-World Bank Board papers, including
These papers can be downloaded by clicking on the links above.
The HIPC AAP (Assessment and Action Plan) instrument
The indicators were developed specifically for HIPC countries, particularly on the ability of their PFM systems to track and report on the use of public funds, including funds released through HIPC debt relief. Other instruments in use by donors either attempted to audit selected transactions and then generalize to the over-all country PFM system, or followed a think-description approach to describe PFM system procedures, laws, and operations. While some effort was made to link PFM systems to outcomes at a broad level (for example, in World Bank Public Expenditure Reviews), it was still difficult to gauge over-all PFM system performance or track progress over time. Reading numerous such reports for any given country over many years, rarely would two reports mention the same PFM issues, and assessing actual progress in system operation over time was simply not possible.
The HIPC AAP approach identified 16 critical points in the PFM system that, if functioning well, would imply that a country could track and report on spending with a reasonable degree of reliability. If these critical elements were weak, there was little likelihood spending could be reported reliably. So what was new was a systems approach, identification of some critical PFM system elements, and the ability to monitor changes in these elements over time.
A summary of the 16 indicators, and the performance goal for each, is reproduced here (click on the image to enlarge), from page 7 of the 2005 joint Fund-Bank Board paper cited above. Details of the indicators, and guidance on their application, can be found in the Fund-Bank Staff Guidance.
In addition to reporting on the indicator results, the Bank-Fund work also developed with countries and collected information on country PFM action plan implementation and number of technical assistance providers actively supporting PFM reforms in country. Together this information provided a unique set of data, and some interesting conclusions.
The results through 2005
The general results, reported in the 2005 Board paper cited above, were encouraging.
- PFM system performance had improved for the assessed HIPCs from 2001-2005, though the extent of progress is mixed across countries and indicators
- Total benchmarks met increased by about 10 percent from 2001
- PFM areas identified as weakest in 2002 improved significantly by 2004
- Of the three main PFM areas assessed (budget formulation, execution and reporting), budget reporting showed the largest improvement
- Countries with high action plan implementation rates (percentage of planned actions fully implemented) generally show improvements against PFM system performance benchmarks
- The number of donor agencies involved in supporting PFM reforms in HIPCs was significant, with an average number of seven donor agencies involved per country. The presence of active donor agencies did not appear to correlate strongly with improved PFM performance.
Individual country reports from the 2004 assessment exercise are publicly available on the World Bank Public Finance website.
For individual PFM indicators, by 2005,
- 31 percent of countries provided budget data consistent with the GFS definition of general government (Indicator 1), compared with only 9 percent in 2001
- 27 percent of countries had a medium-term perspective integrated into their budget formulation process (Indicator 7), up from 18 percent in 2001
- 31 percent of countries produced timely, functionally based expenditure reports from core accounting data (Indicator 13), up from 14 percent in 2001
- 58 percent of countries closed their accounts within two months of the end of the year (Indicator 14), up substantially from 32 percent in 2001
- 27 percent of countries had final audited accounts presented within 12 months of the end of the year (Indicator 15), up from 18 percent in 2001
The results suggested that countries can make progress in improving their PFM systems, and the progress could be monitored over time.
Broader Lessons
Some of the most interesting results, while included in the paper, did not receive that much attention given the focus on over-all PFM system progress among the countries. These points can be summarized as:
- Realistic expectations of reform progress are needed. Though some countries made more progress, over-all the countries improved 10 percent over three years, or about 3 percent per year. As a general rule, rapid progress should not be the ‘benchmark,’ but steady progress. Some donor-driven reform programs still assume very significant progress (e.g. 50+ percent) over very limited time horizons (for example, a 3-year lending window).
- The "basics" of PFM systems were still not right. As the indicator results reprised above suggest, basic PFM system operations were very weak --- even as countries and assistance providers rushed to the latest reforms (e.g. accrual accounting and budgeting, performance budgeting). Absent timely reporting, comprehensive public finance information, and timely audited financial statements, these more advanced reforms can have limited impact. And focusing on advanced reforms for these countries also distracts them from getting the basics in place.
- A holistic – systems – approach to PFM system analysis and reform development is needed. Rather than looking only at one or a few elements of a PFM system, a holistic view is needed, looking at various dimensions of system performance (e.g. budget formulation, execution, reporting). Investing effort in budget formulation, when the basics of the budget (actual spending results from prior years) are not in place, or the ability to implement them, is not likely to be effective or efficient. Sometimes, critical dimensions of PFM system operation are missed entirely. The HIPC AAP instrument included dimensions of internal audit and control – areas generally neglected in PFM system assessment, but critical for effective PFM system operation. When the HIPC instrument was first applied, it was common to hear from country authorities “internal controls, no one has asked about those before.” Similarly, investing in narrow procurement modernization, when treasury cash management systems are weak, and arrears endemic, will severely limit improvements in procurement system performance. The connections between various PFM system components are understood, but infrequently emphasized or included in reform measure development.
- Country ownership of reforms is the critical variable for PFM system improvement. Country action plan implementation was the best predictor of PFM system improvements. Low rates of action implementation -- a proxy for a low degree of authority commitment to reform --- appeared related to low rates of PFM system improvement. Efforts to improve PFM systems must begin with country ownership – leadership – in assessing performance, setting the reform agenda and implementing reforms.
- How PFM system assessments, reform actions, and implementation are undertaken is at least as important as assessment instruments. Joint action planning between authorities and technical assistance providers can yield better results. One feature of the HIPC AAP process was joint development and agreement between the authorities, IMF, and World Bank staff on actions to ameliorate the PFM system performance problems identified. The fact that these action plans would be monitored directly, in addition to PFM system performance, helped keep the measures more practical, more geared to results versus fads, and may have been taken more seriously during implementation. If a further iteration of HIPC assessments would have been undertaken after 2004-5, more progress might have been seen.
- Narrowly focused action plans, reflecting clear reform priority setting, might improve reform impact and PFM system performance. Countries generally formulated action plans and pursued action implementation to improve performance across all indicators. This is worth highlighting, as many action plans included with donor or assistance provider support still appear as laundry-lists, sometime numbering in the hundreds. While there may be some ‘prioritizing’ in terms of ‘short’ or ‘medium’-term reforms, these are not prioritized in terms of risk to PFM system performance or the ability of the PFM system to deliver on government policy priorities. More work is needed on sequencing reforms, and prioritizing potential reform measures.
- Donor coordination around a more limited set of reforms might help keep reforms focused and yield better results. Despite the large number of donor agencies supporting actively PFM reforms, the low correlation suggests this support may not be yielding as much impact as it might. It is still not uncommon to visit a country and find each donor advocating a particular reform, and find those same reforms advanced across entire regions. Given the diversity in country PFM systems and their development, even within regions, it defies logic to think the same reform is the right reform for all countries.
The lessons of the HIPC work noted above, as well as lessons of PFM reform from the IMF, World Bank, and other donors, were used to develop the Strengthened Approach to PFM reform and associated PEFA (Public Expenditure and Financial Accountability) indicators. The HIPC indicators were formally no longer applied by the IMF and Bank as of the 2005 Board paper, in favor of the PEFA indicators. A future post will highlight the PEFA program, the Strengthened Approach and PEFA performance monitoring framework. For those who cannot wait, see the PEFA website.
