« Structural Breaks in Fiscal Performance: Did Fiscal Responsibility Laws Have Anything to do With Them? | Main | FAD is Seeking Public Financial Management Advisors »

January 17, 2011

PFM Performance Measurement Framework -- Revision of Three Indicators

Posted by the PEFA Secretariat

1.  Background and process

The PEFA program has made adjustments to three indicators where experience over the five years since the PFM Performance Measurement Framework was launched has shown that ratings do not always correctly reflect the level of system performance. In June 2010, the PEFA Steering Committee sought comments from any interested party on proposed revisions to:

• PI-2 Composition of expenditure out-turn compared to original approved budget;
• PI-3 Aggregate revenue out-turn compared to original approved budget;
• PI-19 Transparency, competition and complaints mechanisms in procurement;

Comments were received from the PEFA partners, other donor organizations, partner governments, consultants and individuals. Some of the twenty specific comments made more than one point, but none rejected the proposals (and several comments were really discussions emphasizing a particular aspect).

The proposed revisions were tested against data from earlier assessments during the design phase. This was straightforward for PIs 2 and 3, but more difficult for PI-19 (as the new rating criteria specify evidence that was not required in earlier assessments): however, the revised indicator produces results that are compatible with those obtained from the OECD DAC ‘Methodology for Assessing Procurement Systems’ tool, as had been intended.

The PEFA Steering Committee has approved final versions of three revised indicators for use. The Steering Committee wishes to bring these revised indicators into use as soon as is practicable, which may mean that some assessments currently underway are able to use them. However, all Concept Notes/Terms of Reference for proposed assessments issued after 28 February 2011 should refer to use of the revised indicators.

2. The revised indicators

PI-2. Composition of expenditure out-turn compared to original approved budget

Reasons for revision
In situations where all changes against the original budget are negative (for example, when there is a major cut, arising from a significant revenue shortfall) or positive (perhaps because of windfall revenues collected mid-year), the current methodology results in an ‘A’ rating, even when the changes are unevenly spread. In addition, the accounting treatment of contingencies can have major implications for the rating, depending on the size of the contingency and whether it is transferred (vired) to spending entities or spent/accounted for directly under the contingency head: again – assuming no other variance – PI-2 will give an ‘A’ rating to a government that has a large contingency but accounts for its use directly under the contingency head, while a government which vires the contingency to spending entities will be rated lower (in other words, the scoring criterion penalizes what is generally accepted as good practice).

Basis of the change
To remedy these problems, the current basis for calculating PI-2 has been changed to reflect the good budgetary practice of according equal marginal value to all budget lines, and a second dimension added to focus on contingencies. Dimension (i) will improve the rating of any variance from the original budget appropriations by using relative deviations from an across-the-board adjustment to the budget, to reflect the aggregate actual expenditure. However, to avoid ‘double counting’ the impact of contingencies, they are excluded from this calculation of variances. A new dimension (ii) has been calibrated to avoid penalizing ‘good practice’ by allocating an ‘A’ to a government that records little or no actual expenditure against the contingency vote (because either the contingency has not been used or it has been vired to those spending departments where actual expenditure is incurred and recorded).

PI-3. Aggregate revenue out-turn compared to original approved budget

Reason for revision
The original PI-3 rates the percentage shortfall between the forecast and the actual revenue achieved, but did not consider under-budgeting of revenue. Pessimistic revenue forecasts often result in excess revenue being used for spending that has not been subjected to the scrutiny of the budget process, while optimistic forecasts can lead to unjustifiably large expenditure allocations and to larger than planned fiscal deficits if spending is not reduced should revenue be under-realized.

Basis of the change
Hence the criteria used to score the indicator have been modified to incorporate both positive and negative deviations, although as the consequences of the latter are more severe, especially in the short term, more weight is given to an under-realization of revenue.

PI-19. Transparency, competition and complaints mechanisms in procurement

Reason for revision
Although several PIs impact on or are influenced by procurement, PI-19 – the only indicator devoted to the operation of the public procurement system – has been seen as inadequate given the significance of the volume of public spending that takes place through this system. Two of the three dimensions also proved difficult to rate consistently.

Basis of the change
PI-19 has been made more comprehensive in examining the strength, operation and openness of a national procurement system, by adding an additional dimension and completely reformulating the other three to reflect and provide linkages to the OECD-DAC ‘Methodology for Assessing Procurement Systems’ (MAPS) tool. The revised PI-19 draws on information collected as part of a MAPS exercise, or, if none has been recently completed, guides PEFA Assessors to appropriate sources of information and evidence by referring to the MAPS documentation.

The full text to replace the existing wording in the PEFA Performance Measurement Framework is available.

3. Support for Implementation

The Secretariat has put in place a number of measures to assist users in implementing these changes:
• Assessors’ aids: Tools such as calculation spreadsheets and guidance on interpretation and sources of evidence have been updated and posted on the website.
• Training materials: PEFA presentations at all upcoming training (and dissemination) events will highlight the indicator revisions. Training materials relating to the revised indicators will be updated to reflect the changes.
• Translation: all the materials relating to the revised indicators will be translated into French & Spanish as a matter of urgency: and, in due course, into Russian, Portuguese & Arabic.

4.  Comparability with previous assessments

As an increasing proportion of the assessments now being undertaken are in fact ‘Repeats’, (conducted with the specific intention of assessing changes in the performance of the PFM system since an earlier assessment), the Steering Committee has agreed that, where:

• the same numeric data is the primary basis for rating both the original and revised version of the indicator (as for PI-2 Dim (i), PI-3 and for PI-19 dim (iii)), the Assessment Team should be required to recalculate the earlier score using the amended criteria, but to do this solely for the purpose of identifying the direction of change over time;

• the evidence required to rate an indicator or dimension is not quantitative (which is the case for some aspects of PI-19), a ‘recalculation’ approach is unlikely to be feasible (as the – subsequent – Assessors may be unable to access or recreate the data that had been applicable at the time of the previous assessment but would now be necessary to satisfy the revised criteria), hence it may be necessary to conclude that the ratings are not directly comparable but require Assessors to describe the situation in their narrative.

NB: The database of assessment scores maintained by the Secretariat will identify instances where ratings using these revised indicators are not comparable with those from an earlier assessment.

For further information on the indicator revisions contact the PEFA Secretariat pefa@worldbank.org

Note: The posts on the IMF PFM Blog should not be reported as representing the views of the IMF. The views expressed are those of the authors and do not necessarily represent those of the IMF or IMF policy.


TrackBack URL for this entry:

Listed below are links to weblogs that reference PFM Performance Measurement Framework -- Revision of Three Indicators:


Just wondering why the Secretariat did not use the International Cities and Counties Association's (ICMA) Financial Trend Monitoring System (FTMS)? I would like to provide an "overview" so that you can determine its applicability to the PFM Performance Measurement Framework.

The Financial Trend Monitoring System (FTMS) was developed by the International City/County Management Association (ICMA) as a method for monitoring the financial condition of local governments. This system identifies factors that effect financial condition and sets the framework for their analysis. The indicators described in the ICMA publication, Evaluating Financial Condition, A Handbook for Local Government, are designed to give local governments a method of monitoring financial condition using data that is easily accessible. Using this model local government’s can provide a report to policy makers, citizens, employees, bond rating agencies, and anyone else who may be interested in the their financial wellbeing. The FTMS is intended to be used as a management tool that can help to shape long term policy priorities. The measures include:

Financial condition, as defined by the FTMS, is the ability of a locality to maintain existing service levels, withstand local and regional economic disruptions, and meet the demands of natural growth decline, and change. These conditions are examined by looking at four areas of a localities fiscal condition as follows:

1. Cash Solvency – the ability to pay the bills over the next 30 or 60 days
2. Budgetary Solvency – the ability to cover expenditures with revenues and other resources over the normal budget period
3. Long-Run Solvency – the ability to meet expenditures as they come due in the future
4. Service Level Solvency – the ability to provide services at the level and quality that are required for the health, safety, and welfare of the community and that the citizens desire and expect.

The ICMA provides a list of over 40 indicators that can serve as a litmus test for the financial condition of a locality. These indicators are broken down into specific categories for further analysis.

Adjusting for inflation converts current dollars into constant dollars. The conversion from actual dollars to constant dollars allows for analysts to take into account the appearance of growth that may be due to inflation. Adjusting for inflation involves three steps. The first step is selecting a price index. For this report the Consumer Price Index (CPI) was used. The CPI tracks the prices of good and services used by average wage earners. The goods and services include items such as food, housing, clothing, transportation, health, and recreation. The second step is selecting a base year as the starting point for comparison. The data for this report dates back to 1996 so it was used as the base year. The third step is the actual conversion from actual to constant dollars by multiplying the actual dollar amount by the conversion factor. The conversion factor is equal to the 1996 CPI divided by the CPI of following years. The table below depicts the CPI, conversation factors used for this report, and the percentage change from the previous year.

There are significant variations in the way that local governments manage their finances. These variations make it difficult to develop standards that apply from organization to organization. Therefore, there are no defined benchmarks for many of the indicators. Benchmarks for these indicators should be set by the individual municipality. A few of the indicators do have benchmarks that are generally set by bond rating agencies or organizations such as the Government Finance Officers Association (GFOA). The FTMS focuses on trends rather than defined benchmarks. For each indicator a warning trend has been defined. City staff has evaluated each indicator and assigned ratings according to the following rating scheme:

+ Green – the trend is favorable. The indicator meets any policy or performance measure set by the City.
+ Yellow – the trend is uncertain. The indicator should be watched carefully because it may move in a direction that could have a negative impact on the City’s financial health.
+ Red – the warning trend has been observed. The indicator does not meet the policy or performance measure set by the City. More information should be gathered and corrective action should be taken.

Just a thought.

Anthony H. Rainey

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.


Post a comment

Comments are moderated, and will not appear until the author has approved them.

Back to top of page
©2007 IMF. All Rights Reserved. About Us | Terms of Use
/************* DO NOT ALTER ANYTHING BELOW THIS LINE ! **************/ var s_code=s.t();if(s_code)document.write(s_code)//-->