PFM Performance Measurement Framework -- Revision of Three Indicators

Posted by the PEFA Secretariat

1.  Background and process

The PEFA program has made adjustments to three indicators where experience over the five years since the PFM Performance Measurement Framework was launched has shown that ratings do not always correctly reflect the level of system performance. In June 2010, the PEFA Steering Committee sought comments from any interested party on proposed revisions to:

• PI-2 Composition of expenditure out-turn compared to original approved budget;
• PI-3 Aggregate revenue out-turn compared to original approved budget;
• PI-19 Transparency, competition and complaints mechanisms in procurement;

Comments were received from the PEFA partners, other donor organizations, partner governments, consultants and individuals. Some of the twenty specific comments made more than one point, but none rejected the proposals (and several comments were really discussions emphasizing a particular aspect).

The proposed revisions were tested against data from earlier assessments during the design phase. This was straightforward for PIs 2 and 3, but more difficult for PI-19 (as the new rating criteria specify evidence that was not required in earlier assessments): however, the revised indicator produces results that are compatible with those obtained from the OECD DAC ‘Methodology for Assessing Procurement Systems’ tool, as had been intended.

The PEFA Steering Committee has approved final versions of three revised indicators for use. The Steering Committee wishes to bring these revised indicators into use as soon as is practicable, which may mean that some assessments currently underway are able to use them. However, all Concept Notes/Terms of Reference for proposed assessments issued after 28 February 2011 should refer to use of the revised indicators.

2. The revised indicators

PI-2. Composition of expenditure out-turn compared to original approved budget

Reasons for revision
In situations where all changes against the original budget are negative (for example, when there is a major cut, arising from a significant revenue shortfall) or positive (perhaps because of windfall revenues collected mid-year), the current methodology results in an ‘A’ rating, even when the changes are unevenly spread. In addition, the accounting treatment of contingencies can have major implications for the rating, depending on the size of the contingency and whether it is transferred (vired) to spending entities or spent/accounted for directly under the contingency head: again – assuming no other variance – PI-2 will give an ‘A’ rating to a government that has a large contingency but accounts for its use directly under the contingency head, while a government which vires the contingency to spending entities will be rated lower (in other words, the scoring criterion penalizes what is generally accepted as good practice).

Basis of the change
To remedy these problems, the current basis for calculating PI-2 has been changed to reflect the good budgetary practice of according equal marginal value to all budget lines, and a second dimension added to focus on contingencies. Dimension (i) will improve the rating of any variance from the original budget appropriations by using relative deviations from an across-the-board adjustment to the budget, to reflect the aggregate actual expenditure. However, to avoid ‘double counting’ the impact of contingencies, they are excluded from this calculation of variances. A new dimension (ii) has been calibrated to avoid penalizing ‘good practice’ by allocating an ‘A’ to a government that records little or no actual expenditure against the contingency vote (because either the contingency has not been used or it has been vired to those spending departments where actual expenditure is incurred and recorded).

PI-3. Aggregate revenue out-turn compared to original approved budget

Reason for revision
The original PI-3 rates the percentage shortfall between the forecast and the actual revenue achieved, but did not consider under-budgeting of revenue. Pessimistic revenue forecasts often result in excess revenue being used for spending that has not been subjected to the scrutiny of the budget process, while optimistic forecasts can lead to unjustifiably large expenditure allocations and to larger than planned fiscal deficits if spending is not reduced should revenue be under-realized.

Basis of the change
Hence the criteria used to score the indicator have been modified to incorporate both positive and negative deviations, although as the consequences of the latter are more severe, especially in the short term, more weight is given to an under-realization of revenue.

PI-19. Transparency, competition and complaints mechanisms in procurement

Reason for revision
Although several PIs impact on or are influenced by procurement, PI-19 – the only indicator devoted to the operation of the public procurement system – has been seen as inadequate given the significance of the volume of public spending that takes place through this system. Two of the three dimensions also proved difficult to rate consistently.

Basis of the change
PI-19 has been made more comprehensive in examining the strength, operation and openness of a national procurement system, by adding an additional dimension and completely reformulating the other three to reflect and provide linkages to the OECD-DAC ‘Methodology for Assessing Procurement Systems’ (MAPS) tool. The revised PI-19 draws on information collected as part of a MAPS exercise, or, if none has been recently completed, guides PEFA Assessors to appropriate sources of information and evidence by referring to the MAPS documentation.

The full text to replace the existing wording in the PEFA Performance Measurement Framework is available.

3. Support for Implementation

The Secretariat has put in place a number of measures to assist users in implementing these changes:
• Assessors’ aids: Tools such as calculation spreadsheets and guidance on interpretation and sources of evidence have been updated and posted on the website.
• Training materials: PEFA presentations at all upcoming training (and dissemination) events will highlight the indicator revisions. Training materials relating to the revised indicators will be updated to reflect the changes.
• Translation: all the materials relating to the revised indicators will be translated into French & Spanish as a matter of urgency: and, in due course, into Russian, Portuguese & Arabic.

4.  Comparability with previous assessments

As an increasing proportion of the assessments now being undertaken are in fact ‘Repeats’, (conducted with the specific intention of assessing changes in the performance of the PFM system since an earlier assessment), the Steering Committee has agreed that, where:

• the same numeric data is the primary basis for rating both the original and revised version of the indicator (as for PI-2 Dim (i), PI-3 and for PI-19 dim (iii)), the Assessment Team should be required to recalculate the earlier score using the amended criteria, but to do this solely for the purpose of identifying the direction of change over time;

• the evidence required to rate an indicator or dimension is not quantitative (which is the case for some aspects of PI-19), a ‘recalculation’ approach is unlikely to be feasible (as the – subsequent – Assessors may be unable to access or recreate the data that had been applicable at the time of the previous assessment but would now be necessary to satisfy the revised criteria), hence it may be necessary to conclude that the ratings are not directly comparable but require Assessors to describe the situation in their narrative.

NB: The database of assessment scores maintained by the Secretariat will identify instances where ratings using these revised indicators are not comparable with those from an earlier assessment.

For further information on the indicator revisions contact the PEFA Secretariat pefa@worldbank.org

Note: The posts on the IMF PFM Blog should not be reported as representing the views of the IMF. The views expressed are those of the authors and do not necessarily represent those of the IMF or IMF policy.

Recent