« How Much Butter Are Those Guns Costing? | Main | Whole of Government Accounts – There Aren’t Any, Batman! »

March 16, 2012

PEFA Quality Assurance – Will the Check Bounce?

Posted by Richard Allen

The PEFA Program has recently announced a new “PEFA Check” (see the March 12 PFM blog post) mechanism to enhance the quality of PEFA assessment reports.  This initiative was approved by the Steering Committee at its meeting in Brussels last December and is outlined in a note available on the PEFA website.  PEFA Check is in part a response to the recent independent evaluation of the PEFA program, which found that the PEFA brand “remains vulnerable until a more systematic method to guarantee quality can be established”.

“PEFA Check” focuses on the process of preparing the assessment reports, not the actual quality of the information itself.  First, a panel of at least four reviewers, three of which must be independent of the lead agency undertaking the assessment, would be established. Second, the reviewers would make comments on the concept note or terms of reference used in preparing the assessment, and their comments would be recorded. Third, the independent reviewers would comment on the draft assessment report, and their comments would similarly be recorded. Fourth, a template would be attached to the final assessment report certifying that the various processes had been followed and completed, and on what dates, together with the names of the independent reviewers. The comments of the reviewers, however, would not be disclosed.

The proposed new mechanism, which will come into effect with the start of Phase V of the PEFA Program in July 2012, can be welcomed as a step in the right direction, but does it do enough to deal with the quality issue?  There are several reasons for being skeptical on this issue.

First, by creating two classes of PEFA assessment report – the first with the quality “stamp” and the second without – the mechanism will likely put pressure on agencies and governments sponsoring PEFA assessments to “volunteer” to go through the validation process. Not having an endorsement will signify publicly that the reports are sub-standard in an important sense. Nevertheless, it is unfortunate that the Steering Committee did not agree to make “PEFA check” a mandatory procedure for all assessments.

Second, “PEFA Check” will not include an endorsement that a report has been completed within a defined period and that the final report has been published.  It would have been helpful if the quality certificate had included these elements.

Third and most important, by focusing on process and not substance, the Steering Committee has missed an important opportunity.  The Secretariat, when requested, already reviews the technical compliance of PEFA scores with the assessment methodology. This is a much more important check than the endorsement of process.  It would have been a relatively simple matter to add to the “PEFA Check” a certification by the Secretariat that, for each assessment report, all the indicators reviewed comply with the required methodology.  Such an endorsement would not strain the capacity of the Secretariat, one of the somewhat questionable reasons given for not going further on quality control.

Why have the PEFA partners been so reluctant and so slow to grasp the quality control nettle?

  • First, institutional pride. The decentralized process of carrying out PEFA assessments was a decision taken at the beginning of the program, and in retrospect looks mistaken. Individual agencies have insisted on relying on their own internal quality control mechanisms, and understandably resist intrusion on their turf.  The rigor and discipline of these different quality control mechanisms varies hugely, however, and is a major source of variable quality in assessment reports, on which the new procedures will have a limited impact.
  • Second, a Secretariat which is kept under a tight rein by the Steering Committee, and has been given neither the mandate nor sufficient resources to exercise an independent and effective role in the quality control process.  
  • Third, by the same token, the Steering Committee - for all its success in leading the PEFA program - displays familiar characteristics of an international bureaucracy whose members have widely dispersed values and incentives: lack of flexibility, failure to recognize problems until too late, and slowness to reach agreement on reasonable solutions that would strengthen the PEFA program.

Agencies taking the lead in PEFA assessments could unilaterally address some of the above-mentioned weaknesses in “PEFA Check” by committing to:

  • Use experts with direct experience of working in the countries concerned to review the reports. Such experts would be more likely to identify weaknesses in the scoring of the indicators.
  • Publish external comments received on draft PEFA reports.
  • Publish PEFA reports, subject to approval by the national authorities.

By meeting this challenge, each stakeholder will have an opportunity to show where they stand on the quality assurance issue.

Note: The posts on the IMF PFM Blog should not be reported as representing the views of the IMF. The views expressed are those of the authors and do not necessarily represent those of the IMF or IMF policy.

TrackBack

TrackBack URL for this entry:
https://www.typepad.com/services/trackback/6a00e54ef0059588340168e8deeede970c

Listed below are links to weblogs that reference PEFA Quality Assurance – Will the Check Bounce?:

Comments

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Comments are moderated, and will not appear until the author has approved them.

Back to top of page
©2007 IMF. All Rights Reserved. About Us | Terms of Use
/************* DO NOT ALTER ANYTHING BELOW THIS LINE ! **************/ var s_code=s.t();if(s_code)document.write(s_code)//-->