"The Overall Evaluation of the Performance of the PEFA Programme is a Resoundingly Positive One"

Posted by Michel Lazare

"The overall evaluation of the performance of the PEFA programme is a resoundingly positive one" is the first sentence in the main findings section of executive summary of the recently published independent evaluation report of the PEFA programme.

In November 2010, the Steering Committee launched an independent evaluation of the  PEFA Program, covering the period 2004 to 2010. The evaluation was undertaken by a team of consultants led by Andrew Lawson, and the final report has now been completed and is available on the PEFA website (www.pefa.org) or clicking on the following link: Download PEFAEvaluationRevisedFinalReportJuly2011[1].

The report notes a number of markedly positive achievements of the PEFA programme. Lets just mention the following three: (a) "Across the world 90 per cent of low income, 75 per cent of middle income and 8 per cent of high income countries had been assessed, were in the process of assessment or were going to be assessed by October 2010;" (b) "The PEFA Programme has succeeded in creating a credible framework for the assessment of PFM functionality, which manages to be comprehensive in its coverage and yet sufficiently simple for the non-technical user to understand;" and (c) "The PEFA assessment framework is now used by all major development agencies working with PFM systems, either as a tool to support the design and monitoring of PFM reforms or as a key element of fiduciary risk assessment processes."

 

The independent report nevertheless notes some shortcomings or threats to the program. Chief among them is a point about quality:  "The [PEFA] brand remains vulnerable until a more systematic method to guarantee quality can be established. The Secretariat’s data suggest that there has been a significant improvement in quality over time. Yet, a number of stakeholders did identify occasional problems with quality, particularly for assessments not passing through the Secretariat’s QA process. These concerns related only to a small minority of the assessments undertaken to date, but confidence in the quality of the PEFA instrument as a whole may be undermined by the continued presence of this small minority. Steps to move as closely as possible to a full guarantee of quality are therefore recommended [in the evaluation report.]

The PEFA Steering Committee has prepared a Management Response to the final report, which provides comments on the evaluators' main findings and recommendations. The Management Response is also available on the PEFA website (www.pefa.org) or by clicking on the following link: Download EvaluationofthePEFAProgramme2004-2010-ManagementResponse-finalSept22[1].

We are reproducing below a large extract of the Management Response:

 

"Management Response by the PEFA Steering Committee

September 22^nd, 2011

    1.     Introduction

• The importance of strong public financial management (PFM) systems for achieving economic growth and equality cannot be over-emphasized. PFM systems are instrumental in a government’s ability to deliver macro-fiscal stability, policy-based allocation of resources and operational efficiency in the use of resources for service delivery. Improving PFM systems’ performance is therefore a key component of the international development effectiveness agenda.
•  The objectives of establishing the PEFA Program by its seven partner agencies[1] - almost a decade ago - were the development of a more strategic, comprehensive and collaborative approach to assessing and reforming countries’ public expenditure and financial accountability systems, and identification of performance indicators and benchmarks, in order to address both developmental and fiduciary objectives. Against this background, this independent evaluation represents an important effort to critically analyze the objectives, activities and impact of the PEFA Program since an early evaluation in 2004 and to draw lessons that may help to shape the program’s future direction.
• The evaluation was initiated by the PEFA Steering Committee in late 2010 in order to obtain an independent view of the Program’s achievements - partly building on the monitoring reports and impact studies prepared by the PEFA Secretariat previously or in parallel - in order to establish a basis on which to decide on a continuation of the Program beyond the end of the current phase III on December 31, 2011 and, if affirmative, to provide inputs to the formulation of a phase IV commencing in 2012. The findings were discussed by the PEFA Steering Committee meeting in June-July 2011 and this management response was subsequently prepared.

    2.     Main Findings and Conclusions

• We appreciate the evaluation's main conclusion that “the performance of the PEFA programme is a resoundingly positive one” which confirms the general impression we have developed during recent years and is in particular based on the Program’s creation of a credible Performance Measurement Framework that has been used for over 220 systems assessment in more than 125 countries since its launch in 2005.
• In terms of program effectiveness, we concur with the evaluators’ findings that the program has been effective in securing global adoption of the PEFA Framework both in the implementation of baseline assessments (covering about 90% of LICs and 75% of MICs) and subsequent repeat assessments (more than 50 completed), that donor collaboration has been improved and that government engagement in and ownership of the assessments is on a positive trend. The report highlights the widespread use of PEFA assessments for PFM reform discussions at country level (in 9 out of 11 country cases investigated by the evaluators) and for donors’ decisions on the use of country systems in aid operations - including by all donors consulted outside the partnership group – as well as an emerging use for empirical work by researchers.
• We accept the finding that publication of assessment reports can be further improved – from the current level of about 60% of finalized assessment reports – and will strive to improve this level through intensified efforts. Nevertheless, the encouraging developments in use of PEFA reports suggest that sharing of unpublicized reports through relevant country networks is widespread. However, it also underlines the desire for reports to be more widely disseminated in a timely manner for the benefit of additional stakeholder groups beyond country central finance agencies and their international development partners.
• We note the findings in the body of the report on the issue of quality of PEFA assessment reports. The evaluation of the program’s institutional development impact concludes that since 2008: “…. it has proven possible to significantly raise the quality of the assessment process and of the final product ….” (ref. box 5). The steady emergence of some 40 new assessments annually for six years running – including additional countries and new sponsors – indicates that the Framework has proven robust for the intended purposes. Quality review coverage of draft reports by the Secretariat has in recent years reached a very high level as has the average level of compliance with methodology. On the basis of these positive findings, outlined in the body of the report, we consider that the report overstates the case regarding quality concerns. This is by no means to say that we underestimate the importance of assessment quality, and especially the reputational risk posed by a few bad apples. High quality of assessments, therefore, is and will remain a focus of the program, particularly due to its decentralized approach to implementation of the assessments.  We accept that quality reviews of concept notes and terms of reference for assessments remain at a much lower rate and this needs to improve. On the other hand, quality is not only a result of quality assurance arrangements, which can never provide full guarantee of quality; quality also reflects the skills and experience of the assessors and the terms under which they are engaged. In this context the expansion of the pool of PEFA assessors and the efforts of the SC members and the Secretariat in monitoring application, training all stakeholders as well as issuing relevant technical and process guidance are key contributing factors to the continued improvements in quality.
• In terms of program relevance, we agree with the evaluators’ findings that the program has been consistent with the overall development strategies and policy priorities of the principal stakeholders, and with the broader donor harmonization and aid effectiveness agenda. On the other hand, we also accept that the program has provided relatively weak linkages to two of the three stated global objectives – namely (i) enhanced country ownership of PFM reform programs and (ii) improved alignment and coordination of donor support to country-led PFM reform. This is partly due to the comparative advantage of the program which lies primarily in the establishment of the common pool of information through the PEFA based assessments, and partly due to the program’s implementation strategy which has been to focus initially on establishing this common information pool, before taking on the other two objectives. Yet the evidence presented in the report, including the country case studies, points to some positive recent developments. The clear emergence of greater government involvement in the assessments, the wide range of international development agencies involved in the assessments and the frequent use of PEFA reports to guide the preparation of reform action plans, are noted as a sound platform for future action towards meeting those objectives.
• The report states that lack of awareness of the scope and potential of the PEFA Framework remains an issue. We believe this finding should be viewed with caution. The findings on effectiveness and institutional impact clearly suggest that awareness of the potential of the Framework is very high among international development agencies and the central finance agencies of partner governments in the many countries where the Framework has been applied. In countries where the Framework has not been applied the awareness is naturally much lower and an intensive sensitization effort will always be required in connection with a first application. We do agree, however, that more could be done to increase awareness of the potential use of the assessments for reform discussions between the central finance agencies and other government institutions as well as with parliaments and civil society.
• We appreciate the evaluators’ findings on program efficiency. The evaluators conclude that the program has been very efficient in production of outputs and achieved good internal coherence in its work. We note the evaluators concern that human resource capacity is a key constraint. The latter is the result of a deliberate effort to keep the Secretariat small and to mainstream as many of the program functions as possible into partner organizations or other institutions that present a comparative advantage, rather than building up a larger Secretariat capable of undertaking all such activities. In fact, keeping the Secretariat small is itself a means to promote collaboration between stakeholder institutions at the operational level, rather than replacing such operational collaboration by Secretariat interventions. However, having a small number of staff entails the risk that illnesses or departures could have a more significant impact in terms of temporarily limiting the Secretariat's capacity. The report suggests that the possibility of increasing the operational autonomy of the Secretariat should be explored. A future meeting of the Steering Committee will discuss the special relationship between the program Secretariat and the World Bank which is hosting it, in order to address this issue. . 
• We welcome the positive conclusions on the program’s sustainability. We note the need to consider an enhancement of staff capacity at the Secretariat to meet the expanding demand for its core services. At the same time, we agree that financing of the program has never been at risk and ongoing discussions among the partners suggest that its funding is secure into the medium term. It would have been helpful, however, if more had been done by the evaluators to discuss the specific sustainability issues concerning a global common good, which essentially is the nature of the PEFA Framework.
• The evaluation concludes that the program has performed well in respect of institutional development impact against its global objectives. This is a particularly welcome finding based on country level interviews which indicates that the PEFA assessment reports are widely used to inform PFM reform agendas at country level. We totally agree with the evaluators that a well designed process of implementing an assessment is a crucial part of achieving this impact.
• The evaluators judge the program’s implementation process as having worked as planned, overcoming any barriers encountered as well as having respected all agreements between the partners. We concur with this conclusion and believe this aspect of the program has been instrumental in building mutual trust between the partners as well as trust in the program’s outputs.
• We agree with the general conclusions on the program’s governance arrangements. The limitation of the partnership to seven organizations, all permanently represented on the Steering Committee, has certainly been an important factor in creating consensus, consistency and effectiveness in decision making. We recognize also that this arrangement does not incorporate the voice of other stakeholders; in fact the range of such other stakeholders is rapidly expanding as the PEFA Framework is gaining increasing recognition as an industry standard. During the entire period under evaluation, the program has addressed this issue by close collaboration with the OECD-DAC Task Force on PFM (formerly the Joint Venture on PFM) through which interaction with many other stakeholders has been developed and maintained. Whilst the number of participating partner governments and CSOs in this task force is increasing, the evaluators’ pinpointing of the limitations of this arrangement in terms of both the range of stakeholders and continuity is well taken and will be given further attention.

    3.     Recommendations

• Generally, the evaluation's recommendations and conclusions are well aligned with our own assessment and lessons learnt from program implementation. This is particularly true with respect to the need to stay on the course established by the overall objectives of the program and building on its demonstrated value-added. We also share the evaluators’ perception of most of the issues that should be a focus for deciding future development of the program. For several of these issues, the evaluators discuss a number of options and eventually conclude by recommending adoption of a particular option as the most appropriate one. We agree with the evaluators’ conclusion on the most appropriate option in a number of cases, but not all. Table 1 below presents - for each of the main issues raised in the executive summary of the report - the recommendations or preferred options suggested by the evaluators along with the Steering Committee’s opinion on the issue. The following table 2 presents supplementary recommendations made in chapter 3 of the report – but not repeated in the Executive Summary - along with the Steering Committee’s opinion on these issues. [download the full text of the Management Response to see  this table: Download EvaluationofthePEFAProgramme2004-2010-ManagementResponse-finalSept22[1] ]."