Posted by Teresa Curristine
Performance Reporting: Insights from International Practice by Richard Boyle[1] is one of the latest additions to the Managing for Performance and Results series published by the IBM Center for the Business of Governments. This publication evaluates and compares a selection of performance reports produced in four countries (Australia, Canada, Ireland, and the United States). Despite the abundance of performance indicators developed (80% of OECD countries develop performance measures) there are very few cross-country comparisons of performance indicators produced by government departments. To date, research on performance reporting has largely concentrated on analyzing central agencies’ performance frameworks, systems, and guidelines. This paper makes a valuable contribution by providing a cross national comparative analysis of the nature and quality of output and outcome performance indicators contained in departmental government performance reports in four countries and three sectors (Agriculture, Health, and Transportation). It is important to note that this report concentrates on the indicators themselves not on the actual results produced by governments. Even this type of cross-country comparison can be challenging due to governments’ different organizational structures and dissimilar degrees of functional decentralization.
The study produces a number of interesting findings. The United States' government’s performance reports clearly come out top. Nearly all the indicators contained in the reports examined are SMART (Specific, Measurable, Achievable, Relevant, Time bound). Over 80% are outcome indicators, reflecting the US focus on reporting on programme as opposed to agency performance. Almost all indicators are quantitative, and have targets and baseline data associated with them. Many also include multiple year trend data. Boyle attributes the United States’ good practice in performance reporting to among other things having a strong performance management framework which promotes a standardized approach to presentation of performance information. In addition he stresses that having the Mercatus Center an influential independent external reviewer of agencies’ performance reports has helped improve report quality.
In contrast, Boyle highlights a number of basic issues with the Australian performance reports. Australia has the highest number of performance indicators in all three sectors, however, less that 40% are quantitative and only approximately 30% have a target. Just around 20% have baseline data associated with an indicator. Without targets and baseline data it is difficult to assess changes and progress in performance overtime. The majority of Australian indicators examined are not SMART, less than three-quarters can be classified as specific or measurable. In addition, despite a strong central government focus on outcomes and having an outcome based budget for nearly 10 years, only approximately a third of the indicators in these reports are outcomes.
These findings are surprising given that Australia is often seen as an international leader in the performance area. Australia, similar to the US and Canada, has for ten years had a central framework and guidelines underpinning its performance system, however, it appears to have been less successful than these other countries in having a consistent approach across ministries and in ensuring that ministries follow central guidelines. It is also possible that the three sectors under review are outliers, however they are sectors where one would usually expect to find more developed performance measures.
Comparatively, Ireland is a novice in the development of performance indicators. In 2006 the Irish government launched its latest performance initiative which required government ministries by 2007 to publish annual output statements. Unsurprisingly, given that it is a recent initiative, Ireland has the highest proportion of activity indicators as opposed to output or outcome indicators. Less than 30% of these indicators are quantitative or have a target associated with it and just 10% have baseline data. The targets are often not SMART and over three-quarters are not specific or measurable.
In addition to examining different country’s indicators, Richard Boyle also discusses variations across sectors. An interesting finding is that there is a greater use of outcomes in agriculture and of quantitative indicators in health.
Boyle concludes by providing six recommendations for improving performance reports. These are as follows:
1. When developing performance measurement systems, use a consistent, comparable, and structured approached to performance information across all agencies and programs.
2. Provide a good performance story to accompany the indicators.
3. Specify outcome indicators, and fully explain the results reported against the indicator.
4. Provide both target and baseline data.
5. Ensure effective use of technology in presenting the performance data collected. And,
6. Present agency performance information which includes output and activity indicators in addition to outcome indicators.
For government ministries that are developing performance indicators the inclusion in the report of examples of both good quality performance indicators and bad indicators to be avoided is almost as valuable as the recommendations.
The report makes an important contribution and helps us look behind the reform rhetoric and guidelines from the central governmental agencies to the reality on the ground for ministries in these countries.
Download Performance Reporting
--------------------------------------------------------------------------------
[1] Richard Boyle, PhD, is Head of Research with the Institute of Public Administration (IPA) in Ireland. He has worked with the IPA since 1986. His main research interests focus on public service modernization, managing for results in the public sector, and developing and implementing effective performance management and evaluation systems.
Note: The posts on the IMF PFM Blog should not be reported as representing the views of the IMF. The views expressed are those of the authors and do not necessarily represent those of the IMF or IMF policy.
