Posted by Juan-Ramon Ruiz
Evaluation of government policies has taken place in Spain for many years, with a marked acceleration and qualitative improvement since 2005. However, despite Spain’s standing as an OECD country and EU member, it still has not developed a consolidated, government-wide evaluation system. Setting up evaluation systems and institutions which can operate them can be a complex and long-term process as the Spanish experience shows. The World Bank’s Independent Evaluation Group (IEG) recently produced an in-depth report on the challenges facing evaluation practices in Spain. The report can be downloaded here. The IEG usually provides technical assistance only to developing countries assisting them in design and implementation of effective monitoring and evaluation (M&E) systems and strengthening government evaluation capacities as an important part of sound governance. The study done for Spain focuses on the success factors and institutional aspects of the recently created State Agency for Public Policy Evaluation (AEVAL). The study highlights that developed countries are also still struggling to establish well-functioning government evaluation frameworks.
Spanish governments, like many others, have had a long-term interest in improving public services. This led initially to the establishment of sector-oriented institutions aimed at assessing levels of demand and types of service provision, and subsequent evaluation of the effects of sectoral policies on their target populations. Regional government also undertook, in the early 1990s, some initiatives on program evaluation and public service assessment, especially in the field of social services. Most evaluation efforts were descriptive and focused on measurement of objectives and performance; often assessments used to query citizens’ opinions and needs.
A major drawback was that evaluation activities and competencies were scattered among different institutions, including sector ministry units and internal and external auditing bodies at the central and regional levels. The creation of the Spanish Evaluation Agency, AEVAL, did much to advance the goal of institutionalizing evaluation. The agency was created at the end of 2006 under the responsibility of Ministry of Public Administrations. Since 2009 it has been located under the Office of the President. While this move signals the increased political importance of policy evaluations, its location in the administration could also undermine the agency’s objectivity and its societal acceptance.
The mission of the agency is to promote evaluation, to evaluate public policies and programs, and, with the support of management (being evaluated), to enhance the quality of services in order to improve the use of public resources and accountability. More specifically, the agency seeks to achieve two main objectives: (i) more effective and efficient use of public resources; and (ii) stronger accountability of public services to the general public, including through increased transparency and participation.
AEVAL covers the whole policy intervention cycle, from the planning process (ex-ante evaluation and appraisal), through implementation (interim evaluation) to ex-post evaluation. So far, most of evaluations have dealt with policies under implementation. Also, a distinction is made between (i) evaluation of an organization’s internal performance (microquality assessment); (ii) external assessment of service delivery, combining subjective and objective elements (mesoquality); and (iii) evaluation of public policies focused on societal impact, the degree of social legitimation and alignment with the principles of good governance (macroquality).
AEVAL’s approach for the evaluation of the quality of public services has been developed, taking into account the model of the European Foundation for Quality Management (EFQM). Its approach is also in accordance with the Common Assessment Framework (CAF), and the Evaluation Model for Learning and Improvement (EVAM) models. The latter integrates citizens’ perceptions of the quality and organizational perspectives of public services, using several practical tools.
The agency has a staff of 60 people, most of them trained in economics and law, and its budget is around $7 million. It has extensive links with regional evaluation initiatives, a critical component in the highly decentralized political setting of Spain. The development of evaluation at the sub-national level has, however, been quite limited to date. This will be an important challenge for the future, particularly since sectors such as health and education are primarily implemented at the regional level, including responsibility for their evaluation, with the central government relegated to a coordinating role.
As a new organization, AEVAL is still very much in a learning mode, developing its own approaches and procedures. At the start, AEVAL lacked clear evaluation guidelines, and evaluations were launched without a clear terms of reference. However, already the organization seems to fill an evaluation gap in Spanish government without attempting to monopolize the evaluation function. Also, the partnering with regional bodies should support the development of an evaluation culture in Spain.
Performance and evaluation information in Spain is, as in other countries, used in the annual government budget process, especially during negotiations between the Ministry of Finance and line ministries. The budget process includes several instances in which information from formal evaluations, assessments and audits, such as those conducted by National Audit Office (Ministry of Finance) could be used, and such use enriches the quality of the debate and analysis of budget proposals. AEVAL’s evaluations could be also used in this process, but its evaluations are not actually linked to current budget allocation decisions, they are presently not taken into account for decision-making on the appropriations.
The Secretary of State for Finance and Budgeting in Spain established in 2004 several working groups to study measures for further improving performance information and better integrating evaluation results in the budget process. The key limitations identified by the working groups were: (i) most of the information provided by program managers was financial in nature, and the indicators for monitoring objectives still had significant weaknesses; (ii) the possible use of evaluation for annual budgeting is limited, given the budget’s rigidities (about 90 % of the budget is committed to public debt, salaries, transfers to regional governments, and entitlements); (iii) the definition of program objectives is often very general and not strongly linked to operational objectives; (iv) most of the indicators were based on outputs, not on outcomes. All these weaknesses revealed the poor involvement of program managers in performance budgeting.
In conclusion, the use of evaluation results and recommendations for decision making and policy development in Spain is still weak. Links between the results from evaluation and budgeting decisions are, to date, very limited. The lack of such links reduces the influence that evaluation can have on the budgeting process, and on the quality of public expenditure. Therefore, a stronger political commitment to the reform process of evaluation is needed. It is important to develop formal and pragmatic integrated procedures between evaluation and budgeting. Also an important step in this direction is facilitating public access to audits and evaluations, so that citizens are informed about the results of government policies and programs.
Note: The posts on the IMF PFM Blog should not be reported as representing the views of the IMF. The views expressed are those of the authors and do not necessarily represent those of the IMF or IMF policy.
