One of the main challenges of impact evaluation is the lack of good data that properly link specific expenditure programs to the performance indicators that are supposed to be impacted by such programs. With the SDGs, this has become even more difficult, as it is not only necessary to establish such a link, but to simultaneously account for numerous policy issues (social, health-related, economic, environmental, etc.) and their interdependencies.
Accounting for all these factors is impossible using statistical frameworks such as regression analysis. It is not scalable under benchmarking and expert opinion frameworks. And even machine learning methods struggle because data on indicators and government expenditure tends to have a poor temporal resolution. For instance, a rule of thumb for performing regression analysis is that the number of observations in a dataset should be 8 times the number of indicators. This requirement is a lot higher in machine learning algorithms such as neural networks. The problem is that development indicators are usually collected on an annual basis, so is it impossible to meet this requirement for the 232 official SDG indicators, for example.
Five years ago, we started investigating new ways to perform quantitative analyses on these data. We made use of a specific type of AI called agent computing (also known as multi-agent systems and agent-based modelling). In agent computing, we try to compensate the lack of observations by formulating theoretically-sound models of the socioeconomic mechanisms that generate the data. This allows the production of synthetic observations from computational simulations, for example, simulated development indicators.
These models can be validated by reproducing key features (like summary statistics) of the data that we have at hand. More importantly, because agent-computing models are very explicit about their mechanisms, they facilitate working with PFM experts who can validate and inform some of their assumptions. This work led us to create the research program of Policy Priority Inference (PPI), and to collaborate closely with international organizations such as the United Nations Development Programme (UNDP) and the Global Initiative for Fiscal Transparency (GIFT), as well as multiple national and subnational governments. In a nutshell, PPI takes data on government expenditure and development indicators, and infers how the former affects the latter by simulating a policy-prioritization process with spillovers between indicators and expenditure inefficiencies.
So, where do the SDGs fit into all this? Well, in order to get the most out of PPI, we need to classify the SDG indicators into budgetary tranches. This classification can be very broad or highly granular. Of course, the more detailed the information, the more reliable the inference will be. In our early academic work, we only considered the entire budget and let our artificial government decide how to spend it across a set of indicators. However, as soon as we started working with policymakers, we quickly realized the enormous potential that the SDGs perspective offered. Coincidentally, both the UNDP and GIFT had been working for some time to help governments map their expenditure categories into the SDGs. This provided us with the first datasets to investigate the nuanced and complex relationship between expenditure and indicators through PPI.
In our research, the SDGs have become an interface between expenditure data and development indicators that are relevant to numerous policy issues. This interface can vary in detail from one government to another. We have been fortunate to investigate the different levels of granularity of this interface through data from many countries in Latin America, the leading region in the world when it comes to producing SDG-linked expenditure data.
For example, in a project with the UNDP and several policymakers, we studied the feasibility of the 2018-2026 Mexican National Development Plan using indicators classified into the 17 SDGs. More recently, we worked with data from Colombia (https://bit.ly/SDG_Taxonomy_CO), Peru, and Uruguay that are classified at the level of the 169 SDG targets. Furthermore, we are currently collaborating with the Mexican social policy evaluator, CONEVAL, in applying PPI to some highly specific expenditure programs that have been identified as the main drivers of various poverty-related indicators.
Clearly, the use of agent-computing AI has been facilitated by the important efforts of PFM teams and international organizations in linking expenditure data to the SDGs. Unfortunately, sometimes, the benefits from such efforts are not self-evident to public administrators. However, we hope to show, from our experience, that embarking on such data-construction endeavors has a high potential impact. For instance, PPI is very helpful for aiding government agencies in their planning and budgeting chores. It is also very handy for NGOs in doing ex-post policy evaluations, and for political parties in designing their long-term platforms for the country’s development goals. In fact, recently, we published a worldwide study demonstrating how this tool can be used to detect bottlenecks that require structural changes instead of more expenditure (https://doi.org/10.1007/s11625-022-01095-1).
While we, as academics, continue to develop AI methods that can exploit these new datasets, it is important that public servants are aware of these initiatives and open to discussing and exploring them. Only then governments will be able to ‘cash out’ the benefits from having invested in the intersection between AI and the SDGs.
[1] Omar A. Guerrero is Head of Computational Social Science Research at The Alan Turing Institute, London (@guerrero_oa); Gonzalo Castañeda is Professor of Economics at the Centro de Investigación y Docencia Económica (CIDE), Mexico City (@Gon_CastanedaR).
Note: The posts on the IMF PFM Blog should not be reported as representing the views of the IMF. The views expressed are those of the authors and do not necessarily represent those of the IMF or IMF policy.
