Posted by Tim Irwin
Though intended for other purposes, sovereign credit ratings arguably have a role to play in public financial management. In particular, they provide an independent assessment of the strength of public finances that quickly allows citizens, journalists, members of parliament, and others to judge the financial risks being run by their government. Unlike many other sources of information on public finances, credit ratings are summarized in a grade, which makes them easy to understand. And, partly because grades allow countries to be ranked, they attract much more attention than, say, a .
They are controversial, because they not only inform investors but are deeply embedded in rules and regulations. A government’s credit rating can now determine whether its bonds can be bought by some investors, whether they are included in indices that other investors track, and whether they are accepted as collateral by central banks. It can determine how much capital banks have to set aside to cover possible losses on the government’s debt and whether investment banks will contract with the government or keep credit lines and derivative contracts open. So a downgrade can limit the government’s funding options, increase the interest rates it must pay, prevent it from hedging risks, and trigger sudden payment demands.
They are also controversial because their accuracy is sometimes questionable. At the onset of the global financial crisis, credit-rating rating agencies were rightly criticized for having underestimated the risks presented by securitized mortgages. Problems in sovereign ratings are less stark, but the Greek government had good ratings not long before it got into trouble, and the Japanese government has paid extremely low interest rates for years—suggesting that investors consider it very creditworthy—even though its ratings are not the highest.
From the point of view of public financial management—not to mention the regulation of financial markets—their accuracy is crucial. They are useful indicators only to the extent that they predict default or financial distress.
This chapter of the IMF's Global Financial Stability Report of October 2010 finds that credit-rating agencies “quite accurately rank sovereign default risk (that is, defaults tend to cluster in the lowest rating grades), particularly over short time horizons” (p. 105). More specifically, using Standard & Poor’s as an example, it reports that the agency’s sovereign ratings have a CAP (cumulative accuracy profile) accuracy ratio over a five-year horizon of 0.8. (The reported accuracy ratios are somewhat higher for other horizons.)
But what does this mean? Well, CAP analysis is a technique for assessing the accuracy of diagnostic systems of all kinds that is widely used in finance. To understand the meaning of a CAP accuracy ratio of 0.8, it’s helpful to translate it into the language of ROC (receiver operating characteristic) analysis, another technique for assessing diagnostic systems that is more commonly used in other fields, including medicine, engineering, and psychology, and is also sometimes used in finance.
In particular, a CAP accuracy ratio of 0.8 implies an area under the ROC curve of 0.9, which in turn implies the following: if you were repeatedly to choose two sovereigns at random, one from the group that defaulted over the course of the next 10 years and one from the group that didn’t, you’d be right about 9 out of 10 times. This doesn’t tell us whether sovereign credit ratings are better or worse than other assessments of the financial risks faced by governments—such as indicators drawn from accounts and statistics or the prices of sovereign bonds, credit default swaps, and prediction-markets contracts. But the appraisal of accuracy using CAP or ROC analysis makes it possible to address such questions.
The theory that underlies the interpretation of the CAP accuracy ratio is discussed in this working paper (Appraising Credit Ratings: Does the CAP Fit Better than the ROC?), which explains the similarities and differences between ROC and CAP analysis and argues that, though both can both be used to appraise credit ratings, ROC analysis has a couple of advantages, one of them being that its measure of accuracy—the area under the ROC curve—has this natural interpretation that the CAP ratio lacks.
Note: The posts on the IMF PFM Blog should not be reported as representing the views of the IMF. The views expressed are those of the authors and do not necessarily represent those of the IMF or IMF policy.
