Bard vs. GPT-4 on Fiscal Consolidations (i.e. fiscal adjustments)
Like everyone else, I’m fascinated by what LLMs (AI) can produce. Recently Bard reached out - because I’m special and not because Google is rolling out tests to a wide audience - so I thought I’d give it a spin. Since I’m a subscriber to Open AI’s Chat GPT, and I like to instigate a good fight, I decided to pit the two against each other.
I focused on a topic that is a bit obscure (unless you’re an economist or academic) in order to test out the chops of both, but relatively searchable enough that either machine should have been able to fire out a good result. The study of “fiscal consolidations” is something I’m also relatively familiar with, so I figured I could pick out the hogwash more easily.
The advantage was solidly in favor of GPT-4. It may not have been a good comparison as Bard’s release is likely more akin to GPT-3, but hey, it’s latest model vs. latest model in this do-or-die world. When I say “solidly in favor” I’m being kind. Bard’s output didn’t really compare. I hesitate to knock it though as we’re all pretty spoiled at this point in the recent AI revolution. In fact, Bard’s response would have blown me away . . . one month ago. My, how the world has changed.
I plan to play around and give both some more intense scrutiny with requests across the spectrum, probably keeping it within this field of study to allow for some comparison (and to hopefully teach the audience a thing or two about fiscal policy), but Bard comes across right now as GPT-3-like. Soooo early 2023. While the response to the same question was a bit more simplistic, it also did something researchers have been complaining about, and is really pretty fascinating: It just made up citations (based on a Google Scholar search and my own library). Fascinating, however, was the fact that the citations, like any good lie, were based on a small bit of truth. The researchers cited do work in the field, well part of their names do, and have even published papers on “fiscal consolidations.” There is a working paper at the IMF that has the same numbers as the one spit out by Bard - just not this working paper. The research produced by the IMF during the time period requested might have also supported the statements provided by Bard, which is ostensibly why Bard came up with the commentary it did, which leads me to wonder: why didn’t it just cite a real paper?
For GPT-4, well the response isn’t perfect, but it’s solid, paints a clearer picture, and offers citations that can lead one to articles that will give you what you want. Though the citations weren’t perfect, you can find related works that easily match what you’re looking for. The language is also more in line with that of the field and far more trustworthy in its accuracy. If I am going to start anywhere, Bard may give me a general idea of the research I wanted to look up, GPT-4 does more of the work for me and may even fool an academic who doesn’t specialize in this field of research.
Anyway, below are the responses with some commentary in bold throughout. You be the judge.
—————-
Question posed: Which type of fiscal consolidation is best for a country's economy? Cite literature from the 90s, as well as an academic article within the last year. Provide APA citations of articles noted.
Bard’s (Google) response:
There is no one-size-fits-all answer to this question, as the best type of fiscal consolidation for a country's economy will depend on a number of factors, including the country's specific economic situation and its goals for fiscal consolidation. (mostly correct, but too general to refute, or really gain much from)
However, in general, it is generally believed that fiscal consolidations that are implemented gradually and that focus on reducing spending rather than raising taxes are more likely to be successful (correctly stated, per much of the literature, though not unrefuted). This is because these types of consolidations are less likely to cause a shock to the economy and are more likely to be seen as credible by investors. (well, this is one reason noted by researchers)
A 1999 study by the International Monetary Fund (IMF) found that fiscal consolidations that were implemented gradually were more likely to be successful than those that were implemented quickly. The study also found that fiscal consolidations that focused on reducing spending were more likely to be successful than those that focused on raising taxes. (while this might capture the crux of most papers produced by IMF at the time, this particular paper…well, there is no paper to be found)
A 2022 academic article by researchers at the University of California, Berkeley, found that fiscal consolidations that are implemented gradually and that focus on reducing spending are more likely to be successful. (pretty amateur repeat of previous) The article also found that fiscal consolidations that are implemented in a way that is fair to all citizens are more likely to be successful. (some have said this, but not this one)
The article's findings are consistent with the IMF's findings, and they suggest that fiscal consolidations that are implemented gradually, that focus on reducing spending, and that are seen as fair to all citizens are more likely to be successful.
IMF (1999). Fiscal consolidations: How do they work and when do they work? Washington, DC: International Monetary Fund. (Sounds like the title of so many papers, so kudos on the attempt Bard!)
Ghosh, S., Ghosh, A., & Ostry, J. D. (2022). Fiscal consolidations: A review of the evidence. IMF Working Paper, No. 22/112. (this working paper number points to another article on capital markets, with a different author…good try “Bard”)
GPT-4’s Output:
Fiscal consolidation refers to a set of policy measures aimed at reducing budget deficits and stabilizing public debt. There are two main types of fiscal consolidation: expenditure-based and revenue-based. The choice of the best strategy depends on a country's specific circumstances, economic conditions, and preferences. (Crisply-worded and correct, with language variability that would require a bit of knowledge of the field.)
In the 1990s, several studies compared the effectiveness of these two approaches. A seminal work in this area is "Fiscal consolidations: lessons from recent history" by Alesina, Alberto, and Silvia Ardagna (1998). This paper argues that expenditure-based consolidations are more effective in achieving lasting fiscal stabilization and have fewer negative effects on economic growth. Another study supporting this view is "Successful Fiscal Consolidations" by Giavazzi, Francesco, and Marco Pagano (1996). They find that expenditure-based consolidations are more likely to lead to a reduction in public debt and are less recessionary than revenue-based consolidations. (Both citations are slightly off, but anyone doing some research could find the correct paper titles, dates, etc. Correctly noted that both of these sets of authors have produced seminal works in the field, something Bard/GPT-3 didn’t seem to get close to.)
In more recent literature, a paper from the last year worth mentioning is "Fiscal Consolidation Strategies: A Comparative Analysis" by Smith, Jane, and Michael Johnson (2022). The authors (made up, but one is close to the real thing) analyze a wide range of fiscal consolidation episodes across advanced and emerging economies. They find that expenditure-based consolidations have generally been more successful in reducing debt-to-GDP ratios and fostering economic growth, especially when combined with growth-enhancing structural reforms (correctly state conclusion found by some researchers, including myself, just not THESE researchers). However, they also note that the choice between expenditure- and revenue-based strategies should be made carefully, considering the specific conditions of each country, as well as distributional and political economy factors.
In conclusion, while there is a general consensus in the literature that expenditure-based fiscal consolidations tend to be more effective in achieving lasting fiscal stabilization and fostering economic growth, the choice of the best strategy for a particular country depends on its specific circumstances and policy preferences.
Citations:
Alesina, A., & Ardagna, S. (1998). Fiscal consolidations: lessons from recent history. Economic Policy, 13(26), 487-545. DOI: https://doi.org/10.1111/1468-0327.00023 (this one even included a doi link, common in academia, albeit one that points to a very different article. The pieces of the pie are great though!)
Giavazzi, F., & Pagano, M. (1996). Non-Keynesian effects of fiscal policy changes: international evidence and the Swedish experience. Swedish Economic Policy Review, 3(1), 67-103. (this is a solid citation as far as I can tell!)
Smith, J., & Johnson, M. (2022). Fiscal Consolidation Strategies: A Comparative Analysis. Journal of International Economics and Finance, 41(2), 298-325. (I personally haven’t heard of these researchers and can’t find a similar piece, but it’s quite the effort that I’m sure would trick just about anyone)