Soren asked why I don’t think predictive accuracy is necessarily the best way to test whether someone is a good economist. First let me point out that economic theory itself predicts that it should almost impossible to make unconditional predictions of asset prices, or to predict the business cycle beyond a few months. So the fact that Irving Fisher didn’t predict the stock market crash, for instance, is totally irrelevant to his ability as an economist. But there is certainly something to Soren’s point. If economic models cannot make at least conditional predictions, what use are they?
I do believe that predictive power is very important, indeed I have argued that what most people call “objective truth,” is nothing more than statements with highly predictable implications. So I don’t want to dismiss prediction, but I think there is more to knowledge than just predictive power. It’s been a while since I read any methodology, so forgive me if I have forgotten the correct terminology, or just offer warmed up Richard Rorty (via McCloskey.)
Rorty said that “truth is what my peers let me get away with.” At first glance that seems wrong, surely truth is what’s true—not simply what people believe? Unfortunately we don’t have any direct access to reality, all we know about the world is what we believe to be true based on observation, experiment, etc. So this raises two questions; what criteria do my peers use to judge what is true, and can we think of better criteria? I believe most people are methodological pluralists (regardless of what they say), and I cannot imagine how it could be otherwise. Let’s look at some techniques that one might employ to persuade, which do not necessarily involve prediction. (I am no expert here, so I will probably make a few mistakes. But I think my general approach would withstand close scrutiny.)
1. Model elegance
2. Logic
3. Theoretical justification
4. Coherence
5. Accuracy of assumptions
6. Metaphors
Model Elegance: I don’t know if this story is apocryphal, but I recall reading that immediately after Copernicus first developed his sun-centered model of the solar system, it did not predict as well as the established Ptolemaic model, which had the earth at the center. Nevertheless, the Copernican system offered the advantage of simplicity. And it was not simply a matter of scientists preferring the easier model to work with, but also a deep, almost religious, intuition that the structure of the universe is aesthetically elegant.
Logic: In Pop Internationalism, Paul Krugman defends some of his views with simple accounting. For instance, he dismisses the fear that America will suffer from both a trade deficit and a capital outflow, by noting that the capital account is the mirror image of the current account. So if the data are collected accurately, it should be impossible for both accounts to be simultaneously in deficit. The appeal of Krugman’s model is not its predictive power, but its logic.
Theoretical Justification: Over time, the field of economics has developed an array of well-established models, most famously supply and demand. In my study of the price level during the Great Depression I applied the model of supply and demand to the international gold market. I did not pick this market because gold market shocks predicted inflation better than other markets. Indeed, I am almost certain that if one searched thousands of other markets, one would probably find some commodities that would better explain price level changes than the gold market. I chose the gold market because we have strong theoretical priors suggesting that the market might be pivotal. Unlike other commodities, gold was the medium of account throughout almost all of the 1930s. So changes in the price level were, by definition, changes in the value of gold. Thus one would expect supply and demand shocks in the gold market to be especially important, and it would make no difference to me if someone found the interwar price level was better predicted by shocks in the market for strawberries.
To make this point clearer, let me mention an article I once read (in the AER no less!) that in my view lacked theoretical justification. If my memory is correct, the article showed that over a certain period of time the return on equities on the New York Stock Exchange was slightly lower on rainy days. When I first read this article I immediately thought “data mining.” When I learned a bit more about research I realized that data mining is an inevitable (and necessary) part of the research process. In principle, one should use one set of data to construct a model, and another set to test it. But that is not always easy to do. In any case, perhaps the real problem was not data mining, but rather that there was no well-established economic theory linking rainy days and stock returns, so it looked a lot like the researcher had simply searched thousands of possible correlations, and found two variables that were significant at the 95% confidence level. (And I seem to recall that even that required a one-tailed test!)
I would not be impressed if told that the predictive power of the rainy day study is slightly higher than my gold standard study. We can never know for certain how well a model that has performed well with one set of data, will do with a new set of data. But if the model is theoretically justified, I will have a bit more confidence that any previous successes were not just a fluke.
Accuracy of Assumptions: In my research on 1933 I often refer to government statistics that show wages jumped about 20% between July and September. Such an increase in the midst of a Depression is nearly incredible, and thus I have sometimes wondered if the data are accurate. Perhaps businesses reported these numbers to please the Roosevelt administration. It seems to me that my study would be more convincing if I had outside evidence for this wage shock—such as narrative accounts from business or labor leaders indicating that something like this did occur in 1933. Adding such a narrative to my study of the Depression would give the reader more confidence that my data was accurate. It would not change the R-squared in any of the regressions, but it would make my findings more persuasive.
Coherence: Rorty argued against the “correspondence theory of truth,” the idea that our models of reality in some sense correspond to or mirror nature. He was a pragmatist who preferred to speak in terms of a “coherence theory of truth.” This means that we find explanations persuasive if they help us make sense out of a previously bewildering set of observations. Thus a historian might make sense out of the complex political changes between the 19th and 20th centuries by utilizing abstract concepts such as nationalism. If nationalism is no longer the problem it was 100 years ago, then the theory may not predict anything useful today. But it might nonetheless provide a unifying explanation for seemingly unrelated phenomena during the earlier period of history.
Milton Friedman was a forceful proponent of the view that predictive accuracy is the true test of a model. But did he really practice what he preached? His most famous book (co-authored with Anna Schwartz) was the Monetary History of the U.S. What makes that book so famous? The statistical analysis is extremely rudimentary by contemporary standards. But then why is it that when people want to argue that monetary forces caused the Great Depression, they generally refer to Friedman and Schwartz, and not some more recent VAR study of the Depression? I would argue that it is precisely the methodological pluralism of the Monetary History (and the fact that it is done so well) that makes it so persuasive. Even the footnotes are well worth reading, as they embellish the narrative in ways that support Friedman and Schwartz’s argument that causation went from money to spending, i.e. they provide a sense of which policy decisions should be regarded as exogenous. Their work is impressive on almost every level, it’s accuracy, its theoretical justification, its coherence, its rigorous logic and sophisticated understanding of economic theory. I don’t know about anyone else, but I find a study to be much more persuasive when I can see that it has been produced by superb scholars.
Metaphors: I seem to recall that McCloskey once argued that even metaphors play an important role in persuading our peers. Thus Adam Smith’s “invisible hand” metaphor nicely conveys the intuition behind his argument that markets allocate resources efficiently. On the other hand, the “trickle down” metaphor does a poor job of illustrating the intuition behind supply-side economics, which may be why it is more often employed by opponents of supply-side polices. You might say that metaphors don’t really show whether a model is correct or not, they merely persuade economists to accept a model. But what is the difference?
To summarize, economists are persuaded by all sorts of arguments. Prediction is certainly important, perhaps the most important test of a theory. But it has its limits. We cannot do a series of controlled experiments of the Great Depression, changing one factor at a time, so its not clear how prediction could be used to establish which model of the Depression is best.
My methodological pluralism does have one limit, however, any argument should be honest. Unfortunately the ethics of my profession are shameful—many economists keep searching for more “predictive power,” defined in terms of misleading indicators such as t-statistics, with little regard for whether the results actually mean anything. For too many economists, getting published is the only thing that matters.