Friday, January 01, 2010

On Prediction Markets and Self-Fulfilling Prophecies

Over on the Freakonomics blog, Ian Ayers writes:
One of the great unresolved questions of predictive analytics is trying to figure out when prediction markets will produce better predictions than good old-fashion mining of historic data. I think that there is fairly good evidence that either approach tends to beat the statistically unaided predictions of traditional experts.

But what is still unknown is whether prediction markets dominate statistical prediction.
In asking the "which is better" question, it is important to distinguish between two very different types of events for which prediction markets currently exist. Some events have a likelihood of occurrence that can safely be assumed to be independent of market predictions: they do not become more or less likely simply because beliefs about their likelihood change. Whether Justice Stevens will be next to depart the bench or snowfall in Central Park will exceed twenty inches this season are examples of such events (contracts on both are currently available on Intrade, and each is estimated to occur with 80% probability according to the price at last trade). Such events may be described as exogneous.
There is an entirely different class of events that may be termed endogenous: their likelihood of occurrence is sensitive to beliefs about this likelihood. Political campaigns, especially for party nominations in major elections, have this character. A candidate who is considered to be a prohibitive favorite will have a major fund-raising advantage, for instance if early donors believe that they will be rewarded with access, perks, or appointments. George W. Bush leveraged an aura of inevitability into a massive financial advantage in the contest for the Republican nomination in 2000, and Hillary Clinton attempted to do the same eight years later.  By the same token, a campaign that is perceived to have little chance of success may never get off the ground at all, regardless of the strengths of the candidate in question. Hence managing expectations about the likelihood of success is often a major campaign priority.
Paradoxically, the very same market characteristics that serve to enhance predictive accuracy in the case of exogenous events could undermine accuracy in forecasting endogenous events. Accurate forecasting of exogneous events requires broad participation and high levels of market visibility and liquidity, so that decentralized information can be effectively aggregated. But in the case of endogenous events, the more reliable a market is perceived by the public to be, the greater the incentives to manipulate prices at the margin. The problem is especially severe when there is a positive feedback loop between subjective beliefs and objective probabilities, as in the case of contested elections. The costs of such manipulation are small when compared with the costs of prime time advertising, and the returns can be enormous if the viability of one's campaign (or that of a competitor) is at stake.
In an earlier post I discussed some of these issues in the context of a proposal by Robin Hanson arguing for the development of prediction markets for climate change (Nate Silver was supportive of the idea, while Matt Yglesias was skeptical). Would such markets be dealing with exogenous or endogenous events? At first glance, it might seem that the events are exogenous, as in the case of this season's snowfall. But when forecasting temperatures several decades into the future there is an important feedback loop to be considered. A credible prediction that temperatures will remain stable will have the effect of stalling efforts to curtail greenhouse gas emissions, and this in turn could affect the future course of climate change. Note, however, that in this case the feedback is negative rather than positive: an decrease in the perceived likelihood of warming will result in less aggressive curtailment of emissions, and hence an increase in the objective probability of warming. As a result, any attempt at market manipulation by those who stand to lose from abatement policies will become progressively more expensive as temperatures rise. 
To put it another way, when the feedback between subjective beliefs and objective probabilities is positive, successful manipulation of prices can pay for itself by changing beliefs in a manner that becomes self-fulfilling. But when the feedback is negative, manipulation must eventually undermine its own success, since it results in beliefs that are systematically self-falsifying. For this reason I remain (cautiously) optimistic about the prospects for developing accurate prediction markets for climate change.


  1. The possibility of self-fulfilling or self-defeating prophecies is an issue with any forecasting mechanism where forecasters have any incentives to offer more, vs. less, accurate forecasts. It is not a problem particular to prediction markets.

  2. True, but the anonymity of participation in prediction markets means that we cannot control for "house effects" in interpreting the data. Most individuals discount partisan polls for this reason and they are often excluded from polling aggregates. We can't exclude partisan traders in the same way.

  3. Furthermore, pollsters and academics have reputations to protect, unlike (anonymous) traders. On the other hand, manipulators can face rapid and strong push back from other traders if they go too far. It's the balance of these two effects that will determine which system provides more accurate forecasts. My point is simply that the accuracy of prediction markets relative to other forecasting mechanisms will depend on whether we are dealing with exogenous or endogenous events, and (if the latter) whether or not the feedback between subjective beliefs and objective probabilities is positive or negative.

  4. My favorite answer is to disallow anonymous trading. When every trade and every trader is public, manipulation becomes harder to hide.

  5. My favorite answer is to disallow anonymous trading. When every trade and every trader is public, manipulation becomes harder to hide.