Using Prediction Markets to Test the Peer Review Process
http://www.theatlantic.com/science/archive/2015/11/gambling-on-the-reliability-on-science-literally/414834/
This article in The Atlantic gives an interesting example of a crossing between gambling and science. In Chapter 22 we discussed how a prediction market can help aggregate individual information using the simple mechanism of gambling. In the model we constructed in class, each participant has a utility function and an initial endowment of wealth. Then he or she splits the wealth to bet on different items. The bettor’s goal is to maximize her total expected payoff. We found that in such a setting, each bettor’s best strategy is to bet on her own beliefs about the outcome. If total payoffs of all players equal total wealth, then the inverse of the market belief is the average of all individual beliefs weighted by individual wealth share.
This article applies this principle to gambling on psychology studies. The author remarks that psychology is famous for its failure to replicate the results of many of its previous studies. So researchers designed a prediction market on many psychology studies currently being replicated. Each participant has $100 initial endowment, which eliminates the weighting by wealth because everyone has an equal voice, and they are presented with 41 studies that are currently being replicated. Each stock costs $0.5, and the participants get $1 if the replication is successful. The market belief, in this case, shows how accurate people can gauge the odds of a study being successfully replicated when compared with the real probability. The result of the study shows that the people are about 71% accurate, which is quite impressive considering that they only rely on their gut feelings and the information about previous bettors’ decisions, whereas peer reviewers are professionals in the field. This study raises interesting questions about the workings of academic research: if people have gut feelings about whether or not a study can be successfully, why are these papers published anyways? The article shows that information aggregation in prediction markets can be utilized to raise important questions when the field of academics is contrasted with popular opinion. It suggests that individual peer reviewers and researchers might have ulterior motives to have some papers published, while the bettors in prediction markets are solely motivated by their true beliefs, which is directly tied to profit maximization.