The Unfortunate Ecosystem of Science
The Mathematics of Science’s Broken Reward System by Philip Ball
This columnist discusses the evolving ecosystem of papers published by the scientists of today. In the current state of science, researchers have pressure to find statistically significant evidence that supports “high-profile claims.” This pressure stems from a desire for job security and wanting to reap the financial and scientific benefits from having one’s name on a piece of high-profile research. To increase the likelihood of finding significant results, many researchers will take measures such as limiting sample sizes and not ensuring statistical security. To oversimplify the situation for discussion, researchers will “cheat” within the ethical bounds of science in response to the current incentive system that rewards publishing high-profile, significant results. This leads to many papers being published reporting significant results, even though “null” results are just as important to furthering scientific knowledge. Additionally, reporting significant results with weak evidence increases the probability that the conclusion is incorrect, because of the lack of statistical robustness. An incorrect conclusion undermines the base of human knowledge because further studies might cite incorrect papers. Every paper that cites the original incorrect paper or the following incorrect paper is then to some degree also incorrect, which can potentially lead to a lot of ill-founded conclusions.
This situation with scientific research can be explained with principles of evolutionary game theory. Using ill-founded methodologies and statistical analysis can be likened to an aggressive behavior. Examining lower-profile claims and thorough statistical analysis can be compared to passive behavior. As seen in class examples about evolutionary game theory, in situations in which aggressive behaviors are the dominant strategy, the most aggressive species will dominate. To apply such principles to this situation, the “aggressive” research methodologies will dominate the passive ones. As such, most papers will evolve to become more “aggressive” as a product of the broken reward system.
While not discussed the article, there are some interesting theoretical information cascades that occur as a product of the publishing of incorrect conclusions. As discussed previously, one ill-written paper can affect the integrity of all forthcoming papers that cite said paper. In tandem, these two conclusions do not bode well for the future of science. To resolve this issue, we need to examine the incentive system currently in place. Either the benefit of arriving at a well-founded conclusion (but perhaps null) needs to be increased or the benefit of a high-profile claim needs to be decreased. With the current system, it is difficult to say with certainty that the paper was well-written, the methods sound, and statistical analyses thorough.