The famous case of Reinhart and Rogoff in 2013 marked the start of a wave of replication assessments to research papers in various disciplines. Recently, the Federal Reserve took part of this trend and published a paper called: Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say ”Usually Not”. The authors have tried to replicate 67 papers published in 13 reputable economics journals using author-provided replication files that include both data and code. The result was roughly the following: only one third of the papers were directly replicable. With the assistance of the authors, only half of the papers had been successfully replicated. Therefore, about half of the papers are very probably conveying false conclusions.
This Federal Reserve study does not intend to provide a generalized statistical result, for it just considers a very limited list of papers. But the fact that it had focused on well-regarded economics journals makes the conclusion even more alarming.
This phenomenon does not only touch economics. In 2015, Brian Nosek, a social psychologist and head of the Center for Open Science in Virginia and 269 co-authors repeated the work reported in 98 original papers from three psychology journals. The result, as reported in Nature magazine, is shockingly similar to the statistics of the Federal Reserve in the upper paragraph. Only 39 from 100 replication attempts had a successful result. Which means that two third of the psychology literature should probably be distrusted, as concluded by Nature article.
The Economist magazine tried to understand this phenomenon in two interesting articles: How Science Goes Wrong and Trouble in The Lab. The modern academic establishment puts a huge pressure on the shoulders of scientists. They must publish regularly and a lot in order to be able to progress in their careers. Quantity is consequently considered as a much higher priority than quality.
Having said that, this replicability phenomenon cannot overshade the tremendous scientific progress we’re witnessing nowadays. However, it must lead to an urgent review of the academic publication mechanisms. This can lead to better filters and a much faster scientific progress.