Scientific results hold up more often than expected. A study involving scientists from FSV UK questions the extent of the replication crisis
Scientists from the Institute of Economic Studies of the Faculty of Social Sciences, Charles University – Prof. Zuzana Havránková, Prof. Tomáš Havránek and Dr. Jaromír Baxa – participated in a study published in the prestigious journal Nature. They are building on the success from 2022 and 2023, when FSV UK academics published in this journal.
The research analyzed 110 articles published in 2022–2023 in leading economic and political science journals with mandatory data and code sharing. The authors focus on one of the fundamental questions of contemporary science: to what extent can published results be fully trusted. The study shows that most scientific findings can be re-obtained from the same data and code. At the same time, however, it is true that their conclusions are not always stable when using other, albeit legitimate, analytical procedures.
“The study brings us good news. Results in top journals in economics and political science are 85 percent replicable. A number of previous studies claimed that it is much less, even below 50 percent. We talked about the so-called replication crisis,” says Professor Havránek. “The credibility of research is a key topic today. At a time when trust in science is declining – whether in debates about vaccination, climate or security – it is important to show that scientific knowledge is based on verifiable and transparent procedures. It is precisely the open sharing of data and code that significantly strengthens its reliability. The solution is not to question science, but to do it better and more consistently,” adds Dr. Baxa.
The authors focused on two key properties of scientific results: reproducibility, i.e. whether the same results can be obtained again from the original data and code, and robustness, i.e. whether the results can also withstand other, but methodologically justified, methods of analysis. “Previous research has often shown that key results – not only in economics and political science, but in almost all sciences – are difficult to replicate, i.e. to repeat independently. Having such a mapping of the reproducibility and robustness of studies is important for the practical creation of policies based on scientific results,” says Professor Havránek.
The analysis yielded the following findings:
- More than 85% of the results were reproducible: Independent teams were able to recover the published results from the original data and code in most cases.
- Robustness was lower: With alternative analytical procedures, 72 % of the originally significant estimates remained statistically significant and in the same direction.
- Statistical significance often weakens: The proportion of statistically significant results fell from 53 % in the original studies to 43 % in the reanalysis.
- Errors in analyses are not exceptional: Coding or analytical errors were identified in approximately a quarter of the studies; more often in economics than in political science.
Over 100 research teams from around the world participated in the research. The University of Ottawa in Canada became the lead institution. “Studies involving so many research teams are becoming more and more common in our field. If you want to publish in the best journals, the competition is stronger and the demands are greater,” says Professor Havránek.
