For many in the scientific community, 2015 saw a watershed moment when The Center for Open Science attempted to reproduce the findings of 100 published studies in psychology and succeeded in on 39. While some dismissed the project’s results as shoddy analysis aimed at a soft science, others declared a “reproducibility crisis” when similar reproducibility rates were found in subsequent work. Despite increasing the rigor and transparency of the replication methods, enhancing communication with the original authors, and broadening to other fields like cancer biology, attempts to reproduce high profile research seemed to revive and confirm John Ioannidis’s staggering claim from 2005—most published science is in fact false.
Several efforts have been launched to improve reproducibility in recent years. One response has been the creation of new open science publishing models, including F1000Research, where studies undergo a transparent peer review online before publication, including checks by F1000 editors to ensure that methods and data are adequately described and freely available. Another approach is pre-registration of planned analyses to proactively address concerns about reporting bias.
In addition to Brian Nosek’s Center for Open Science, several organizations have formed to create and promote standards for transparency, openness, and reproducibility in the practice and publication of research. Ben Goldacre, the British Medical Journal and others teamed up to form AllTrials, an attempt to create an open registry of methods and results for all clinical trials, followed by the founding of Ioannidis’s Meta-Research Innovation Center at Stanford.
Although the vitriol might suggest this crisis represents something novel–critics have been accused of “methodological terrorism,”” while editors and publishers of journals featuring flawed or retracted articles have been labeled myopic or worse–an examination of the history of science reveals that reproducibility has always been disputed. The above responses to a perceived crisis are necessary and welcome, but a broader socio-historical perspective is necessary to fully understand and rectify the matters at hand.
The objectives of the Better Science Ideathon are four-fold.
- to bring together groups and develop initiatives to improve the research enterprise, focusing specifically on the biomedical
- to draft a road map that connects isolated ongoing efforts – from new publication models to open-access research datasets to trail pre-registration – and aims to provide a clearer path forward.
- to recruit social scientists into the movement given that socio-political forces have played a crucial role in molding the structure and practices of the biomedical research enterprise.
- to engage students at all levels – the future scientists – who are not yet entrenched in the current system to brainstorm and explore fresh ideas for the purpose of remedying the problems of irreproducibility and unreliable research.
As John Ioannidis recently stated, the difficulty in translating basic research into effective therapies may hinge on the faulty underpinnings of published studies. For those who are concerned that the process of biomedical science does not meet its theoretical aims, our work is not the beginning, and certainly not the end. Rather, it joins a storied history of great minds trying to figure out how to understand our world. The Better Science Ideathon hopes to build on the energy of this movement.