Suche
Doing Good Science is Challenging: Withdrawal of an important reproducibility study stimulates self-reflection
The withdrawal of a significant reproducibility study raises questions about scientific integrity and the challenges of good research.
Doing Good Science is Challenging: Withdrawal of an important reproducibility study stimulates self-reflection
The withdrawal of a high-profile paper 1, the Methods for improving the validity of scientific studies tested has highlighted the challenges of such “reproducibility” research. The authors of the retracted paper include several Luminaries in this field.
In the study published last November in Nature Human Behavior, the authors described a rigorous research protocol with features such as large sample sizes to ensure the validity of psychological experiments. The authors applied their protocol to dozens of research projects. They reported that in 86% of replication attempts, the expected results were confirmed - one of the highest "replication rates" ever recorded in such studies. But the journal's editors retracted the paper on September 23, stating in the retraction notice 2, that they “no longer have confidence in the reliability of the results and conclusions”.
The authors agree with the journal's concerns on only one point, which they attribute to an inadvertent oversight. One of the authors, Jonathan Schooler, a psychologist at the University of California, Santa Barbara, told Nature that the research group is working on a new version of the manuscript for resubmission.
Researchers following the case say he... Problems of a principle of open science: preregistration which involves the practice of setting out the details of a study, including hypotheses and planned analyses, in writing before conducting it to prevent data manipulation and selective reporting of results.
“What this shows is that good science is hard, much harder than most people think,” says Sam Schwarzkopf, a visual neuroscientist at the University of Auckland in New Zealand. “Oftentimes, people who preregister find that their well-thought-out plans don’t pan out when they’re confronted with the cold, hard reality of data collection.”
Four teams and 64 replication attempts
The paper describes a complex and extensive experiment: four research teams each conducted registered pilot studies in the social science areas. For example, one of the studies examined whether time pressure affects decision-making 3. If the pilot study discovered an effect, the team tried to confirm the results in a sample of at least 1,500 people. All four teams tried to reproduce the selected experiments to see if they would achieve the same results. Each team attempted to replicate four of their own experiments and four from each of the three other teams.
Of the 64 replication attempts, 86% were successful – that is, they produced the expected and statistically significant results. In comparison, other replication studies in the social science fields report average replication rates of 50%.
The authors of the retracted study attributed its high replication rate to "rigorous practices" such as large sample sizes, Pre-registration and Transparency about methods back. Adopting such practices could help make studies more reliable, the authors wrote.
Shortly after the paper was published, Joseph Bak-Coleman, a social scientist at the University of Konstanz in Germany, and Berna Devezer, who studies marketing at the University of Idaho in Moscow, questioned it in a preprint 4, which was uploaded to the PsyArXiv server, its validity. They noted that the authors had not preregistered some elements of the paper, including the central question: Would the authors' protocol increase reproducibility? Separately, Bak-Coleman sent pages of analysis to the editors of Nature Human Behavior, who launched an investigation that ultimately led to the retraction.
In a comment 5, which accompanied the withdrawal, Bak-Coleman and Devezer wrote that “reproducibility was not the original outcome of the project and analyzes of reproducibility were not preregistered as claimed.” The return notice Echo had these statements. (Nature Human Behavior is published by Springer Nature, which also publishes Nature. Nature's news team is editorially independent of its publisher.)
An authorial recognition
On the day of the withdrawal, six of the authors of Nature Human Behavior published a report of their views 6. In it they admit that some of the study's analyzes were not pre-registered. However, they call other statements in the retraction notice “unaccurate,” such as the authors’ knowledge of the data at the time of the analyzes as determined by the Journal. The Journal disagrees that the withdrawal notice contains inaccuracies.
Brian Nosek, the executive director of the Center for Open Science in Charlottesville, Virginia, and co-author of the retracted study, says it was shocking to learn that the preregistration error was thrown through their project management processes. "I don't know how many times I've read that paper with these erroneous statements about everything that's pre-registered and missed it. It was just a mistake," he says.
Nosek, considered a pioneer in preregistration, also says that the original goal of the project was reproducibility, contradicting Bak-Coleman and Devezer.
Challenges of pre-registration
The saga shows the flaws of pre-registration, says Yoel Inbar, a psychologist at the University of Toronto in Canada. “I have seen many preregistrations that were vague, not strictly followed, or where the final paper was a mix of the preregistered and non-preregistered analysis,” he says.
Inbar is increasingly convinced that a better option is the pre-registration format registered reports is where researchers submit their study protocol, including its rationale and methods, to a journal for review before data collection. The editors decide whether to accept the study based on the importance of the research question and the rigor of the methods and commit to publishing the results if the work is carried out as described.
Others say the journal is part of the problem. Anne Scheel, a metascientist at Utrecht University in the Netherlands, says that although the authors made mistakes, the editors should have noticed the lack of preregistration. Reviewers don't always check preregistration, and major journals like Nature Human Behavior "need processes to actually check preregistration," she says.
A spokesman for the Journal says it is exploring changes to its practices. “The journal is exploring ways to improve transparency, standardization, and reporting requirements for preregistration in the social and behavioral sciences, which will strengthen efforts to monitor preregistration compliance,” the spokesperson adds.
Waste of time for everyone
Large-scale projects in which multiple research groups conduct the same experiments are difficult to manage, explains Olavo Amaral, a reproducibility researcher at the Universidade Federal do Rio de Janeiro in Brazil. He speaks from experience: He leads the Brazilian Reproducibility Project, an attempt to reproduce the results of dozens of biomedical studies conducted in laboratories across the country. “We can always find mistakes,” he says.
He says criticism of the retracted paper needs to be addressed, but the problems do not change his opinion of the work. “The results appear to be fairly replicable,” he says. “I don’t think the criticism of preregistration changes my opinion of the paper much.”
-
Protzko, J. et al. Nature Hum. Behavior 8, 311–319 (2023).
-
Protzko, J. et al. Nature Hum. Behavior https://doi.org/10.1038/s41562-024-01997-3 (2024).
-
Protzko, J., Zedelius, C. M. & Schooler, J. W. Psychol. Sci. 30, 1584–1591 (2019).
-
Bak-Coleman, J. & Devezer, B. Preprint at PsyArXiv https://doi.org/10.31234/osf.io/5u3kj (2024).
-
Bak-Coleman, J. & Devezer, B. Nature Hum. Behavior https://doi.org/10.1038/s41562-024-01982-w (2024).
-
Protzko, L. et al. Preprint at https://osf.io/2s94g (2024).