A surprising number of psychology studies can’t be reproduced

Researchers ought to be able to duplicate the findings of other scientists’ work, but a new study suggests that many published psychology results can't be recreated.

A huge, collaborative research project attempted to recreate 100 studies that were recently published in major psychology journals, and it found that only 39 of those studies' results could be replicated. That could mean that the studies were wrong in the first place, but researchers say that the findings tell more about the difficulty of designing a reproducible study than the accuracy of the studies themselves.

Studies need to be reproducible so that scientists can confirm their effects. That's why scientists have generally pushed toward reproducing studies — and not just in psychology. In part, that's to catch scientific fraud, but it's also simply to make scientific findings more trustworthy. In January 2014, the National Institutes of Health announced it would create new initiatives to address these concerns, but there still aren't widely established reproducibility guidelines. The study being published today speaks to why a bigger focus on reproducibility is necessary.

"The results present an opportunity."

"I don’t see this story as negative or pessimistic," Brian Nosek, a University of Virginia researcher and the study's corresponding author, told reporters this week. "The results present an opportunity, and the project is a demonstration of science demonstrating one of its central qualities: self-correction."

That self-correction even extended to the 39 studies that researchers were able to replicate. Of those, 83 percent had smaller effects that were in the initial reports; the drop in effect size was usually around 50 percent. The report is being published today in the journal Science.

Nosek says that there are three reasons studies' results may not have been replicated or may have changed. First, it's possible that the original result identified an effect that wasn't real. Second, it's possible that the researchers failed to detect a real effect in the reproduction. Finally, it's possible that both studies were accurate but that their methodologies differed in slight but meaningful ways.

The scientists attempting to reproduce studies tried to hew closely to the originals; they even worked with the original researchers and published their study designs for feedback from the over 270 people who worked on the reproduction project. That certainly suggests that errors are being made throughout these experiments, which mostly focused on social psychology and cognitive psychology. But for these researchers, the differing results suggests something else: that there’s a problem with the reproduction process.

"Despite its central importance, there is little known about the reproducibility of research in general," Nosek said. "And there has been a growing concern that reproducibility may be lower than expected or desired."

Research is typically focused on new results, not reproduction

One factor encouraging that, Nosek suggests, is researchers' desire for publication. Succeeding in academia means regularly publishing work, and novel results — rather than reproductions of older results or results that fail to establish a correlation — are far more likely to make it into prestigious journals. "If this occurs on a broad scale, then the published literature may be more beautiful than reality," Nosek says.

To move toward more repeatable studies, researchers and journals need to shift their interests away from novelty, the current focus. That's part of why the people behind this study — operating as part of an initiative from the Open Science Collaboration called the Reproducibility Project: Psychology — worked with the original researchers whose studies they were replicating. It encouraged them to view these experiments as steps forward, not as threats to their work.

E.J. Masicampo, a researcher at Wake Forest University, both conducted one of the reproductions and had one of his own studies reproduced by someone else. "Mine was a study that did not successfully replicate," Masicampo told reporters. His study had looked at whether giving someone a sugary beverage made it easier for them to make mentally challenging decisions while fatigued.

Both Masicampo's study and its replication asked students to decide between apartments of different sizes and distances from a college campus. It turned out that the decision was difficult for students at Florida State, where Masicampo was working, but the question was too easy for students at the University of Virginia, where it was being reproduced. "What we found, I think, was that the [methodology] that worked at the original location ... didn't quite translate for participants at the replication location," Masicampo said.

How exact must a replication be?

The group was trying to be very faithful to the original study, Masicampo said. The fact that doing so resulted in a failed replication, "highlighted the issue of how exact these replications should be and how much things change going from one place to another." Were replication standards to be put in place, these questions might be answered and lead to a clearer view of each finding's accuracy.

The researchers want to underscore that this isn't a failure of science — it's just science working normally. "The real challenge here is that we so much want to have clear answers from any individual study, from any program of research, from any report about studies," Nosek said. "But science doesn't provide certainty. At least not immediately."

Of course, all that may make you wonder: how reproducible are these findings? "It's a very good question and I don't envy the group trying it again," Nosek said. "But it does illustrate the precise point of the reproducibility effort: that even this project itself is not a final word, a last word, a definitive word about reproducibility."