Pseudonyms, foreign bank accounts, IP traces, and an intentionally botched body of work. The elements of a newly unveiled sting operation sound like a top-secret, high-tech undertaking to bring down some criminal mastermind. But in reality, the sting’s target was something far more mundane: a league of peer-reviewed scientific journals that want to give research away for free.

In a report published today by Science, the sting’s organizer, writer John Bohannon, reveals that many of these journals seem to be far less rigorous than they claim. Earlier this year, Bohannon penned a bogus research paper and began submitting it to a number of these publications — just over 300 altogether. The experiments that his paper detailed, he writes, "are so hopelessly flawed that the results are meaningless." Any upstanding publication ought to have rejected the paper after its editor or a reviewer looked it over, he writes, but that didn’t always happen. More than half of the publications that responded ended up accepting the research.

Then, they asked him to pay for it to be published.

The debate over open access has been long running

In the scientific community, there are two places that research papers tend to be published: subscription journals, which only allow paying subscribers to read them, and "open-access" journals, which allow anyone to take a look. There’s long been debate over these oppositional approaches to access, particularly because academic research often receives public funding and then becomes locked behind publication paywalls.

But scientists choose subscription publications nonetheless, and they can hardly be faulted for doing so. A subscription journal like Nature offers a level of prestige and credibility that’s largely unrivaled by open-access journals, simply because of its legacy: many subscription journals are known for only publishing work of the highest caliber, making them an alluring platform for researchers.

That prestige is compounded by the fact that all of the work has been peer reviewed. When a scientific paper goes through peer review, a committee of researchers read through its results to make sure that they add up. The reviewers won’t necessarily repeat the experiments, but the goal is to ensure that nothing with inaccuracies or inconsistencies gets published. Many open-access journals also use peer review, but Bohannan makes it clear that quite a few of them fall short of vigorous quality control.

As his report points out, there may be good reason that open-access as an ideal hasn’t attained a comparable level of prestige. The journals that agreed to publish his paper don’t strictly have to worry about quality — and therefore don’t have to worry about papers passing peer review — because they aren’t earning money from subscribers looking for quality content. Instead, they ask that researchers pay a fee once their work is accepted for publication. Sometimes it’s $90. Sometimes it’s $3,100.

"Several unethical publishers have entered the market."

"During the last three or four years, several unethical publishers have entered the market and are trying to make a business on article-publishing charges," Lars Bjørnshauge tells The Verge. Bjørnshauge is the founder and managing director of the Directory of Open Access Journals, a not-for-profit resource that Bohannan describes as a "who’s who of credible open-access journals." The majority of publications Bohannon submitted to came from the directory, but a whopping 45 percent of those listed accepted his paper. In a further trail of deceit, Bohannon discovered through tracing emails and bank accounts that many of these journals didn’t even hail from the countries they claimed to. "Some of them surely aren't scams but rather just low-quality journals run by people who don't know how peer review really works," Bohannon writes in an email to The Verge. "Some are probably real (intentional) scams."

Bjørnshauge says that only 30 percent of open-access journals collect publication fees, however — a payment model that Bohannon singled out. The rest are usually supported by universities, government agencies, and philanthropic funds. "I see no differences in quality issues from open-access journals to subscription journals," Bjørnshauge tells us. "You have high-quality subscription journals and you have high-quality open-access journals, and you have low quality subscription journals and you have low quality open-access journals."

Subscription journals weren't tested by Bohannon's bogus paper

In a twist worth noting, however, Bohannon’s report on the rigor of open-access journals is both commissioned by and published by the news arm of the magazine Science, which is best known for operating a highly regarded subscription journal. And Bohannon doesn’t examine how subscription journals fare when faced with his bogus paper. "I'm really surprised that a high-quality journal like Science would allow an article to be published on their pages based on a research method which has an obvious problem," Bjørnshauge says, "namely without having a control group."

When asked if there’s a conflict of interest in Science sponsoring a paper that paints a swath of open-access journals as untrustworthy, Bohannon says that he doesn’t see one. "Science doesn't have anything to lose by the rise of open-access journals," Bohannon tells The Verge. He says that’s because journals like Science and Nature cover a wide variety of subjects, maintain their own news departments, and write for broader audiences as well. He also says there’s a "firewall" between the news and journal side of Science, and the latter wasn’t made aware of the sting until it was almost through. "There is no open-access version of journals like Science and Nature," Bohannon says. "It's apples and oranges."

Bohannon adds that he doesn’t take sides when it comes to debates about open access versus the subscription model. "I just wanted to map out the truthfulness of journals' claims to be ‘rigorous’ and ‘peer-reviewed,’" he notes. Bjørnshauge and the Directory of Open Access Journals have been working to make placement on their list subject to much stricter criteria, and Bohannon’s report only underscores that necessity. "I'm sure we will have both models side-by-side in a number of years, [and] I’m sure that in a course of not that many years we will reach the tipping point," Bjørnshauge says. "This will benefit research, it will benefit higher education, it will benefit society."