clock menu more-arrow no yes

Filed under:

Theranos doesn’t want its tech compared to other machines — except when it’s convenient

Hey Theranos: show your work!

Photo by Lisa Lake/Getty Images

Theranos, the hot Silicon Valley blood testing company that’s now under scrutiny after a blockbuster Wall Street Journal article, replied with 6,000 words yesterday claiming that the Journal got their reporting wrong. Part of Theranos’ reply hinges on claiming its tests are too unique to be compared to conventional testing. But when Theranos wanted FDA approval for its herpes test, the company demonstrated the herpes test worked using the style of comparison it said wouldn’t work for its proprietary technology. It appears the Silicon Valley darling’s real disruption was exploiting regulatory loopholes.

The Wall Street Journal last week cited an anecdote from a former employee who said the test results from Theranos' proprietary technologies didn't match up with those returned by conventional machines during a regulatory testing procedure. This mismatch was then covered up, the employee alleged. Theranos wrote yesterday that the WSJ got it wrong, and claimed that its Edison technology is just too sensitive for standard samples from labs.

Theranos' real disruption was exploiting regulatory loopholes

The big claim for Edison is that it needs much less blood, only a finger prick, for running a full range of tests — whatever that means. Theranos has been secretive about publishing any data associated with the system, citing trade secrets even when asked by the WSJ how many tests the Edison machines now handle. In fact, Theranos has apparently been working on "much more than just the 'Edison,'" according to its response. But the company hasn't discussed that, either.

Not all tests are subject to review by the FDA. This is the case for a category of assays called "laboratory-developed tests." The category exists because research hospitals often modify commercial tests to conform to a patient's needs. (For instance, physicians may want to run a urine sample when the manufacturer has only validated the equipment for serum or blood.) Until now, this system hasn’t raised eyebrows because research hospitals typically publish their results anyway. Theranos was able to get to market very quickly by exploiting this system — because it manufactures and conducts its tests in a single lab, Theranos’ assays also fall under "laboratory-developed tests." Using this loophole means that the company hasn’t had to provide the FDA with data that shows its tests work before marketing the tests to consumers. Nor did it follow the typical protocol for academic labs that use this system; Theranos hasn’t published its data.

"There is a segment of the market that realized the laboratory-developed test was sort of a shortcut," says Stephen Master, chief of clinical chemistry at Weill Cornell Medical College. "So that rather than saying ‘I have a new test, I'm going to develop that test and go through the whole FDA approval process, and then I will be able to sell that test as a kit to someone else,’ they said well what I can do is just get myself set up as a certified lab — and then I can make any test that I want."

"I can make any test I want."

There is, however, a law that mandates quality control for labs like Theranos, called Clinical Laboratory Improvement Amendments or CLIA. By law, Theranos' lab has to be inspected on a regular basis, by an organization that’s been accredited by the Centers for Medicare and Medicaid Services. This lab sends samples that have already been tested and analyzed to a lab like Theranos. If the company that’s undergoing testing can’t get the same results as the accredited lab, then its methods could be bad. This process is what the WSJ described. Their investigation claimed that Theranos split some of the samples last year and used them on two machines: Theranos' proprietary technology and a commercially available device. But the two machines returned different results. Because Theranos only sent back the results from the commercial machines to federal regulators, the employee characterized this as a form of "cheating."

Who decided Theranos doesn’t have any peers?

Theranos, perhaps predictably, disputes WSJ’s account, calling it "wrong and misleading." Theranos claims that the testing it underwent in February 2014 wasn’t meant to check on its proprietary technology, but rather to check if its commercial machines were operating as expected. The fact that the company did run the samples on both machines, and did obtain differing results, had nothing to do with the regulatory process, Theranos claims. So the WSJ misled readers by making it sound like Theranos was trying to fool regulators, the company said in its response. Further, Theranos says that the samples sent by the accredited lab — the legal standard for establishing whether tests are accurate — aren’t good enough for the company’s proprietary technology because "proficiency-testing samples are often modified by the proficiency-testing providers with processes like freeze-drying and adding preservatives." Theranos, essentially, is saying its technology is too sensitive to be tested in a normal way.

This is something that Theranos decided itself. No government regulator told the company that it couldn’t compare results from its proprietary equipment to results returned by other machines. Instead, federal regulators simply accepted what Theranos was doing. "Theranos has explained our process to our regulators, and no regulator has ever told us that our use of alternative assessment procedures for laboratory-developed tests is wrong," John Franklin, a spokesperson for Theranos, told The Verge in an email.

"Theranos has explained our process to our regulators."

Theranos may even be right: its technology may not be comparable to anything else on the market when it comes to testing samples with preservatives. Some samples really do return different results because of the way they’re handled, says James Nichols, a pathologist at Vanderbilt University and the government relations committee chair for the American Association for Clinical Chemistry. So those specific samples do have to be compared against the same instrument, just like Theranos claims.

But other types of samples exist, ones that "can be compared between different manufacturers and instruments, Nichols says. (He notes that few of them are whole blood, however.) So, if Theranos really wanted to, the company could opt for a more open regulatory method, similar to the one it used when it submitted its herpes test for review by the FDA. Instead, Theranos has gotten away with running its own blood samples on two different machines in its own lab, with no outside verification. This is a decision that Theranos made itself, by claiming no other tests are comparable; CMS regulators apparently have not disputed this claim. (The Verge called and emailed CMS; no one has responded.)

Theranos expects us to take its word for it

That makes determining accuracy challenging. First, regulators don’t know what the "right" results are for a sample beforehand, so they must rely on Theranos’ own data to determine if Theranos’ tests are accurate. But second, because the startup won’t publish its results, no other scientific entities can determine whether or not Theranos’ claims — both about how sensitive its tests are and about whether its tests even work — are true. Theranos expects us, essentially, to take its word for it. The company’s refusal to offer any evidence of accuracy whatsoever makes its claims suspect.

Comparisons were good enough for FDA

And Theranos has been through a form of comparative testing before: when the company submitted its herpes test for FDA approval, it had to use samples that had already been characterized by the CDC, on a different machine and in a different lab. Theranos received approval from the FDA in July for the test, its only assay that has the agency’s approval. That information is contained in the FDA's decision summary1. Evidently this method of testing worked to the FDA’s satisfaction, and Theranos clearly consented to this when it started submitting its assays to the FDA. So why the change?

From the summary: A panel of well-characterized serum samples (n=100) was obtained from the US Centers for Disease Control and Prevention (CDC). The CDC sample panel was tested with the HSV-1 IgG Assay and the results obtained by Theranos were sent to the CDC for confirmation.

Theranos consented to a comparison for the FDA

Whether the CDC's samples used to approve the herpes test contained preservatives is unclear. But the FDA seems to think it’s possible to compare Theranos' technology to conventional machines. (The FDA declined to comment for this story.) Which makes it all the stranger that Theranos has now turned around and declared that method of testing doesn’t work. So do we in fact not know if the herpes test works, because the conventional equipment isn’t comparable to the Theranos technology? Or does Theranos just want to dodge the kind of regulation that’s in place to let consumers know the results they get from blood tests are accurate? Either way, the company doesn’t look good.

Show your work

There is, of course, a very simple way for Theranos to wholly repudiate the WSJ report. It is to publish its data in peer-reviewed journals, so that independent scientists in other labs can examine them. It’s true that peer review is imperfect; papers are pulled all the time. The reason those papers are retracted is because of the post-publication scrutiny of an entire community of scientists. If Theranos really wanted to prove its accuracy beyond a doubt, a simple way to do so would be publication. "They've shown almost no data," says Weill Cornell’s Master. "And yet they made very expansive claims about what they were able to do."

Theranos won’t publish

Theranos won’t publish, its founder, Elizabeth Holmes, told The New York Times. The behavior is particularly strange, since one of the company’s board members is former CDC director William H. Foege, known for his role in smallpox eradication. It seems odd that someone so invested in public health wouldn’t want his company held to the highest possible safety standards. (Henry Kissinger, the former secretary of state for President Richard Nixon, is also on the board — as are several other former politicians.)

"We’ve never been against peer review," Holmes said to The New York Times on Wednesday. "We decided that the studies we were doing would be the FDA submissions." This is an odd position, since many companies routinely publish their trial results before submitting them to the FDA, or shortly afterward. The two do not oppose each other. Right now, Theranos appears to be trying to have it both ways: claiming FDA approval is paramount, while also claiming its tests can’t be accurately evaluated — and refusing to publish data validating its claims. But if Theranos were to publish its data, we could put the whole matter to rest. Surely patient safety is a bigger concern than trade secrets — right, Theranos?