It’s been a week since the FDA brought the hammer down on personal genetics service 23andMe, and for followers of the company, the future is still uncertain. According to the order, the company has 15 business days (now 11 and counting) to offer the FDA a compliance plan, which is then open for comment from the agency. It’s still unclear if the end result will allow the company to continue selling its $99 saliva test kits, but the ramifications are already being felt. 23andMe announced yesterday that it was ending its marketing campaigns, but it’s likely just the first step in the process. The FDA, for its part, is staying quiet. The agency declined to comment, except to say that the 15-day grace period is standard and the resulting compliance negotiations will be kept private. However long it takes, the rest of the process is likely to occur behind closed doors.
"No one's making serious medical diagnoses based on a $99 test."
In the meantime, the internet is incensed, with many seeing the order as punitive and wrong-headed. Greg Lennon, co-founder of the genome research project SNPedia, said the FDA crackdown was akin to banning mirrors or scales because, in the wrong hands, they could provide misleading information. "I don’t think anyone honestly thinks they’re making serious medical diagnoses based on a $99 test," Lennon says. Forbes’ Matthew Herper even speculated that 23andMe’s incompetence could have been a kind of FDA-baiting, designed to provoke a furious response from Silicon Valley that would help to build a larger movement.
"Every time I read them, they're assuming I'm an idiot."
If so, it seems to be working. Few observers dispute that the FDA was within its legal rights in making the order, but it’s already led many observers to question the powers of the agency at large. Richard Epstein, a law professor at NYU, has been one prominent voice, writing for the neoconservative Manhattan Institute that FDA throttling will prevent the very research that could make personal genomics safe. "The FDA has this terrible culture," Epstein tells The Verge. "I almost resent it viscerally. Every time I read them, they're protecting me and they're assuming I'm an idiot. Well I happen to think that they're not very bright."
But behind the anger, there’s a bigger problem: why didn’t 23andMe do a better job of protecting itself? The company had six years and millions of dollars to navigate the agency — and according to the FDA, it had plenty of warning about what kinds of tests would be required. So why didn’t the company just play along?
What if the signal-to-noise ratio was just too low?
One increasingly popular answer is that 23andMe just didn’t have the goods. What if the company did the testing, and the results didn’t show the benefits that the FDA needed to see? The FDA wanted to see hard evidence that users were being led to better medical outcomes — but what if the signal-to-noise ratio was just too low to support that? It’s hard to say without access to 23andMe’s test results, but it’s an idea that’s grown increasingly plausible in recent days. Lior Pachter, a computational biologist at UC Berkeley, says it’s safe to assume that all of the genomes served by 23andMe contain at least one error, which would certainly give a regulatory agency pause.
The FDA wanted to see hard evidence
The argument is based on the ruthless logic of statistics: each one of 23andMe’s nucleotide tests is quite accurate, with error rates around one in 7,000, but as the 23andMe process tests for thousands of nucleotides individually, the sheer volume makes it likely that at least one of them will be wrong. Among statisticians, it’s known as the "multiple testing problem," and it’s a hard one to get around. Even among the important results like a user’s lifetime likelihood of developing Alzheimer’s, breast cancer, and Parkinson’s, Pachter estimates one in three 23andMe customers got a bad result.
"As you measure more things, you increase the chance that one of them is wrong."
Even worse, it’s not clear how the company might change the system to fix the multiple-testing problem. Testing for individual proteins is much cheaper than testing whole genomes, and it’s how 23andMe was able to lower testing costs from a few thousand dollars to a few hundred (delivered to consumers as $99 plus a subscription). Doctors dodge the multiple-testing problem by only testing for the specific diseases indicated by a patient’s symptoms — but that cuts against the company’s philosophy of the quantified self. If you want to get to know your genome, you need a large battery of tests, and a larger tolerance for error than the FDA seems willing to accept. It’s a crucial problem, not just for 23andMe but for the quantified self movement at large. "You're collecting more and more information," Pachter says, "but as you measure more things, you increase the chance that one of them is wrong."
What many experts anticipate is that 23andMe will simply give up the saliva test’s status as a medical device and settle into life as a regular old service, making more modest claims about its service to consumers and undertaking less aggressive marketing campaigns. It makes for a less compelling pitch, both to investors and consumers, but as long as the company keeps over-promising, it’s likely to run afoul of regulators. For more modest genomics companies and research groups like Greg Lennon’s SNPedia, it’s a familiar problem. "All along there’s been the tension between the science and the business and marketing," Lennon says. "The business folks want to push the hype, and the FDA has a mission to control the hype. The science just gets caught in the middle."