clock menu more-arrow no yes mobile

Filed under:

Thanks a lot,

How Obamacare's epic fail exposed our government's biggest tech problem

Even now that the White House’s self-imposed November 30th deadline has passed, members of the emergency tech team assigned to fix remain glued to computers in a Maryland office, working to repair the most important website the government has ever launched.

On the walls, 16 large flat-screen monitors detail how the site is performing in real time, showing bottlenecks, traffic to different parts of the site, and common error messages.

This, what’s on these screens, is the government’s highest domestic priority right now. isn’t just a website where people buy health insurance, soon to be mandatory under the Patient Protection and Affordable Care Act (ACA). It’s the support system for the way health care works now. By design, there is no way to get correct pricing on insurance without touching some part of

It’s the first time a major law has depended so heavily on the internet. Since the ACA passed, President Barack Obama has reportedly ended every health care meeting the same way: “If the website doesn’t work, nothing else matters.”

"The system is unavailable"

Despite his prescient warning, the president didn’t seem to realize that the website not working was an actual possibility. was frighteningly dysfunctional on day one. Users experienced multi-hour wait times, menus filled with blanks, and bizarre quirks like the "prison glitch" which stopped a user from proceeding until they specified how long they had been incarcerated, even if they had never been to prison.

My first attempts went nowhere. I again tried to set up an account, and I was successful but it didn't show me any policy options. I stared at it for a few minutes and then decided it was going to email me the options once it figured out which policies I could apply for. That was probably a month ago, and I've never gotten an email from them.

Fritz Mills, IL

The site got millions of visits and thousands of applications in the first 24 hours, but only six people were able to enroll in plans. "We assumed that it was up and ready to run," Vice President Joe Biden told reporters.

The site is now basically functional after two months of repairs, but anecdotally, many people who started applications are still hung up on a glitch. Others successfully completed applications but are still waiting for quotes.

There are also problems with the site’s security, the extent of which are unknown, and issues with data being sent to insurance companies, which may cause some people to enroll in the wrong plans. The system to take payment for plans still hasn't been built yet.

Only 50,000 people reportedly enrolled in the first six weeks, far below the expected 500,000. Meanwhile, more than a third of uninsured Americans still haven’t heard of, suggesting there is a crush of traffic to come.

The disastrous launch is now serving as evidence for everything on the Republican agenda from repealing health care reform — a fundamentally divisive policy point between the two parties for decades — to delaying a vote on immigration.

However, the site’s very public flop has also sparked a debate that could turn the travesty into a teachable moment. Suddenly, everyone is talking about how the government builds technology.

Battleship mentality

"Federal IT procurement" is the new buzz-phrase in Washington, and Clay Johnson couldn’t be happier about it. He’s been on a crusade since’s botched launch, co-authoring a New York Times op-ed titled "Why the Government Never Gets Tech Right."

The quotes we had were all really expensive. That was the more discouraging part of the process. I can deal with technical problems, I realize that building a website for millions of people is a bitch, especially with something as sensitive as healthcare. I'm still not sure if I entered my wife's income information successfully.

Richard Stroffolino, OH

Johnson wants higher salaries for government tech workers, a secretary of technology, and a national technology office like the UK’s Government Digital Service, founded after the failure of the $12 billion British national health-care site. But his highest priority is procurement reform.

Procurement is the term for the government’s Byzantine procedure for buying things. Under the current system, the federal government buys technology the way it buys battleships: with broad "Indefinite Delivery / Indefinite Quantity" (ID / IQ) contracts that pre-screen vendors to handle an agency’s projects for up to 10 years.

Johnson is a bit of an IT-procurement celebrity, having worked on a system that makes it easy for small companies to bid on government software projects during his time as a Presidential Innovation Fellow in the White House.

I got one application to complete and got a response saying I'm not qualified to buy insurance. This doesn't match the eligibility requirements published on the site. I've been told that I need to file an appeal.

Ben Simo, AZ

"We’re in a really dangerous spot," Johnson tells The Verge on a recent trip to Washington, DC, after giving a talk about "Technology gets twice as good or half as expensive every 18 months. The government has an efficiency problem when it comes to acquiring technology. So it’s behind that curve."

Because the procurement process is such a headache, agencies often lock in contractors for longer periods. This speeds things up, but it also gives preference to Beltway insiders and excludes smaller companies. As a result, new programming frameworks and development methods take a long time to reach the government. A company that has already bagged a 10-year contract has little incentive to innovate.

The companies that built were selected in 2007, more than two years before the government knew it would be building a health-care shopping site. The Centers for Medicare and Medicaid Services (CMS), the health department agency responsible for the site, issued a $4 billion ID / IQ that covered large tech projects for the next 10 years and approved 16 companies to build them.

When CMS was put in charge of, it turned to the 16 contractors and asked who wanted to build what. Four submitted bids for the fattest contract, which covered the bulk of the marketplace. CMS selected the lowest bid, which came from CGI Federal, a Canadian firm that got into US government work by buying an American company with an abysmal track record for delivering on government contracts.

It took me 20 minutes and was the easiest insurance application I've ever seen. Not really sure what the fuss is all about.

Brent Jackson, TN

Even the president acknowledges that this may have been a mistake. "The way the federal government does procurement and does IT is just generally not very efficient," Obama recently told a gathering of CEOs. "What we probably needed to do on the front end was to blow up how we procure for IT."

Zeroing in on procurement reform lets Obama shift the blame for’s inadequacy, but the rhetoric may have legs: the Federal Information Technology Acquisition Reform Act (FITARA), a bipartisan procurement-reform bill, was just fast-tracked for a vote in the Senate.

"I’m pretty excited now because I think there is an opportunity for change," Johnson says. "It’s hard not to think of better ways to go than the system we have now."

Price of failure

So how much has the fiasco cost? Health secretary Kathleen Sebelius said it was $174 million in October. The Government Accountability Office (GAO) said $318 million in August. GAO director of IT issues Dave Powner estimates it’s $600 million and growing, as contractors run up extra fees for fixing the site.

The true cost of is a mystery, another testament to the insanity of the procurement process. The ID / IQ includes other CMS projects, so isolating means deciphering the 256-character descriptions on thousands of "task orders" in the contracting data system, which can be as vague as "internal and external websites" or simply a purchase order number.

"It's really difficult to say with any accuracy how much it did cost," says Kaitlin Devine, a researcher at the nonprofit Sunlight Foundation who analyzed spending. "I asked somebody who is a procurement expert, who used to work at the GAO. I said, ‘Do you think it's conceivable that they don't even know what they spent?’ She was like, ‘Oh, absolutely. Like, totally.’"

Section TOC Title


The contractor bloat might have been less of a problem if the government had put someone in charge of the site. In the private sector, the CIO and CTO have power over technology decisions. In government, they’re usually subordinate to other officials and mostly advise on policy.

The US government spends $70 billion on federal IT a year. In 2009, the Office of Management and Budget introduced the Federal IT Dashboard, a public list of the largest ongoing technology projects. If the project is green, it’s on time. If it’s red, it’s in trouble. The GAO decided to put the name and photo of the agency CIO next to each project in order to add accountability to tech spending.

Immediately, officials complained.

"When the dashboard was rolled out, a lot of CIOs ran up to the executive office, to the president, saying ‘I don’t have much to do with a lot of these but my picture is on it,’" Powner says. "Well, get something to do with it."

In emails released by the House Oversight Committee, deputy CMS CIO Henry Chao referred to a potential "meltdown" and a plane crashing at takeoff, reminding his team that he told Congress under oath that "we were going to make October 1st." Days before the site debuted, CMS had around 200 to 300 workers test it simultaneously using made-up data. The site "ground to a stop," even though the goal was to support 50,000–60,000 simultaneous users. Yet no one involved delayed the launch or seemingly made an effort to warn the president.

Officials showed the president a mocked-up demo site in early September, because the real thing was nowhere near working. Obama hit the trail, telling people to "just visit, and there you can compare insurance plans, side by side, the same way you'd shop for a plane ticket on Kayak or a TV on Amazon."

Responsibility is hard to assign. Everyone shares a bit of the blame, including CGI Federal, which repeatedly protested that it wasn’t its place to raise questions about when to launch and that it was CMS’ job. QSSI, another main contractor, claims it tried to warn everybody. But there has been only one person who could be considered a casualty of the fiasco: Tony Trenkle, the CIO of CMS, whose mid-November move to the private sector suggests he took the fall.

That leaves in place a long list of arguably responsible parties including CMS administrator Marilyn Tavenner, who signed off on the site; health department CIO Frank Baitman, whose picture was on the IT Dashboard next to; CMS COO Michelle Snyder, who supposedly coordinated all the different agencies; and Sebelius, who said at a hearing, "Hold me accountable for this debacle. I’m responsible." Sebelius remains in office.

Any government IT reform needs to ensure that someone is clearly accountable for every project. "Procurement’s not the problem," says John Scott, an open-source advocate and expert on defense procurement who has analyzed "Everyone wants to look for a silver bullet. That cartridge is empty. We’ve used all the silver bullets. Really it comes down to bad management."

Section TOC Title

Rescue effort

About three weeks in, when it became apparent that was a nightmare, the White House pulled together an emergency tech team led by former budget chief Jeff Zients, a go-to fixer who has been dubbed "Obama’s weed-whacker." Programmers worked through the weekends. Chief technology officer Todd Park slept on the office floor. Officials gathered for a status update twice a day.

Zients announced that the site would be "working smoothly for the vast majority of users" by November 30th, an arbitrary but aggressive deadline. Later, the administration refined this goal, saying the site would work smoothly for 80 percent of users. The deadline for uninsured Americans to sign up for coverage was pushed from December 15th to the 23rd.

On November 30th, the team declared it had met its deadline. More than 400 software and hardware issues had been fixed. Error rate, the percentage of times a user is prevented from advancing to a new page, is down to 0.75 percent from 6 percent, and the response times are now below one second, down from eight. The site is now available 95 percent of the time, as opposed to 42 percent of the time at the beginning of November, the administration says, and can support 50,000 concurrent users.


Those numbers haven’t been audited, however, and at least one independent analysis is less rosy. The response time and error rate for the home page improved dramatically after the launch but haven’t changed much since, says Mike Smith, vice president of engineering at IT services firm Compuware, which has been pinging every 15 minutes from hundreds of real user computers since October 21st.

The site’s responsiveness also varies greatly depending on the user’s location, according to Compuware’s tests. In some states, average response time is 22 seconds.

Who built

More than 55 companies worked on and its state counterparts. It’s difficult to tell exactly who worked on what, but the contractors that worked on technical aspects of the site are Beltway regulars.

Noticeably missing from the list are companies with a reputation for excellence in technology. That’s because the requests were "like a dog whistle to really specific contractors," says Kaitlin Devine, a researcher for the nonprofit Sunlight Foundation. Requirements such as demonstrated experience with health-department systems effectively locked out newcomers.

Here are the firms with the largest contracts:

  • CGI Federal: The main contractor on Got at least $87 million and built most of the site.

  • Quality Software Services Inc.: Got more than $55 million to build the data-services hub. Now being paid more to coordinate the repair effort, despite being accused of a conflict of interest from its owner UnitedHealth, a major health care conglomerate.

  • Terremark Federal: $15.5 million for cloud hosting. Has been cited as the reason for hours of downtime. HP will be taking over the contract, a decision unrelated to Terremark’s performance.

  • The MITRE Corporation: $22 million for project management and security testing.

  • Genova Technology: $16 million for "information technology."

  • IDL Services: $9 million for "enterprise data and design support."

Other contractors that had a hand in building and supporting include Equifax Workforce Solutions, Experian, Northrop Grumman, Lockheed Martin, General Dynamics, and Accenture.

One possible conclusion is that there were too many contractors and subcontractors working on But during Congressional hearings, the government was criticized for not hiring one more: a "systems integrator" to coordinate all its vendors.

Section TOC Title

Beyond will continue to improve, but we now know that it was set up to fail. A convoluted procurement process, lack of accountability, and disempowerment of government tech officers resulted in a product that was very expensive and yet also really, really bad.

Congressional committees continue to hold hearings on what went wrong, and the GAO is working on a long-term postmortem. In the long bitter battle over health care reform, the website was supposed to be the easy part. Now we know better.

Most Americans would probably appreciate the ability to vote online one day, for example, without having to worry about errors, security holes, or the computer casting a ballot for blue when you clearly picked green. The past two months have shown us that we’re nowhere near ready for, but maybe’s failings can show us how to get there.