Xbox, watch TV: inside Microsoft's audacious plan to take over the living room

Taking over input one

"The living room is a special place," Marc Whitten tells me. "It’s not about just sticking a computer underneath your TV."

Whitten is the corporate vice president in charge of Xbox at Microsoft. He’s talking to me with less than a month to go before the new console ships on November 22nd, and he’s been working so hard he’s gotten himself sick. But while his voice is weary, his eyes are sharp. It’s clear he’s been thinking about this stuff for a long time.

Marc Whitten

By hijacking TV, Microsoft can keep the Xbox One interface in front of you all the time

"We want all of your entertainment to get to one place," he says, referring to the fact that game consoles like the Xbox traditionally live apart from television, plugged into a different input on the back of your TV. "You have to keep grinding after it. How do we get all of this entertainment?"

Microsoft’s solution is to partially integrate your cable box into the main Xbox experience. By hijacking TV, Microsoft can keep the Xbox One interface in front of you all the time, relegating live television to secondary status as just another app next to Netflix, Hulu, and Killer Instinct. It’s an industry first in the console business, and it’s an aggressive move. "We’ve been switching inputs for three decades," says Jeff Henshaw, the Xbox One platform engineering manager. "Getting everything on a single input is a huge step in the entertainment world."

Taking over your cable box also means the Xbox can overlay your TV signal with interesting information: a voice-activated channel guide, pop-up notifications when you get a Skype call and Xbox Live invites, a new NFL app that shows you real-time fantasy stats. You can even snap the TV window to the side of the screen while you play games. Your nasty cable interface is still there, but it allows the Xbox One to replace the cable box as the primary living-room entertainment device and go from gaming console to major new computing platform.

"I actually think during this generation of consoles that the primary user may switch from ‘gamer’ to ‘every person in the house,’" says Ben Smith, the head of Xbox TV. "It’s a huge evolution."

But trying to force that evolution has been the tech industry’s downfall for over a decade now, and the challenges aren’t getting any easier.

Attacking the TV

Smith and Whitten first began talking about the greater potential of the Xbox in 2007, just as Microsoft began work on adding Netflix to the 360. "The very specific conversation we had then was, ‘this is the kind of thing that will be key to getting us out of the basement or second bedroom and into the living room,’" says Smith. But five years later, the Xbox has become a mainstream entertainment device and the challenge is breaking through to the next level. "If you want to be the soul of the living room, you really have to have a great TV experience," Smith says. "Otherwise people will just be switching you out on inputs."

But TV isn’t just something you can just add to a platform, like Netflix or Angry Birds. The heart of the TV industry beats with the blood shed from fierce negotiations between networks and cable companies, each of which have developed decades of expertise in raising prices, locking out competition, and forcing virtually all consumer-friendly innovation to go elsewhere. Just consider all the things happening around TV that have nothing to do with your cable box: secondary media players like the Xbox, Roku, PS3, and Apple TV have all exploded in use. Services like Netflix and Hulu have rethought the basic experience of watching TV with binge watching and automated recommendation-engines. Twitter is building a business around shared TV experiences. Movie rentals now happen entirely over the internet; Blockbuster just announced that it’s closing its last 300 stores.

Ben Smith

"I want people to continue to have a relationship with their cable provider."

And yet, your cable box is still a piece of junk. If you want to get really depressed, think about the phone you had five years ago compared to the one you have today, and then think about how little the cable box under your TV has changed in that amount of time.

As long as Time Warner Cable keeps delivering new episodes of Mad Men and Homeland to people, though, there’s no incentive to change the system. And if Comcast decides to do a revolutionary deal to offer TV on a device like the Xbox, there are still hundreds of other cable companies with local monopolies in the US, and thousands more across the world. Even a company with Microsoft’s reach and influence can’t immediately disrupt the status quo at the scale of a product like the Xbox One.

"TV is still a very complicated technological roadmap for consumers, for companies, for regions and all that kind of stuff," says Whitten. "I actually don’t believe that there is one silver-bullet solution." Building a full-on DVR using the troubled CableCard standard in the US was quickly rejected, according to Smith; it wouldn’t work with satellite providers or in any of the other Xbox launch markets around the world.

So the entire Xbox One is designed around what you might call a bold compromise: instead of directly integrating TV, the system hijacks it. Rather than plugging your cable box and Xbox into the TV separately, you first plug the cable box into the Xbox, and then the Xbox into the TV. Your cable box is still there, and still doing all the heavy lifting of providing TV, but now it’s doing it in service of the overall Xbox One experience. Smith describes it as "augmenting" the cable box experience in an effort to eliminate the friction of switching between games, apps, and TV.

The biggest piece of the puzzle is the Xbox’s One Guide program guide, which lists "app channels" with listings for Hulu and Netflix alongside shows from networks like NBC and HBO, and which lets you browse and change channels by speaking to the Kinect. "We really built One Guide for the rest of the family," says Smith, "so the secondary users in the house start to use the console primarily for entertainment activities and increasingly for other things like Skype."

It’s a clever idea, but it’s also fundamentally a hack: your cable box doesn’t know anything about the Xbox One, so you’ll still see your cable interface everywhere even as you use the fancy new One Guide — during my visit Comcast’s UI popped up with every channel change. And the Xbox doesn’t have any way of directly controlling the cable box, so it has to simulate the IR commands of a remote control by cannon-blasting them out of the Kinect — in other words, you can change the channel and adjust the volume, but little else. "One of the things we can’t do is record shows," says Smith. If you want to use your DVR, it’s back to the cable remote. If you want to watch On Demand, it’s the same thing. The Xbox One might sit on top of your cable box, but it’s nowhere close to replacing it.

Jeff Henshaw

It’s a clever idea, but it’s also fundamentally a hack

If this sounds familiar, it’s because it is exactly the same system implemented by Google’s ill-fated Google TV project, which launched to great fanfare in 2010 and has been slowly dying ever since. It’s also the same trick used by Microsoft’s own WebTV platform, which launched to similarly great fanfare in 1996 and painfully lingered on until finally being killed earlier this year. Hacking your own interface on top of the cable box is a great idea in theory, but history suggests different results in practice. I am not shy about expressing my doubts that the Xbox One will fare any better while I’m in Redmond:

Henshaw: We’ve built a really cool technology into Kinect itself, where it is emitting the IR codes…

Nilay: It’s not a really cool technology.

Henshaw: It’s a super cool technology.

Nilay: It’s a super old technology.

Yet my persistent criticism doesn’t seem to faze the Xbox team. If a somewhat clunky TV integration is the price of being the primary interface in people’s living rooms, they’re surprisingly enthusiastic about paying it. "Our goal is to work with what people have today," says Henshaw. "We love Comcast, we love DirecTV, we love Time Warner, we love them all." Whitten agrees. "I actually think those guys do a good job," he says. "I want people to continue to have a relationship with their cable provider. I think that’s a great thing."

Whitten’s also not worried that people will be confused by the appearance of the cable interface while watching TV through the Xbox One — in fact, he thinks people will find it comforting. "I think it’s pretty awesome that if you plug your set-top box in, suddenly it gets these cool new features it hasn’t had before, like the ability to get a Skype call," he says. "That box, the one I’m super comfortable with and familiar with, just got a lot better for me."

And I am reminded again and again that the Xbox One platform is being designed to be around for a decade, and that it will change over time — with the implied promise that eventually you might not need a cable box at all. "As time goes forward, that integration goes deeper and deeper," says Henshaw. Smith agrees. "You get the most reach" by hijacking the cable box right now, he says. "Over time that will evolve."

And getting reach is critical — because if the Xbox One is big enough, Microsoft will have succeeded in putting Windows 8 and a Kinect under millions of TVs worldwide.

One box, three platforms

Boyd Multerer likes to remind people what life was like in 2004.

"We were working on the 360, and it was a completely different world," he says. "No smartphones, no tablets, no app stores, all that stuff. What you do on a box is just play one game, and when you switch games you reboot the box. It’s constantly rebooting into the game."

Multerer is the Xbox One’s director of development, the sort of passionate engineering nerd who’ll swing from the the big picture to arcane technical details and back again in the course of a conversation without breaking a sweat. Sometimes he does it all in a single sentence.

Boyd Multerer

"In a way you could say we gave up."

"When we’re thinking of Xbox One, it’s a different world. There are devices everywhere, there’s tablets everywhere, there’s phones everywhere," he says. "The next-generation gamer is multitasking, they’re used to doing lots of things at the same time. And you sit there as an operating system designer, and you have this tension."

High-end game developers want every bit of power available from the system so they can optimize their code and deliver more realistic graphics and bigger, more immersive worlds. But writing apps for a platform that gives you all that power requires an enormous amount of time, energy, and skill — not the best combination if you also want to attract a wide range of developers and build a big app store.

"In a way you could say we gave up," says Multerer. "We said, well, let’s make two operating systems."

So on one side of the Xbox One lies exactly what you’d expect: a next-generation gaming console that fully dedicates itself to one game at a time, with support for major advances in graphics and gameplay that designers will spend years fully exploring. ("The microcode optimization doesn’t matter as much, but data access patterns matter a tremendous amount more," Multerer casually explains to me. "It’s all about cache utilization.")

And on the other side, there’s Windows 8. Multerer calls it the "shared-app side."

"The core of Windows 8 is really fast, really robust, really good," he says. "From the shell down, it’s Windows 8. The desktop on Windows is probably not the most appropriate metaphor for TV, so we rewrote that, and we have a shell that’s very gaming-centric and works with Kinect."

Running the core of Windows 8 on the Xbox One means that the entire world of Windows developers can write powerful apps for the system and publish them to Xbox Live, which is unprecedented in the world of mainstream set-top boxes. "If you know how to write an app for Windows, you know how to write an app for Xbox One," says Multerer. "We’ve had fairly complex apps come over and start running in 24 hours or less."

Underneath it all lies the magic — a system layer called the hypervisor that manages resources and keeps both platforms running optimally even as users bounce back and forth between games, apps, and TV. To build the hypervisor, Multerer recruited the heaviest hitter he could find: David Cutler, a legendary 71-year-old Microsoft senior technical fellow who wrote the VMS mainframe operating system in 1975 and then came to Microsoft and served as the chief architect of Windows NT. It appears his work bridging the two sides of the One has gone swimmingly: jumping between massively complex games like Forza Motorsport 5, TV, and apps like Skype and Internet Explorer was seamless when I got to play with a system in Redmond. Switching in and out of Forza was particularly impressive: the game instantly resumed, with no loading times at all. "It all just works for people," says Henshaw as he walks me through the demo. "They don’t have to think about what operating system is there."

At launch, there’s not much about the Windows 8 side of the Xbox One that’s particularly surprising. Apps like Amazon Instant Video, ESPN, TED, and Netflix are already familiar to Xbox 360 owners, and while new features like snapping video to the side during gameplay are nice, they aren't major revelations. "If you think about what’s been the big technical thing to figure out in the last decade or so, it’s been streaming video," says Whitten. "So it’s not surprising that most apps are about getting the video to start."

But now that streaming video is well in hand, Whitten thinks we’re due for a whole new range of innovative apps. "There’s this other set of problems which many people thought were too hard, like the shared UI model around multiple users," he says. While phones and tablets have sparked tremendous innovation in interfaces designed for one person, shared interfaces in the living room have been ignored. Whitten sees this as opportunity. "There’s a world of things that are neither games nor entertainment that you’ll start to see people really play with."

And the key to that, Whitten says, is the new Kinect.

The Xbox One can identify who’s speaking in a room even when you’re playing a game at full volume

Building the powerful new Kinect — which now features a 1080p camera that works in total darkness, has the ability to sense your heart rate, and features a biometric log-in system that can recognize you by your skeleton — fell to development manager Kareem Choudhry. "The crazy mad scientist in me is trying to build a platform that understands the people in the room," he says. "Who they are, what they’re saying, and what they’re doing."

The raw capabilities of the new Kinect suggest that he’s gotten much closer to that goal. Using the new Kinect, the Xbox One can identify who’s speaking in a room even when you’re playing a game at full volume; it can recognize when you’ve swapped controllers with another person and adjust your settings, and it can have the camera follow you around the room as you chat on Skype.

At launch, Microsoft’s major focus with Kinect will be on voice commands, which are supposed to work much better than on the previous model. "It’s really a game of data," Choudhry says when I ask about improving the voice system. "We have the power of Microsoft and the Bing organization and all the work they’re doing on speech recognition and being able to recognize common terms on the web or the internet or pop culture."

But while voice commands and a new game called Kinect Sports Rivals are nice, it’s really the next generation of games and apps that will make it clear why Microsoft is insisting on bundling the Kinect with the Xbox One, driving the price $100 higher than the PS4. As far as I can tell, no one’s seen those apps yet, but Choudhry says the surface has barely been scratched. "I think this is one of those situations where content is king," he says. "You build a great platform, you stoke the marketplace, and great content will come."

Choudhry’s confidence in the platform’s potential is shared by everyone else I talk to at Microsoft, and it’s the most intriguing thing about the Xbox One. It was built as ambitiously as possible, even if that ambition isn’t yet fully realized. It was built to change.

The next decade of Xbox

"Ten. That’s my personal hope. I design for ten years." Multerer sounds utterly committed when he answers my question about how long he expects the Xbox One to be around. It’s a confident answer — it’s longer than the Xbox 360, which is eight years old now and showing clear signs of age. But the 360 is also a testament to how much consoles can change while remaining fundamentally the same: it launched in 2005 without built-in storage, Wi-Fi, or even HDMI, and looking at the original "blades" interface is almost embarrassing compared to the current dashboard.

Kareem Choudhry

"When we launch Xbox One, it’s not a finish line. It's a starting line."

"Ten years from now, the Xbox One shell probably won’t look anything like it does today," says Multerer. "I cannot predict how it’s going to change. So we built an environment that’s about embracing change, and I find that very exciting." That even extends to things like 4K video, which Multerer says the Xbox One technically supports. "In the future there will be better hardware that supports it better, and we’ll move along with that."

And the 10-year plan for the One is clearly on the mind of the platform’s other architects as well. "I think we still have 10 years of learning and innovation and improvement that are ahead of us," says Kinect chief Choudhry. "If you look at the leaps we’ve made in just three years, at the technology we have when Kinect first launched and what we’re shipping with Xbox One, we’re not slowing down. If anything, we’re accelerating."

This notion that the Xbox One has been designed from the outset as a computing platform that will grow and evolve over the next decade is the true marker of Microsoft’s ambitions in the living room — the Trojan horse theory come to fruition 12 years after the first Xbox. It’s also a marker of where Microsoft has come as a company: that original Xbox was a skunkworks project built by a collection of rebels, but the Xbox One incorporates pieces of everything Microsoft does, from SkyDrive to Skype to Windows itself. And while development of the new Xbox started well before outgoing CEO Steve Ballmer announced his major "One Microsoft" corporate restructuring, it’s hard not to see it as a standard-bearer for the newly unified company.

"When we launch Xbox One, it’s not a finish line," says Choudhry. "It’s a starting line." Microsoft was fatefully late to the platform race on both phones and tablets, but the Xbox One is a big bet that it can beat everyone else to domination of the living room — and the unexplored potential of an entirely new generation of development.

"This is about admitting that maybe you don’t have a perfect vision of what the future of applications is going to be," says Multerer. "So you come up with a system that’s all about upgrade and change and improvement on that side while still maintaining all the great games which are why you’re going to put this box under a TV in the first place."

That sounds familiar. It sounds like a Trojan horse.