How YouTube perfected the feed

Sometime late last year, as I was playing a video game named Dishonored 2, I did a routine YouTube search about how to beat a tricky section of the game. As usual, I found a video to answer my question. But on my next YouTube visit, the site offered me even more compelling Dishonored videos to watch: clips of people playing Dishonored without ever being detected by their enemies; clips where players killed each enemy in highly creative ways; interviews with the game’s creators; whip-smart satirical reviews. I had visited YouTube seeking an answer to my question, and it had revealed a universe.

Soon afterward, I found myself visiting YouTube several times a day. For the most part, I visited without having a specific destination — I had become accustomed to the site serving up something I would like, unprompted. In January, I grew obsessed with a folk-rock band named Pinegrove, and within weeks YouTube was serving me video of seemingly every live performance ever uploaded to its servers. I started cooking more once I got a new apartment this spring, and after searching for how to make a panzanella salad, YouTube quickly introduced me to its battalion of in-house chefs: Byron Talbott, and Serious Eats’ J. Kenji López-Alt, and the Tasty crew, among others.

YouTube has always been useful; since its founding in 2005, it has been a pillar of the internet. But over the past year or so, for me anyway, YouTube had started to seem weirdly good. The site had begun to predict with eerie accuracy what clips I might be interested in — much better than it ever had before. So what changed?

Over the course of 12 years, YouTube has transformed itself from a site driven by search to a destination in its own right. Getting there required hundreds of experiments, a handful of redesigns, and some great leaps forward in the field of artificial intelligence. But what really elevated YouTube was its evolution into a feed.

It can be hard to remember now, but at the beginning YouTube was little more than infrastructure: It offered an easy way to embed video onto other websites, which is where you were most likely to encounter it. As the site grew, YouTube became a place to find archival TV clips, catch up on late-night comedy, and watch the latest viral hits. Along with Wikipedia, YouTube is probably the web’s most notorious rabbit hole. Your coworkers mentioned the Harlem Shake at the water cooler, and so you went to YouTube and watched Harlem Shake videos for the rest of the evening.

Meanwhile, Facebook had invented the defining format of our time: the News Feed, an infinite stream of updates personalized to you based on your interests. The feed took over the consumer internet, from Tumblr to Twitter to Instagram to LinkedIn. YouTube’s early approach to personalization was much more limited: it involved asking users to subscribe to channels. The metaphor was borrowed from television, and had mixed results. A huge subscription push in 2011 had some success, but the average time a person spent watching YouTube stayed flat, according to data from ComScore.

Channels no longer dominate YouTube as they once did. Open YouTube on your phone today and you’ll find them hidden away in a separate tab. Instead, the app opens to a feed featuring a mix of videos tailored to your interests. There are videos from channels you subscribe to, yes, but there are also videos related to ones that you’ve watched before from channels you may not have seen.

This is why, after searching for straightforward Dishonored videos, I started seeing the recommendations for stealth runs through the game and satirical reviews. YouTube developed tools to make its recommendations not only personalized but deadly accurate, and the result has lifted watch time across the site.

“We knew people were coming to YouTube when they knew what they were coming to look for,” says Jim McFadden, the technical lead for YouTube recommendations, who joined the company in 2011. “We also wanted to serve the needs of people when they didn’t necessarily know what they wanted to look for.”

I first visited the company in 2011, just a few months after McFadden joined. Getting users to spend more time watching videos was then, as now, YouTube’s primary aim. At the time, it was not going particularly well. “YouTube.com as a homepage was not driving a ton of engagement,” McFadden says. “We said, well, how do we turn this thing into a destination?”

The company tried a little bit of everything: it bought professional camera equipment for top creators. It introduced “leanback,” a feature that queued new videos for you to watch while your current video played. It redesigned its home page to emphasize subscribing to channels over individual videos.

Videos watched per user remained flat, but a change made the following spring finally moved the needle: instead of basing its algorithmic recommendations on how many people had clicked a video, YouTube would instead base them on how long people had spent watching it.

Nearly overnight, creators who had profited from misleading headlines and thumbnails saw their view counts plummet. Higher-quality videos, which are strongly associated with longer watch times, surged. Watch time on YouTube grew 50 percent a year for the next three years.

I subscribed to some channels and counted myself a regular visitor to YouTube. But for it to become a multiple-times-a-day destination, YouTube would need a new set of tools — tools that only became available within the past 18 months.

When I visited the company’s offices this month, McFadden revealed the source of YouTube’s suddenly savvy recommendations: Google Brain, the parent company’s artificial intelligence division, which YouTube began using in 2015. Brain wasn’t YouTube’s first attempt at using AI; the company had applied machine-learning techniques to recommendations before, using a Google-built system known as Sibyl. Brain, however, employs a technique known as unsupervised learning: its algorithms can find relationships between different inputs that software engineers never would have guessed.

“One of the key things it does is it’s able to generalize,” McFadden said. “Whereas before, if I watch this video from a comedian, our recommendations were pretty good at saying, here’s another one just like it. But the Google Brain model figures out other comedians who are similar but not exactly the same — even more adjacent relationships. It’s able to see patterns that are less obvious.”

To name one example: a Brain algorithm began recommending shorter videos for users of the mobile app, and longer videos on YouTube’s TV app. It guessed, correctly, that varying video length by platform would result in higher watch times. YouTube launched 190 changes like this one in 2016, and is on pace to release 300 more this year. “The reality is, it’s a ton of small improvements adding up over time,” said Todd Beaupre, group product manager for YouTube’s discovery team. “For each improvement, you try 10 things and you launch one.”

The Brain algorithms also work faster than YouTube has before. In past years, it might have taken days for a user’s behavior to be incorporated into future recommendations. That made it difficult to identify trending subjects, the company said. “If we wanted to bring users back to find out what’s happening right now, we’ve kind of fixed that problem,” Beaupre said. “The delay, instead of multiple days, is measured in minutes or hours.”

Integrating Brain has had an immense impact: more than 70 percent of the time people spend watching videos on the site is now driven by YouTube’s algorithmic recommendations. Each day, YouTube recommends 200 million different videos to users, in 76 languages. And the aggregate time people spend watching videos on YouTube’s home page has grown 20 times larger than what it was three years ago.

That roughly matches my own behavior. Years ago I started visiting YouTube’s home page regularly on my lunch break, to have something to look at while I ate. But the suggestions were good enough that I started taking more regular YouTube breaks. This week I broke down and signed into YouTube on my PlayStation 4, so that I might watch its recommendations on the largest screen I own.

That’s the power of a truly personalized feed. And yet it’s striking to me how different YouTube’s feels from any of the others that inform my digital life. Facebook’s feed is based on what your friends post, along with posts from pages you like. It’s useful for knowing who’s gotten engaged or had a baby, and yet I find little pleasure in my friends’ posts beyond those milestone events. Twitter has tweets from the people you follow, plus anything those people have chosen to retweet. As a journalist I am all but required to live on Twitter, even though these days the home timeline is little more than an endless, anxious scream.

Each feed still has its strengths, though 2017 has diminished them. On Twitter, politics dominate the discussion no matter whom you follow. Facebook’s momentary enthusiasms for features like events and groups lead the feed to transform week to week in ways that are jarring, and leave me feeling less connected to everyone I’m friends with. (Image-heavy Instagram still feels like an oasis, and it’s little wonder the app is still growing so fast.)

Facebook, Twitter, Instagram — it seems notable that all these feeds ask you constantly to perform for them. YouTube is driven by performances, obviously, and yet a tiny fraction of its users ever upload a video — and YouTube never pressures them to. YouTube can be enjoyed passively, like the television channels it has worked so hard to replace. In a frantic age, there’s something calming about not being asked for my reaction to the day’s news.

YouTube’s emphasis on videos related to ones you might like means that its feed consistently seems broader in scope — more curious — than its peers. The further afield YouTube looks for content, the more it feels like an escape from other feeds. In a dark year, I’ll take all the escapism YouTube has to offer.

In 2013, writing in the Atlantic, Alexis Madrigal posited that the feed as we know it had peaked. The future, he suggested, would belong to finite experiences: email newsletters, Medium collections, 10-episode Netflix series. Endless streams of content are, after all, exhausting. “When the order of the media cosmos was annihilated, freedom did not rush into the vacuum, but an emergent order with its own logic,” Madrigal wrote. “We discovered that the stream introduced its own kinds of compulsions and controls. Faster! More! Faster! More! Faster! More!

Four years on, YouTube’s approach suggests the feed is only becoming more important. An ever-growing repository of videos, matched with ever-improving personalization technology, will be difficult to resist. YouTube now surveys users about how much they enjoyed the videos that are recommended to them; over time, the results will make YouTube smarter — and lead to more video being consumed.

Beaupre described this process to me as crossing a chasm. “There’s stuff that’s closely related to what you already liked, and stuff that’s trending and popular. But in between, that’s the magic zone.” And if YouTube’s rivals can’t find a way to cross that chasm, they may find it very difficult to compete.

Microsoft

Microsoft angers the .NET open source community with a controversial decision

Mobile

How to stop annoying robocalls on your iPhone or Android phone

Google

Google Meet hosts will be able to keep participant mics and cameras off

View all stories in Tech

Comments

While I agree that the self serving-ness of youtube’s algorithms has gotten better, for me it’s also gotten worse. In the last 2 months it’s been severely lacking in content I want. Whereas prior
3 months it was just an endless stream of stuff I wanted to watch. Now I’ve just grown complacent in my youtube viewing.

Are we using different YouTube’s? Because this has not been my experience at all. The vast majority of recommendations are dumb or irritating, and often poorly matched. YouTube will latch onto a single (broad) category, and wont stop pushing that single point of relationship.

For instance, if I’m watching a video about building a mahogany highboy all with hand tools, I do not want to see recommendations for a cheap Anna White number nailed together out of pallet boards, or a review of the new DeWalt hammer-drill. A hammer drill isn’t even used for woodworking.

What’s even worse is marking videos as "I’m not interested" won’t help either, at least in the short run.

you need to take care of your history, if you watch a video that you don’t like make sure to erase that from the history or go incognito beforehand, that solved the recommendations and trending issues

Yeah, every once in a while I like to delve into Alex Jones clips for a laugh, and then YouTube throws me into a hell pit. Deleting history is like screaming hail Marrys at Mountain View.

I’m with the author on this one. I used to never visit the youtube home page other than to click the search bar. Sometime in the last six months, that changed.

I don’t know if the recommendations got better or something happened that made me notice them more but I actually click on the recommendations now. That NEVER used to happen.

Yep. Reddit used to be what kept me up late in bed. At some point earlier this year Youtube started dominating. Really glad to see everyone else reacting the same to Google Brain’s prominence. Me and my friend have discussed it at work.

"Has Youtube like, started to suck you in the past few months?"
"You too?"

Me three. It’s fascinating to experience how it changed with the improved AI.

A couple of years back: here are latest videos from the channels you subscribe to

Then: here are latest videos from the channels you subscribe to, and old ones you haven’t watched yet

Then: here are a ton of videos just like the last video you watched

Then: here are a ton of videos similar to the last video you just watched

Then: here are videos related to ones you’ve been watching recently

And finally: here’s a mix of videos you are probably gonna want to watch

The last change that really made it work for me was when it widened out from being "closely related" (which spammed the feed with a block of stuff I was likely no longer interested in) to "sorta related" (which gave me stuff that was new and interesting).

YouTube tends to have good recommendations for SOME types of content. Things like vlogs, gaming videos, popular/trending content, etc. Once you start getting off that well-defined path though, the algorithm doesn’t perform as well. Like others have said, despite your subscriptions, asking not to see that content again, it just doesn’t work as well.

I disagree.

Just a few days ago, Youtube recommended this for me:

https://www.youtube.com/watch?v=5FzCJwc2llM

That’s hardly popular and trending content. I believe it recommended it because I sub to a few maker channels like Clickspring and This Old Tony.

I sat though at watched that. Shame on you! Now My home page is going to be flooded by this.

I do run into interesting things. If It’s not something I care about, I just don’t worry about it or click on it.

Haha, same. I watched it and enjoyed it even though I have no interest in making dioramas or anything of the sort.

I did have a "holy shit, that’s brilliant" moment, though, at seeing how he used static electricity to make fake grass stand up.

i ended up watching the whole thing too… and i have 12th grade exams coming up really soon

Lately I found myself checking more often Youtube than Facebook.

I agree with the author… I’ve found myself in these YouTube rabbit holes (similar to what I’d have with Wikipedia article links). Also using FB and Twitter less.

That’s me ever since the election drama last year. Now YouTube is my go to time wasting app when I have free time.

I noticed a similar thing, and also how much I now use the "watch later" functionality.
I now use youtube much in the same way as Pocket, which also has a great way of serving recommendations for articles that I might be interested in and save for later.

Now if I there only was a simple way to save the "watch later" playlist offline…

… YouTube has a feed?

It does if you go to your Subscriptions page.

You’ll see all the new videos from the channels you’ve subscribed to.

I don’t visit the actual YouTube homepage anymore.

That page looks like my Netflix homepage. I don’t think I’ve ever subscriped to a channel, nor is it showing me anything related to my taste. I mean … Katy Perry and Justin Bieber

Ah yeah… if you don’t subscribe to channels… then the Subscriptions page won’t do you any good

Like I said… I go straight to the Subscriptions page… so I only see videos from the channels I subscribe to. Therefore the videos should be what I’m interested in.

No Bieber for me!

YouTube’s UI and layout have seen dozens of updates over its lifetime, which, including the latest, are almost universally less intuitive and increasingly garish. It’s proven a great example of ‘if it’s not broke, don’t fix it’.
This ‘brain’ algorithm has improved YouTube immensely, however.

including the latest, are almost universally less intuitive and increasingly garish.

Blasphemy! The latest added dark mode. It’s the best youtube has ever looked.

While dark mode doesn’t really do much for me, the other changes are largely aesthetic that make the site more in line with the modern, minimal standard, at the cost of clarity and hierarchy.

The suggestion videos based on my views are good and relevant. But I will not get so much good content in future if Youtube keep getting worse in the monetization algorithm, flagging history videos, vintage weapons and even vintage computer, like Lazy Gamer videos. Is really bad and is strange The Verge don’t have any article on this subject.

View All Comments
Back to top ↑