Skip to main content

Why are so many AI systems named after Muppets?

Why are so many AI systems named after Muppets?

/

An inside joke that says a lot about AI development

Share this story

Say “01101000 01101001” to BERT and ERNIE.
Say “01101000 01101001” to BERT and ERNIE.
Illustration: Getty Images / The Verge / Alex Castro

One of the biggest trends in AI recently has been the creation of machine learning models that can generate the written word with unprecedented fluidity. These programs are game-changers, potentially supercharging computers’ ability to parse and produce language. 

But something that’s gone largely unnoticed is a secondary trend — a shadow to the first — and that is: a surprising number of these tools are named after Muppets.

To date, this new breed of language AIs includes an ELMo, a BERT, a Grover, a Big BIRD, a Rosita, a RoBERTa, at least two ERNIEs (three if you include ERNIE 2.0), and a KERMIT. Big tech players like Google, Facebook, and the Allen Institute for AI are all involved, and the craze has global reach, with Chinese search giant Baidu and Beijing’s Tsinghua University contributing models. The naming convention is so well established that these systems are sometimes referred to as “Muppetware.” But who started the convention and why? 

As you might have guessed, the simple answer is: it’s an inside joke, with researchers naming AI models after Muppets because other researchers have named AI models after Muppets. But it’s a joke that happens to highlight a particular characteristic of AI research, demonstrating how labs pay homage to and build upon one another’s work. 

The trend started with ELMo, a model devised by the Allen Institute and first published online in October 2017. As is often the case with research that breaks new ground, the team behind the work wanted to come up with a snappy acronym for their model. The paper’s lead author, Matt Peters, told The Verge over email that they brainstormed ideas on Slack. 

“We had a list of letters usable in an acronym,” says Peters. “Language Model, Contextual, Embeddings, etc.” It was an engineer named Joel Grus who came up with “ELMo” to stand for “Embeddings from Language Models,” he says, and the name “instantly stuck.”

“My oldest son was about three at the time and it was also my way of dedicating the paper to him.”

“I liked it because it is somewhat whimsical but memorable,” says Peters. “My oldest son was about three at the time and it was also my way of dedicating the paper to him.”

ELMo might have been a one-off had it not been for BERT — a language model created by Google’s AI team in 2018. This model proved to be powerful and influential, and pushed a number of novel ideas about language generation into the AI mainstream.

BERT itself officially stands for Bidirectional Encoder Representations from Transformers, and although Google refused multiple requests from The Verge to discuss the origins of the name, it’s widely assumed that the researchers, like those from Allen, had the Muppets in mind. In Google’s own blog post on the topic, the company says “BERT builds upon recent work in pre-training contextual representations — including ... ELMo.”

BERT achieved state-of-the-art results on a number of tests, and has been so successful that Google recently incorporated it into its search engine. Once the model was released, the floodgates of Muppetware opened, and it was soon followed by many clever algorithms sporting brute-force acronyms, including ERNIE (Enhanced Representation through Knowledge Integration), KERMIT (Kontextuell Encoder Representations Made by Insertion Transformations), and Big BIRD (Big Bidirectional Insertion Representations for Documents).

Muppetware is having a big impact: Google incorporated BERT into its search engine to improve language comprehension.
Muppetware is having a big impact: Google incorporated BERT into its search engine to improve language comprehension.
Credit: Google

But the trend is more than just a joke. As Oren Etzioni, CEO of the Allen Institute, explains, it’s also a serious way to recognize “intellectual debt” within the AI world. “ELMo was named thus as a whim, but BERT builds directly on the insights of ELMo; Grover utilizes BERT, etc.,” Etzioni told The Verge over email. “Emphasizing the credit that is due to ELMo is very important to us ... Snuffaluffagus can’t be far behind!”

Mitchell Stern, a PhD student at Berkeley who helped create KERMIT and Big BIRD, said the naming convention was mostly fun, but it also had a “branding aspect.”

Naming models after Muppets is a way to recognize intellectual debt

“Given how widespread this trend has become, people working in this area will naturally recognize new papers containing a Sesame Street-themed name,” Stern told The Verge by email. And while not every AI language model using these new techniques is named after a Muppet (OpenAI’s well-traveled GPT-2 is one exception, though “Snuffleupagus, or Snuffy for short” was considered as a name before being rejected as too flippant), it’s a pretty sure thing that if you see a Muppetware model you know what approaches it’s using.

All this, in turn, helps us to understand how the AI world depends on openness and collaboration to generate and refine ideas. AI isn’t a discipline where lone scientists toil away in the lab at night, pumping electricity through processors, and cackling “It’s aliiiive” over a glowing command line. (Disclaimer: this certainly does happen, but it’s not always the most productive approach.) Instead, advances tend to be iterative and collaborative, with groups of researchers building upon one another’s work and ideas.

And while it’s possible that the Muppetware joke will wear thin pretty soon, until that happens, it’s a fitting tradition. After all, collaboration and respect are exactly the sort of characteristics that Sesame Street characters would be proud of.

Update Wed 11th Dec, 12:00PM ET: Story updated to note that OpenAI considered calling GPT-2 “Snuffleupagus, or Snuffy for short,” according to policy director Jack Clark.

Today’s Storystream

Feed refreshed Sep 24 Striking out

E
External Link
Emma RothSep 24
California Governor Gavin Newsom vetoes the state’s “BitLicense” law.

The bill, called the Digital Financial Assets Law, would establish a regulatory framework for companies that transact with cryptocurrency in the state, similar to New York’s BitLicense system. In a statement, Newsom says it’s “premature to lock a licensing structure” and that implementing such a program is a “costly undertaking:”

A more flexible approach is needed to ensure regulatory oversight can keep up with rapidly evolving technology and use cases, and is tailored with the proper tools to address trends and mitigate consumer harm.


A
Youtube
Andrew WebsterSep 24
Look at this Thing.

At its Tudum event today, Netflix showed off a new clip from the Tim Burton series Wednesday, which focused on a very important character: the sentient hand known as Thing. The full series starts streaming on November 23rd.


A
The Verge
Andrew WebsterSep 24
Get ready for some Netflix news.

At 1PM ET today Netflix is streaming its second annual Tudum event, where you can expect to hear news about and see trailers from its biggest franchises, including The Witcher and Bridgerton. I’ll be covering the event live alongside my colleague Charles Pulliam-Moore, and you can also watch along at the link below. There will be lots of expected names during the stream, but I have my fingers crossed for a new season of Hemlock Grove.


A
Andrew WebsterSep 24
Looking for something to do this weekend?

Why not hang out on the couch playing video games and watching TV. It’s a good time for it, with intriguing recent releases like Return to Monkey Island, Session: Skate Sim, and the Star Wars spinoff Andor. Or you could check out some of the new anime on Netflix, including Thermae Romae Novae (pictured below), which is my personal favorite time-traveling story about bathing.


A screenshot from the Netflix anime Thermae Romae Novae.
Thermae Romae Novae.
Image: Netflix
J
Twitter
Jay PetersSep 23
Twitch’s creators SVP is leaving the company.

Constance Knight, Twitch’s senior vice president of global creators, is leaving for a new opportunity, according to Bloomberg’s Cecilia D’Anastasio. Knight shared her departure with staff on the same day Twitch announced impending cuts to how much its biggest streamers will earn from subscriptions.


T
Twitter
Tom WarrenSep 23
Has the Windows 11 2022 Update made your gaming PC stutter?

Nvidia GPU owners have been complaining of stuttering and poor frame rates with the latest Windows 11 update, but thankfully there’s a fix. Nvidia has identified an issue with its GeForce Experience overlay and the Windows 11 2022 Update (22H2). A fix is available in beta from Nvidia’s website.


A
External Link
If you’re using crash detection on the iPhone 14, invest in a really good phone mount.

Motorcycle owner Douglas Sonders has a cautionary tale in Jalopnik today about the iPhone 14’s new crash detection feature. He was riding his LiveWire One motorcycle down the West Side Highway at about 60 mph when he hit a bump, causing his iPhone 14 Pro Max to fly off its handlebar mount. Soon after, his girlfriend and parents received text messages that he had been in a horrible accident, causing several hours of panic. The phone even called the police, all because it fell off the handlebars. All thanks to crash detection.

Riding a motorcycle is very dangerous, and the last thing anyone needs is to think their loved one was in a horrible crash when they weren’t. This is obviously an edge case, but it makes me wonder what other sort of false positives we see as more phones adopt this technology.


A
External Link
Ford is running out of its own Blue Oval badges.

Running out of semiconductors is one thing, but running out of your own iconic nameplates is just downright brutal. The Wall Street Journal reports badge and nameplate shortages are impacting the automaker's popular F-series pickup lineup, delaying deliveries and causing general chaos.

Some executives are even proposing a 3D printing workaround, but they didn’t feel like the substitutes would clear the bar. All in all, it's been a dreadful summer of supply chain setbacks for Ford, leading the company to reorganize its org chart to bring some sort of relief.