Skip to main content

Meta open sources early-stage AI translation tool that works across 200 languages

Meta open sources early-stage AI translation tool that works across 200 languages

/

Meta’s ambitions to build a ‘universal translator’ continue

Share this story

An image of the Meta logo.
Illustration by Alex Castro / The Verge

Social media conglomerate Meta has created a single AI model capable of translating across 200 different languages, including many not supported by current commercial tools. The company is open-sourcing the project in the hopes that others will build on its work.

The AI model is part of an ambitious R&D project by Meta to create a so-called “universal speech translator,” which the company sees as important for growth across its many platforms — from Facebook and Instagram, to developing domains like VR and AR. Machine translation not only allows Meta to better understand its users (and so improve the advertising systems that generate 97 percent of its revenue) but could also be the foundation of a killer app for future projects like its augmented reality glasses.

The model’s translations definitely won’t be flawless

Experts in machine translation told The Verge that Meta’s latest research was ambitious and thorough, but noted that the quality of some of the model’s translations would likely be well below that of better-supported languages like Italian or German.

“The major contribution here is data,” Professor Alexander Fraser, an expert in computational linguistics at LMU Munich in Germany, told The Verge. “What is significant is 100 new languages [that can be translated by Meta’s model].”

Meta’s achievements stem, somewhat paradoxically, from both the scope and focus of its research. While most machine translation models handle only a handful of languages, Meta’s model is all-encapsulating: it’s a single system able to translate in more than 40,000 different directions between 200 different languages. But Meta is also interested in including “low-resource languages” in the model — languages with fewer than 1 million publicly-available translated sentence-pairs. These include many African and Indian languages not usually supported by commercial machine translation tools.

“what would it take to produce translation technology that works for everybody?”

Meta AI research scientist Angela Fan, who worked on the project, told The Verge that the team was inspired by the lack of attention paid to such lower-resource languages in this field. “Translation doesn’t even work for the languages we speak, so that’s why we started this project,” said Fan. “We have this inclusion motivation of like — ‘what would it take to produce translation technology that works for everybody’?”

Fan says the model, described in a research paper here, is already being tested to support a project that helps Wikipedia editors translate articles into other languages. The techniques developed in creating the model will also be integrated into Meta’s translation tools soon.

How do you judge a translation?

Translation is a difficult task at the best of times, and machine translation can be notoriously flaky. When applied at scale on Meta’s platforms, even a small number of errors can produce disastrous results — as, for example, when Facebook mistranslated a post by a Palestinian man from “good morning” to “hurt them,” leading to his arrest by Israeli police.

To evaluate the quality of the new model’s output, Meta created a test dataset consisting of 3001 sentence-pairs for each language covered by the model, each translated from English into a target language by someone who is both a professional translator and native speaker.

The researchers ran these sentences through their model, and compared the machine’s translation with the human reference sentences using a benchmark common in machine translation known as BLEU (which stands for BiLingual Evaluation Understudy).

Meta’s model delivers improved benchmarks, but they can’t tell the whole story

BLEU allows researchers to assign numerical scores measuring the overlap between pairs of sentences, and Meta says its model produces an improvement of 44 percent in BLEU scores across supported languages (compared to previous state-of-the-art work). However, as is often the case in AI research, judging progress based on benchmarks requires context.

Although BLEU scores allow researchers to compare the relative progress of different machine translation models, they do not offer an absolute measure of software’s ability to produce human-quality translations.

Remember: Meta’s dataset consists of 3001 sentences, and each has been translated only by a single individual. This provides a baseline for judging translation quality, but the total expressive power of an entire language cannot be captured by such a small sliver of actual language. This problem is in no way limited to Meta — it’s something that affects all machine translation work, and is particularly acute when assessing low-resource languages — but it shows the scope of the challenges facing the field.

Christian Federmann, a principal research manager who works on machine translation at Microsoft, said the project as a whole was “commendable” in its desire to expand the scope of machine translation software to lesser-covered languages, but noted that BLEU scores by themselves can only provide a limited measure of output quality.

“Translation is a creative, generative process which may result in many different translations which are all equally good (or bad),” Federmann told The Verge. “It is impossible to provide general levels of ‘BLEU score goodness’ as they are dependent on the test set used, its reference quality, but also inherent properties of the language pair under investigation.”

Fan said that BLEU scores had also been complemented with human evaluation, and that this feedback was very positive, and also produced some surprising reactions.

“One really interesting phenomenon is that people who speak low-resource languages often have a lower bar for translation quality because they don’t have any other tool,” said Fan, who is herself a speaker of a low-resource language, Shanghainese. “They’re super generous, and so we actually have to go back and say ‘hey, no, you need to be very precise, and if you see an error, call it out.’”

The power imbalances of corporate AI

Working on AI translation is often presented as an unambiguous good, but creating this software comes with particular difficulties for speakers of low-resource languages. For some communities, the attention of Big Tech is simply unwelcome: they don’t want the tools needed to preserve their language in anyone’s hands but their own. For others, the issues are less existential, but more concerned with questions of quality and influence.

Some communities just don’t want Big Tech controlling their language

Meta’s engineers explored some of these questions by conducting interviews with 44 speakers of low-resource languages. These interviewees raised a number of positive and negative affects of opening up their languages to machine translation.

One positive, for example, is that such tools allow speakers to access more media and information. They can be used to translate rich resources, like English-language Wikipedia and educational texts. At the same time, though, if low-resource language speakers consume more media generated by speakers of better-supported languages, this could diminish the incentives to create such materials in their own language.

Balancing these issues is challenging, and the problems encountered even within this recent project show why. Meta’s researchers note, for example, that of the 44 low-resource language speakers they interviewed to explore these questions, the majority of these interviewees were “immigrants living in the US and Europe, and about a third of them identify as tech workers” — meaning their perspectives are likely different to those of their home communities and biased from the start.

Professor Fraser of LMU Munich said that despite this, the research was certainly conducted “in a way that is becoming more of involving native speakers” and that such efforts were “laudable.”

“Overall, I’m glad that Meta has been doing this.”

“Overall, I’m glad that Meta has been doing this. More of this from companies like Google, Meta, and Microsoft, all of whom have substantial work in low resource machine translation, is great for the world,” said Fraser. “And of course some of the thinking behind why and how to do this is coming out of academia as well, as well as the training of most of the listed researchers.”

Fan said Meta attempted to preempt many of these social challenges by broadening the expertise they consulted on the project. “I think when AI is developing it’s often very engineering — like, ‘Okay, where are my computer science PhDs? Let’s get together and build it just because we can.’ But actually, for this, we worked with linguists, sociologists, and ethicists,” she said. “And I think this kind of interdisciplinary approach focuses on the human problem. Like, who wants this technology to be built? How do they want it to be built? How are they going to use it?”

Just as important, says Fan, is the decision to open-source as many elements of the project as possible — from the model to the evaluation dataset and training code — which should help redress the power imbalance inherent in a corporation working on such an initiative. Meta also offers grants to researchers who want to contribute to such translation projects but are unable to finance their own projects.

“I think that’s really, really important, because it’s not like one company will be able to holistically solve the problem of machine translation,” said Fan. “It’s everyone — globally — and so we’re really interested in supporting these types of community efforts.”

Today’s Storystream

Feed refreshed An hour ago Dimorphos didn’t even see it coming

R
Twitter
Richard LawlerAn hour ago
A direct strike at 14,000 mph.

The Double Asteroid Redirection Test (DART) scored a hit on the asteroid Dimorphos, but as Mary Beth Griggs explains, the real science work is just beginning.

Now planetary scientists will wait to see how the impact changed the asteroid’s orbit, and to download pictures from DART’s LICIACube satellite which had a front-row seat to the crash.


M
The Verge
We’re about an hour away from a space crash.

At 7:14PM ET, a NASA spacecraft is going to smash into an asteroid! Coverage of the collision — called the Double Asteroid Redirection Test — is now live.


E
Twitter
Emma RothSep 26
There’s a surprise in the sky tonight.

Jupiter will be about 367 million miles away from Earth this evening. While that may seem like a long way, it’s the closest it’s been to our home planet since 1963.

During this time, Jupiter will be visible to the naked eye (but binoculars can help). You can check where and when you can get a glimpse of the gas giant from this website.


Asian America learns how to hit back

The desperate, confused, righteous campaign to stop Asian hate

Esther WangSep 26
E
Twitter
Emma RothSep 26
Missing classic Mario?

One fan, who goes by the name Metroid Mike 64 on Twitter, just built a full-on 2D Mario game inside Super Mario Maker 2 complete with 40 levels and eight worlds.

Looking at the gameplay shared on Twitter is enough to make me want to break out my SNES, or at least buy Super Mario Maker 2 so I can play this epic retro revamp.


R
External Link
Russell BrandomSep 26
The US might still force TikTok into a data security deal with Oracle.

The New York Times says the White House is still working on TikTok’s Trump-era data security deal, which has been in a weird limbo for nearly two years now. The terms are basically the same: Oracle plays babysitter but the app doesn’t get banned. Maybe it will happen now, though?


R
Youtube
Richard LawlerSep 26
Don’t miss this dive into Guillermo del Toro’s stop-motion Pinocchio flick.

Andrew Webster and Charles Pulliam-Moore covered Netflix’s Tudum reveals (yes, it’s going to keep using that brand name) over the weekend as the streamer showed off things that haven’t been canceled yet.

Beyond The Way of the Househusband season two news and timing information about two The Witcher projects, you should make time for this incredible behind-the-scenes video showing the process of making Pinocchio.


R
External Link
Russell BrandomSep 26
Edward Snowden has been granted Russian citizenship.

The NSA whistleblower has been living in Russia for the 9 years — first as a refugee, then on a series of temporary residency permits. He applied for Russian citizenship in November 2020, but has said he won’t renounce his status as a U.S. citizen.


E
External Link
Emma RothSep 26
Netflix’s gaming bet gets even bigger.

Even though fewer than one percent of Netflix subscribers have tried its mobile games, Netflix just opened up another studio in Finland after acquiring the Helsinki-based Next Games earlier this year.

The former vice president of Zynga Games, Marko Lastikka, will serve as the studio director. His track record includes working on SimCity BuildIt for EA and FarmVille 3.


A
External Link
Vietnam’s EV aspirant is giving big Potemkin village vibes

Idle equipment, absent workers, deserted villages, an empty swimming pool. VinFast is Vietnam’s answer to Tesla, with the goal of making 1 million EVs in the next 5-6 years to sell to customers US, Canada and Europe. With these lofty goals, the company invited a bunch of social media influencers, as well as some auto journalists, on a “a four-day, multicity extravaganza” that seemed more weird than convincing, according to Bloomberg.


J
James VincentSep 26
Today, 39 years ago, the world didn’t end.

And it’s thanks to one man: Stanislav Petrov, a USSR military officer who, on September 26th, 1983, took the decision not to launch a retaliatory nuclear attack against the US. Petrov correctly guessed that satellite readings showing inbound nukes were faulty, and so likely saved the world from nuclear war. As journalist Tom Chivers put it on Twitter, “Happy Stanislav Petrov Day to those who celebrate!” Read more about Petrov’s life here.


Soviet Colonel who prevented 1983 nuclear response
Photo by Scott Peterson/Getty Images
J
The Verge
James VincentSep 26
Deepfakes were made for Disney.

You might have seen the news this weekend that the voice of James Earl Jones is being cloned using AI so his performance as Darth Vader in Star Wars can live on forever.

Reading the story, it struck me how perfect deepfakes are for Disney — a company that profits from original characters, fans' nostalgia, and an uncanny ability to twist copyright law to its liking. And now, with deepfakes, Disney’s most iconic performances will live on forever, ensuring the magic never dies.


E
External Link
Hurricane Fiona ratcheted up tensions about crypto bros in Puerto Rico.

“An official emergency has been declared, which means in the tax program, your physical presence time is suspended,” a crypto investor posted on TikTok. “So I am headed out of the island.” Perhaps predictably, locals are furious.


R
The Verge
Richard LawlerSep 26
Teen hacking suspect linked to GTA 6 leak and Uber security breach charged in London.

City of London police tweeted Saturday that the teenager arrested on suspicion of hacking has been charged with “two counts of breach of bail conditions and two counts of computer misuse.”

They haven’t confirmed any connection with the GTA 6 leak or Uber hack, but the details line up with those incidents, as well as a suspect arrested this spring for the Lapsus$ breaches.