Skip to main content

The next challenge for facial recognition is identifying people whose faces are covered

The next challenge for facial recognition is identifying people whose faces are covered

/

Current methods are unreliable, but progress is being made — and quickly

Share this story

If you buy something from a Verge link, Vox Media may earn a commission. See our ethics statement.

Anonymous Hold Their Annual Million Mask March On Bonfire Night
Photo by Jack Taylor/Getty Images

Facial recognition is becoming more and more common, but ask anyone how to avoid it and they’ll say: easy, just wear a mask. In the future, though, that might not be enough. Facial recognition technology is under development that’s capable of identifying someone even if their face is covered up — and it could mean that staying anonymous in public will be harder than ever before.

The topic was raised this week after research published on the preprint server arXiv describing just such a system was shared in a popular AI newsletter. Using deep learning and a dataset of pictures of people wearing various disguises, researchers were able to train a neural network that could potentially identify masked faces with some reliability. Academic and sociologist Zeynep Tufekci shared the work on Twitter, noting that such technology could become a tool of oppression, with authoritarian states using it to identify anonymous protestors and stifle dissent.

The paper itself needs to be taken with a pinch of salt, though. Its results were far less accurate than industry-level standards (when someone was wearing a cap, sunglasses, and a scarf, for example, the system could only identify them 55 percent of the time); it used a small dataset; and experts in the field have criticized its methodology.

“It doesn’t strike me as a particularly convincing paper,” Patrik Huber, a researcher at the University of Surrey who specializes in face tracking and analysis, told The Verge. He pointed out that the system doesn’t actually match disguised faces to mugshots or portraits, but instead used something called “facial keypoints” (the distances between facial features like eyes, noses, lips, etc) as a proxy for someone’s identity.

An image from the recent study, showing how the neural networks estimate “facial keypoints” even when the face is covered.
An image from the recent study, showing how the neural networks estimate “facial keypoints” even when the face is covered.

But although the paper has its flaws, the challenge of recognizing people when their faces are covered is one that plenty of teams are working on — and making quick progress.

Facebook, for example, has trained neural networks that can recognize people based on characteristics like hair, body shape, and posture. Facial recognition systems that work on portions of the face have also been developed (although, again; not ready for commercial use). And there are other, more exotic methods to identify people. AI-powered gait analysis, for example, can recognize individuals with a high degree of accuracy, and even works with low-resolution footage — the sort you might get from a CCTV camera.

One system for identifying masked individuals developed at the University of Basel in Switzerland recreates a 3D model of the target’s face based on what it can see. Bernhard Egger, one of the scientists behind the work, told The Verge that he expected “lots of development” in this area in the near future, but thought that there would always be ways to fool the machine. “Maybe machines will outperform humans on very specific tasks with partial occlusions,” said Egger. “But, I believe, it will still be possible to not be recognized if you want to avoid this.”

There are ways to trick these systems — wearing a rigid, full-face mask, for example

Wearing a rigid mask that covers the whole face, for example, would give current facial recognition systems nothing to go on. And other researchers have developed patterned glasses that are specially designed to trick and confuse AI facial recognition systems. Getting clear pictures is also difficult. Egger points out that we’re used to facial recognition performing quickly and accurately, but that’s in situations where the subject is compliant — scanning their face with a phone, for example, or at a border checkpoint.

Privacy advocates, though, say even if these systems have flaws, they’re still likely to be embraced by law enforcement. Last month, for example, police in London used real-time facial recognition to scan people attending the annual Notting Hill Carnival. Before the event they assembled a “bespoke dataset” with images of more than 500 people who were either banned from attending or wanted for arrest and then set up cameras at one of the Carnival’s main thoroughfares. According to a report from human rights group Liberty, only one attendee was successfully identified using this system (and even then his arrest warrant was out-of-date) while there were 35 false positives. The police still deemed it a success.

If you combine this attitude with the increasing adoption of police body cameras, the growth of facial recognition databases, and new AI techniques for analyzing data, it seems clear that public anonymity is being undermined. And in the current political climate, where protests are becoming more common and more violent, this is potentially very dangerous. And as Tufekci noted on Twitter, this new technology is often developed without considering the uses it might be put to.

Amarjot Singh, the lead researcher behind the recent paper published on arXiv, said he thought the systems themselves were neutral, and whether they would have a harmful effect on society depended on how they were deployed. “There are more benefits to this technology than harm,” he told The Verge. “Everything can be used in a good way and a negative way. Even a car.” He added that he and his colleagues were working to get funding to improve their system, and that they might eventually commercialize it. “To expand the dataset we might try and make a product out of it,” said Singh. “We’re not very sure of that yet, but we will definitely be expanding the dataset.”

Today’s Storystream

Feed refreshed An hour ago The tablet didn’t call that play by itself

E
Twitter
Emma RothAn hour ago
Missing classic Mario?

One fan, who goes by the name Metroid Mike 64 on Twitter, just built a full-on 2D Mario game inside Super Mario Maker 2 complete with 40 levels and eight worlds.

Looking at the gameplay shared on Twitter is enough to make me want to break out my SNES, or at least buy Super Mario Maker 2 so I can play this epic retro revamp.


R
External Link
Russell BrandomAn hour ago
The US might still force TikTok into a data security deal with Oracle.

The New York Times says the White House is still working on TikTok’s Trump-era data security deal, which has been in a weird limbo for nearly two years now. The terms are basically the same: Oracle plays babysitter but the app doesn’t get banned. Maybe it will happen now, though?


Asian America learns how to hit back

The desperate, confused, righteous campaign to stop Asian hate

Esther Wang12:00 PM UTC
R
Youtube
Richard LawlerTwo hours ago
Don’t miss this dive into Guillermo del Toro’s stop-motion Pinocchio flick.

Andrew Webster and Charles Pulliam-Moore covered Netflix’s Tudum reveals (yes, it’s going to keep using that brand name) over the weekend as the streamer showed off things that haven’t been canceled yet.

Beyond The Way of the Househusband season two news and timing information about two The Witcher projects, you should make time for this incredible behind-the-scenes video showing the process of making Pinocchio.


E
External Link
Emma Roth4:13 PM UTC
Netflix’s gaming bet gets even bigger.

Even though fewer than one percent of Netflix subscribers have tried its mobile games, Netflix just opened up another studio in Finland after acquiring the Helsinki-based Next Games earlier this year.

The former vice president of Zynga Games, Marko Lastikka, will serve as the studio director. His track record includes working on SimCity BuildIt for EA and FarmVille 3.


A
External Link
Andrew J. Hawkins3:37 PM UTC
Vietnam’s EV aspirant is giving big Potemkin village vibes

Idle equipment, absent workers, deserted villages, an empty swimming pool. VinFast is Vietnam’s answer to Tesla, with the goal of making 1 million EVs in the next 5-6 years to sell to customers US, Canada and Europe. With these lofty goals, the company invited a bunch of social media influencers, as well as some auto journalists, on a “a four-day, multicity extravaganza” that seemed more weird than convincing, according to Bloomberg.


J
James Vincent3:17 PM UTC
Today, 39 years ago, the world didn’t end.

And it’s thanks to one man: Stanislav Petrov, a USSR military officer who, on September 26th, 1983, took the decision not to launch a retaliatory nuclear attack against the US. Petrov correctly guessed that satellite readings showing inbound nukes were faulty, and so likely saved the world from nuclear war. As journalist Tom Chivers put it on Twitter, “Happy Stanislav Petrov Day to those who celebrate!” Read more about Petrov’s life here.


Soviet Colonel who prevented 1983 nuclear response
Photo by Scott Peterson/Getty Images
J
The Verge
James Vincent3:03 PM UTC
Deepfakes were made for Disney.

You might have seen the news this weekend that the voice of James Earl Jones is being cloned using AI so his performance as Darth Vader in Star Wars can live on forever.

Reading the story, it struck me how perfect deepfakes are for Disney — a company that profits from original characters, fans' nostalgia, and an uncanny ability to twist copyright law to its liking. And now, with deepfakes, Disney’s most iconic performances will live on forever, ensuring the magic never dies.


E
External Link
Elizabeth Lopatto2:41 PM UTC
Hurricane Fiona ratcheted up tensions about crypto bros in Puerto Rico.

“An official emergency has been declared, which means in the tax program, your physical presence time is suspended,” a crypto investor posted on TikTok. “So I am headed out of the island.” Perhaps predictably, locals are furious.


R
The Verge
Richard Lawler2:09 PM UTC
Teen hacking suspect linked to GTA 6 leak and Uber security breach charged in London.

City of London police tweeted Saturday that the teenager arrested on suspicion of hacking has been charged with “two counts of breach of bail conditions and two counts of computer misuse.”

They haven’t confirmed any connection with the GTA 6 leak or Uber hack, but the details line up with those incidents, as well as a suspect arrested this spring for the Lapsus$ breaches.


R
The Verge
Richard Lawler1:00 PM UTC
Green light.

Good morning to everyone, except for the intern or whoever prevented us from seeing how Microsoft’s Surface held up to yet another violent NFL incident.

Today’s big event is the crash of a NASA spaceship this evening — on purpose. Mary Beth Griggs can explain.


D
David Pierce12:54 PM UTC
Thousands and thousands of reasons people love Android.

“Android fans, what are the primary reasons why you will never ever switch to an iPhone?” That question led to almost 30,000 comments so far, and was for a while the most popular thing on Reddit. It’s a totally fascinating peek into the platform wars, and I’ve spent way too much time reading through it. I also laughed hard at “I can turn my text bubbles to any color I like.”