Amazon’s facial recognition matched 28 members of Congress to criminal mugshots

Illustration by James Bareham / The Verge

The American Civil Liberties Union tested Amazon’s facial recognition system — and the results were not good. To test the system’s accuracy, the ACLU scanned the faces of all 535 members of congress against 25,000 public mugshots, using Amazon’s open Rekognition API. None of the members of Congress were in the mugshot lineup, but Amazon’s system generated 28 false matches, a finding that the ACLU says raises serious concerns about Rekognition’s use by police.

“An identification — whether accurate or not — could cost people their freedom or even their lives,” the group said in an accompanying statement. “Congress must take these threats seriously, hit the brakes, and enact a moratorium on law enforcement use of face recognition.”

Reached by The Verge, an Amazon spokesperson attributed the results to poor calibration. The ACLU’s tests were performed using Rekognition’s default confidence threshold of 80 percent — but Amazon says it recommends at least a 95 percent threshold for law enforcement applications where a false ID might have more significant consequences.

“While 80% confidence is an acceptable threshold for photos of hot dogs, chairs, animals, or other social media use cases,” the representative said, “it wouldn’t be appropriate for identifying individuals with a reasonable level of certainty.” Still, Rekognition does not enforce that recommendation during the setup process, and there’s nothing to prevent law enforcement agencies from using the default setting.

Amazon’s Rekognition came to prominence in May, when an ACLU report showed the system being used by a number of law enforcement agencies, including a real-time recognition pilot by Orlando police. Sold as part of Amazon’s Web Services cloud offering, the software was extremely inexpensive, often costing less than $12 a month for an entire department. The Orlando pilot has since expired, although the department continues to use the system.

The ACLU’s latest experiment was designed with a particular eye towards Rekognition’s partnership with the Washington County Sheriff’s Department in Oregon, where images were compared against a database of as many as 300,000 mug shots.

“It’s not hypothetical,” says Jacob Snow, who organized the test for the ACLU of Northern California. “This is a situation where Rekognition is already being used.”

The test also showed indications of racial bias, a long-standing problem for many facial recognition systems. 11 of the 28 false matches misidentified people of color (roughly 39 percent), including civil-rights leader Rep. John Lewis (D-GA) and five other members of the Congressional Black Caucus. Only twenty percent of current members of Congress are people of color, which indicates that false-match rates affected members of color at a significantly higher rate. That finding echoes disparities found by NIST’s Facial Recognition Vendor Test, which has shown consistently higher error rates for facial recognition tests on women and African-Americans.

Running faces against a database with no matches might seem like a recipe for failure, but it’s similar to the conditions that existing facial recognition systems face every day. The system used by London’s Metropolitan Police produces as many as 49 false matches for every hit, requiring police to sort through the false-positives manually. What’s more significant is the rate at which the false positives cropped up in the Rekognition tests, with more than five percent of the subject group triggering a false match of some kind.

In practice, most facial recognition IDs would be confirmed by a human before they led to anything as concrete as an arrest — but critics say even checking a person’s identity can do damage. “Imagine a police officer getting a false match for somebody with a concealed weapon arrest,” says Snow. “There’s a real danger if that information is surfaced to the officer during a stop. It’s not hard to imagine it turning violent.”

The test also raises concerns over how easily Rekognition can be deployed without oversight. All the ACLU’s data was collected from publicly available sources, including the 25,000 mug shots. (The organization declined to name the specific source for privacy reasons, but many states treat mug shots as public records.) Amazon’s system is also significantly cheaper than non-cloud-based offerings, charging the ACLU only $12.33 for the tests.

The test has already inspired significant reaction from three members of Congress. Shortly after the test was published, Sen. Markey (D-MA), Rep. Gutiérrez (D-IL) and Rep. DeSaulnier (D-CA) signed onto an open letter to Amazon CEO Jeff Bezos asking for a full list of law enforcement agencies using the technology and inquiring about safeguards for using it on children younger than thirteen.

“Serious concerns have been raised about the dangers facial recognition can pose to privacy and civil rights,” the letter reads, “especially when it is used as a tool of government surveillance.”

Update 2:59PM ET: Updated with letter to Amazon.

Comments

Only 28? It’s clearly not effective if it can’t even identify our highest profile white collar criminals. Back to the drawing board!

If developing a criminal identification system, one should begin by hard coding the faces of each member of congress and making sure they are properly ID’d as the criminals they are each and every time they appear.

You’re likely onto something here, but it’s going to be the total opposite. Exclusion lists are going to become commonplace.

I think it’s an entirely effective way to screen 25,000 mugshots for facial matching. When you use a human being for the final confirmation, you can quickly dismiss any false positives.

I agree that you can use a human to go in after and get rid of the false positives. I wonder what the false negative rate is.

Agreed. Between this and Amazon’s response to the 80% threshold, this really doesn’t warrant much concern for actual usage.

Mmm not sure if isn’t more a headline grabber than a real story. That it shouldn’t be able to be used as evidence is a given.
But using a low threshold is a surefire way to get false positives.
But if those kinds of algorithms are only used to speed up search or discard suspects why not ?
Is really not about what kind of Algo they use but more how they use it. This is were safeguards should be put, trying to blanket ban visual algorithms is just counter productive.

Concerning the racial "bias" I would be cautious with that term’s use. An algorithm is not biased the same way humans can be biased. They are not even biased like their creator. They reproduce "bias" in the data. The fact that their is less person of colour make it harder for the algorithms unless you put disproportionately more people of colour in the dataset. Which is in itself questionable (not everybody wants to be in a dataset).

tldr having an imperfect tool is ok as long as you don’t take it at face value. Who actually uses I’m feeling lucky on Google ?

Concerning the racial bias, yes, they should put more minorities into the data set because that will increase the accuracy of recognition for people of color. In the case of training this AI, or algorithm, whichever you please, the researchers need the same amount of data for all races, gender, etc. or else the data that will be accrued is just simply inaccurate.

It’s a goal to strive for but not really achievable :
Imagine that in China they are hundred of millions of Han people and compare that to Samoa or island with a few hundred thousand individual. To reach a good accuracy on the Chinese you would need so many people in the dataset that it would excess a reasonable sampling size for smaller countries. (If your dataset is a significant proportion of the country population you might reintroduce bias)

If it’s not achievable then it shouldn’t be used. the rate of false positives is too high to be reliable and that’s precisely the problem.

You’re not advised to use a higher threshold, and if you think cops will use discretion and nuance with this kind of thing I have a bridge to sell you in San Francisco.

Politicians being identified as criminals. Sounds like it is working perfectly

Scan and jail. Worry about false-positives later?

Just be prepared the next time you get pulled over and they think you have a fake license.

I’m sure after a few hours or days in jail they will figure it out and let you out.

I think it’s important to temper people’s expectations about tech. A lot of people think this stuff just "works".

Like self-driving: It’s going to be great when it works. We are a LONG way from there.

Ironic thing is that the Congressman they specifically call out, civil rights leader John Lewis, was arrested multiple times during the civil rights movement and also more recently as part of other protest movements, so he likely would have a mug shot on file somewhere, though presumably not in the database the ACLU used.

Its a feature, not bug!

Amazon recommends a 95% confidence threshold for law enforcement uses. That means out of 100 uses, 5 will can be expected to be wrong. That’s way, way too high of a failure rate when someone’s liberty is at stake.

That… is not how confidence thresholds work. All that says is the machine declares "I am 95% sure that picture matches one from the database". The number of false positives could be greater or lower than 5/100. It’s a subtle difference.

95% of the data points are within threshold vs 80%. It’s not a failure rate. It’s so that it’s scale able with any size data sample. That said, are we going to force shaving people down for mugshots at some point so that the AI can work better?

I like how it was necessary to specify that the sample of 25,000 mugshots did not include any of the members of congress.

FAKE NEWS! Your favorite congress did nothing wrong. This is just a witch hunt by the very, very, very bad people at AMAZON to distract you from the REAL criminals. AMAZON THEMSELVES!!!

…huh?

Poking fun at Trump. I’m 95% confident that’s what he would tweet. Down to the capitalization.

So, it works!

Gotta hand it to the ACLU – this is a fantastic demonstration to prove their point.

View All Comments
Back to top ↑