The American Civil Liberties Union tested Amazon’s facial recognition system — and the results were not good. To test the system’s accuracy, the ACLU scanned the faces of all 535 members of congress against 25,000 public mugshots, using Amazon’s open Rekognition API. None of the members of Congress were in the mugshot lineup, but Amazon’s system generated 28 false matches, a finding that the ACLU says raises serious concerns about Rekognition’s use by police.
“An identification — whether accurate or not — could cost people their freedom or even their lives,” the group said in an accompanying statement. “Congress must take these threats seriously, hit the brakes, and enact a moratorium on law enforcement use of face recognition.”
Reached by The Verge, an Amazon spokesperson attributed the results to poor calibration. The ACLU’s tests were performed using Rekognition’s default confidence threshold of 80 percent — but Amazon says it recommends at least a 95 percent threshold for law enforcement applications where a false ID might have more significant consequences.
“An identification... could cost people their freedom”
“While 80% confidence is an acceptable threshold for photos of hot dogs, chairs, animals, or other social media use cases,” the representative said, “it wouldn’t be appropriate for identifying individuals with a reasonable level of certainty.” Still, Rekognition does not enforce that recommendation during the setup process, and there’s nothing to prevent law enforcement agencies from using the default setting.
Amazon’s Rekognition came to prominence in May, when an ACLU report showed the system being used by a number of law enforcement agencies, including a real-time recognition pilot by Orlando police. Sold as part of Amazon’s Web Services cloud offering, the software was extremely inexpensive, often costing less than $12 a month for an entire department. The Orlando pilot has since expired, although the department continues to use the system.
The ACLU’s latest experiment was designed with a particular eye towards Rekognition’s partnership with the Washington County Sheriff’s Department in Oregon, where images were compared against a database of as many as 300,000 mug shots.
“It’s not hypothetical,” says Jacob Snow, who organized the test for the ACLU of Northern California. “This is a situation where Rekognition is already being used.”
“It’s not hypothetical”
The test also showed indications of racial bias, a long-standing problem for many facial recognition systems. 11 of the 28 false matches misidentified people of color (roughly 39 percent), including civil-rights leader Rep. John Lewis (D-GA) and five other members of the Congressional Black Caucus. Only twenty percent of current members of Congress are people of color, which indicates that false-match rates affected members of color at a significantly higher rate. That finding echoes disparities found by NIST’s Facial Recognition Vendor Test, which has shown consistently higher error rates for facial recognition tests on women and African-Americans.
Running faces against a database with no matches might seem like a recipe for failure, but it’s similar to the conditions that existing facial recognition systems face every day. The system used by London’s Metropolitan Police produces as many as 49 false matches for every hit, requiring police to sort through the false-positives manually. What’s more significant is the rate at which the false positives cropped up in the Rekognition tests, with more than five percent of the subject group triggering a false match of some kind.
In practice, most facial recognition IDs would be confirmed by a human before they led to anything as concrete as an arrest — but critics say even checking a person’s identity can do damage. “Imagine a police officer getting a false match for somebody with a concealed weapon arrest,” says Snow. “There’s a real danger if that information is surfaced to the officer during a stop. It’s not hard to imagine it turning violent.”
The test also raises concerns over how easily Rekognition can be deployed without oversight. All the ACLU’s data was collected from publicly available sources, including the 25,000 mug shots. (The organization declined to name the specific source for privacy reasons, but many states treat mug shots as public records.) Amazon’s system is also significantly cheaper than non-cloud-based offerings, charging the ACLU only $12.33 for the tests.
The test has already inspired significant reaction from three members of Congress. Shortly after the test was published, Sen. Markey (D-MA), Rep. Gutiérrez (D-IL) and Rep. DeSaulnier (D-CA) signed onto an open letter to Amazon CEO Jeff Bezos asking for a full list of law enforcement agencies using the technology and inquiring about safeguards for using it on children younger than thirteen.
“Serious concerns have been raised about the dangers facial recognition can pose to privacy and civil rights,” the letter reads, “especially when it is used as a tool of government surveillance.”
Update 2:59PM ET: Updated with letter to Amazon.