Skip to main content

Chicago’s predictive policing tool just failed a major test

Chicago’s predictive policing tool just failed a major test


A RAND report shows that the ‘Strategic Subject List’ doesn’t reduce homicides

Share this story

Dylan Lathrop

Struggling to reduce its high murder rate, the city of Chicago has become an incubator for experimental policing techniques. Community policing, stop and frisk, "interruption" tactics — the city has tried many strategies. Perhaps most controversial and promising has been the city’s futuristic "heat list" — an algorithm-generated list identifying people most likely to be involved in a shooting.

The hope was that the list would allow police to provide social services to people in danger, while also preventing likely shooters from picking up a gun. But a new report from the RAND Corporation shows nothing of the sort has happened. Instead, it indicates that the list is, at best, not even as effective as a most wanted list. At worst, it unnecessarily targets people for police attention, creating a new form of profiling.

It unnecessarily targets people for police attention

Funded through a $2 million grant from the National Institute of Justice, the list’s algorithm identifies people by looking not only at arrests, but also whether someone is socially connected with a known shooter or shooting victim. The program also has a kind of pre-crime feature in which police visit people on the list before any crime has been committed.

One of the list’s most promising aspects was that it wasn’t just a police officer who would visit. Social workers would show up, too — employees of the Chicago Violence Reduction Strategy group at John Jay College. The list was designed to let Chicago police engage with at-risk (and potentially dangerous) citizens, but also to provide social services, such as access to counseling, to people who were in danger.

"We want to show them the carrot and the stick," said Christopher Mallette, executive director of the Chicago Violence Reduction Strategy group, in a conversation with The Verge last year. "We want them to know they can get help — but we also want them to know that if they don’t keep in line, there’s a jail cell waiting for them."

"We want to show them the carrot and the stick."

CPD wasn’t shy about touting the importance of the list, later rebranded as the Strategic Subjects List, or SSL. In 2014, the CPD official in charge of the program, Commander Jonathan Lewin, told The Verge: "This will inform police departments around the country and around the world on how best to utilize predictive policing to solve problems. This is about saving lives."

But the study from RAND, which was granted extraordinary access to CPD when it launched the list in 2013, found that the program has saved no lives at all. The RAND researchers were allowed to view the list, sit in on internal meetings, and generally observe how the tool was being used. They discovered that CPD wasn’t using the list as a way to provide social services; instead, CPD was using it as a way to target people for arrest.

"The individuals on the SSL were considered to be ‘persons of interest’ to the CPD," according to the report. "Overall ... there was no practical direction about what to do with individuals on the SSL, little executive or administrative attention paid to the pilot, and little to no follow-up with district commanders."

CPD was using it as a way to target people for arrest.

John S. Hollywood, one of the report’s authors, explained to The Verge that there were as many as 11 different violence reduction initiatives going on within CPD at the time the list was being rolled out. "The list just got lost," he said.

It was no surprise, then, that when Hollywood and his colleagues compiled data to figure out whether the list changed the city’s murder rate or reduced the likelihood that someone on the list might be involved in a shooting, they found it made no significant difference.

"[A]t-risk individuals were not more or less likely to become victims of a homicide or shooting as a result of the SSL, and this is further supported by city-level analysis finding no effect on the city homicide trend," according to the report.

Instead of being used to prevent violence, the list essentially served as a way to find suspects after the fact. "We do find, however, that SSL subjects were more likely to be arrested for a shooting," the report said.

The list essentially served as a way to find suspects after the fact

CPD’s Lewin declined to comment about the report, but CPD issued a press release in response. It stressed that RAND "evaluated a very early version" of the list, "which has since evolved greatly and has been fully integrated with the Department’s management accountability process." It also points out that "the prediction model discussed in the report is the very early, initial model (Version 1), developed in August, 2012. We are now using Version 5, which is significantly improved."

Hollywood agreed that the list was in an early stage when it was evaluated, and that it’s possible that it has improved. (CPD has agreed to allow RAND researchers to evaluate an updated version of the list, Hollywood said.)

The RAND report is significant, however, as a rare look at the effectiveness of a major predictive policing tool that was touted as the future of policing — and may instead be a failed experiment.

"Creating a data-driven ‘most-wanted’ list misses the value."

Andrew G. Ferguson, a law professor and predictive policing expert at the University of the District of Columbia, summarized the problems identified in the RAND report.

"Just creating a data-driven ‘most-wanted’ list misses the value of big data prediction," Ferguson said in an email. "The ability to identify and proactively intervene in the lives of at risk youth is a positive, but you have to commit to the intervention piece."

"Just directing police toward those individuals for traditional policing is not enough," he said.