Whistleblower, activist, and Senate candidate Chelsea Manning spoke extensively at SXSW about the dangers of unchecked data collection and misplaced trust in algorithms. “The algorithms that I worked on in Iraq have found their way into policing, and also into the way the corporate world works, whether it’s your credit report or advertising data,” said Manning, who was released from prison last May after former President Barack Obama commuted her 35-year sentence for leaking classified intelligence. “All these different tools that we saw being used in one context have found their way everywhere else.”
In a conversation with Sally Singer of Vogue, Manning compared her work on predictive analysis in the Army a decade ago to how she fears modern programmers have approached artificial intelligence. “The idea of using algorithms in government, and in making decisions about credit reporting, for instance, is that it’s ‘better.’ That if we just write a better algorithm, more accurate algorithm, if I just math the crap out of this problem ... ‘If I just math it really well, I can problem-solve.’ And I came into Iraq with that mindset,” she said. “The algorithms themselves are not unbiased. we put our biases in there when we write it. And we also feed it data that might be biased to begin with.”
Manning was echoing a common critique of artificial intelligence systems, and one that she’s made before. Facial recognition algorithms largely trained on white faces fail to recognize dark skin, for instance, and predictive policing input is skewed by departments heavily policing certain neighborhoods. Manning called for an ethical framework that would govern software development. “We as technologists and as developers, especially those of us that work on systems that affect millions of people — and yes, I’m talking about the Twitter algorithms, the Google algorithms, as well as predictive policing — we need to be aware of the consequences of what we’re making,” she said. “Like doctors have a code of ethics, software developers should have a code of ethics.”
Part of the problem, Manning said, was the collection of huge amounts of information that can be repurposed over time. “I operate at paranoia levels of security. That said, what I advise people the most is: be self-aware of the information you’re putting out there.” That includes information that’s technically voluntary, like allowing location tracking by phone apps, as well as data that’s incredibly difficult to keep private, like purchase histories that can be sold to advertisers.
But she’s also upset by what she sees as a lack of action. “Our complacency is what’s dangerous,” she said. “People have known for at least four years, five years, now that surveillance is bad, and that it’s happening on a large scale [...] We know it’s a problem. But I think everybody’s expecting somebody else to do something about it.” While Manning agreed not to talk about her political candidacy at SXSW, that’s implicitly why she’s running for Senate in Maryland. Congress voted earlier this year to reauthorize controversial NSA surveillance powers under Section 702, although Manning’s Democratic opponent Benjamin Cardin did vote against the measure.
Despite her self-described information paranoia, Manning keeps an active public presence on Twitter, though she says she only uses it on “a secure laptop” and tries to avoid getting caught up in negative feedback. “I wanted people to know: I’m not afraid of being a former prisoner. I’m not afraid of being a trans person. I’m not afraid of being who I am,” she said. “It was really an act of defiance.”
Manning is just one of many people in the technology world who’s worried about artificial intelligence promoting bias, surveillance, and division — as opposed to the apocalyptic scenarios feared by fellow SXSW speaker Elon Musk. But as someone who was largely isolated from the outside world during her imprisonment, she’s in a unique position to observe the situation.
“I was in my own little world, fighting to get out, even though I knew that things were happening outside,” she said. “Now that I’m out here, I’m realizing, oh wow, we really have the militarization of police, the styles of policing, and the rhetoric of politics — everything’s changed. It’s become so much darker and so much scarier. And really, this has been an ongoing process. This has been decades in the making. It’s not an aberration.”
Update 10:30PM ET: Added context about Manning’s SXSW interviewer.