Skip to main content

AI picks up racial and gender biases when learning from what humans write

AI picks up racial and gender biases when learning from what humans write

/

There is no objectivity

Share this story

Artificial intelligence picks up racial and gender biases when learning language from text, researchers say. Without any supervision, a machine learning algorithm learns to associate female names more with family words than career words, and black names as being more unpleasant than white names.

For a study published today in Science, researchers tested the bias of a common AI model, and then matched the results against a well-known psychological test that measures bias in humans. The team replicated in the algorithm all the psychological biases they tested, according to study co-author Aylin Caliskan, a post-doc at Princeton University. Because machine learning algorithms are so common, influencing everything from translation to scanning names on resumes, this research shows that the biases are pervasive, too.

“Language is a bridge to ideas, and a lot of algorithms are built on language in the real world,” says Megan Garcia, the director of New America’s California branch who has written about this so-called algorithmic bias. “So unless an alg is making a decision based only on numbers, this finding is going to be important.”

We can’t take for granted any kind of objectivity, ever

An algorithm is a set of instructions that humans write to help computers learn. Think of it like a recipe, says Zachary Lipton, an AI researcher at UC San Diego who was not involved in the study. Because algorithms use existing materials — like books or text on the internet — it’s obvious that AI can pick up biases if the materials themselves are biased. (For example, Google Photos tagged black users as gorillas.) We’ve known for a while, for instance, that language algorithms learn to associate the word “man” with “professor” and the word “woman” with “assistant professor.” But this paper is interesting because it incorporates previous work done in psychology on human biases, Lipton says.

For today’s study, Caliskan’s team created a test that resembles the Implicit Association Test, which is commonly used in psychology to measure how biased people are (though there has been some controversy over its accuracy). In the IAT, subjects are presented with two images — say, a white man and a black man — and words like “pleasant” or “unpleasant.” The IAT calculates how quickly you match up “white man” and “pleasant” versus “black man” and “pleasant,” and vice versa. The idea is that the longer it takes you to match up two concepts, the more trouble you have associating them.

The test developed by the researchers also calculates bias, but instead of measuring “response time,” it measures the mathematical distance between two words. In other words, if there’s a bigger numerical distance between a black name and the concept of “pleasant” than a white name and “pleasant,” the model’s association between the two isn’t as strong. The further apart the words are, the less the algorithm associates them together.

Caliskan’s team then tested their method on one particular algorithm: Global Vectors for Word Representation (GLoVe) from Stanford University. GLoVe basically crawls the web to find data and learns associations between billions of words. The researchers found that, in GLoVe, female words are more associated with arts than with math or science, and black names are seen as more unpleasant than white names. That doesn’t mean there’s anything wrong with the AI system, per se, or how the AI is learning — there’s something wrong with the material.

“AI is biased because it reflects effects about culture and the world and language,” says Caliskan. “So whenever you train a model on historical human data, you will end up inviting whatever that data carries, which might be biases or stereotypes as well.” One limitation is that the team is only able to calculate the bias associated with single words. Next, they’re interested in expanding to multiple words or phrases, and also in other languages.

The solution is not necessarily to change the model, according to Caliskan. AI just captures the world as it is — it just happens that our world is full of bias. Changing the way the algorithm works would make it less effective. Withholding information might not work anyway, says Lipton. Even if you don’t tell the model if someone is male or female, you’re going to provide other information that is correlated to gender, and the model will probably figure it out. “We can’t take for granted any kind of objectivity, ever,” he says.

Instead, humans can work on the other end. For example, Google Translate can be tweaked to improve its translation. So if you see that “doctor” is always translated to “he” while a “nurse” is translated to “she,” you can suggest a change.

Garcia, the New America CA director, says that having a more diverse group of people working on building the algorithms might be helpful too. But in the meanwhile, she agrees that the first step is simply encouraging everyone to remember that there will always be biased unless it’s corrected: “Computer-generated bias is everywhere we look.”