Skip to main content

Google is releasing an open source harassment filter for journalists

Google is releasing an open source harassment filter for journalists

/

Starting with Thomson Reuters Foundation reporters in June

Share this story

Illustration by Alex Castro / The Verge

Google’s Jigsaw unit is releasing the code for an open source anti-harassment tool called Harassment Manager. The tool, intended for journalists and other public figures, employs Jigsaw’s Perspective API to let users sort through potentially abusive comments on social media platforms starting with Twitter. It’s debuting as source code for developers to build on, then being launched as a functional application for Thomson Reuters Foundation journalists in June.

Harassment Manager can currently work with Twitter’s API to combine moderation options — like hiding tweet replies and muting or blocking accounts — with a bulk filtering and reporting system. Perspective checks messages’ language for levels of “toxicity” based on elements like threats, insults, and profanity. It sorts messages into queues on a dashboard, where users can address them in batches rather than individually through Twitter’s default moderation tools. They can choose to blur the text of the messages while they’re doing it, so they don’t need to read each one, and they can search for keywords in addition to using the automatically generated queues.

A picture of the Harassment Manager dashboard as described in the post
Google

Harassment Manager also lets users download a standalone report containing abusive messages; this creates a paper trail for their employer or, in the case of illegal content like direct threats, law enforcement. For now, however, there’s not a standalone application that users can download. Instead, developers can freely build apps that incorporate its functionality and services using it will be launched by partners like the Thomson Reuters Foundation.

Jigsaw announced Harassment Manager on International Women’s Day, and it framed the tool as particularly relevant to female journalists who face gender-based abuse, highlighting input from “journalists and activists with large Twitter presences” as well as nonprofits like the International Women’s Media Foundation and the Committee To Protect Journalists. In a Medium post, the team says it’s hoping developers can tailor it for other at-risk social media users. “Our hope is that this technology provides a resource for people who are facing harassment online, especially female journalists, activists, politicians and other public figures, who deal with disproportionately high toxicity online,” the post reads.

A screenshot of the reporting option in Jigsaw’s Harassment Manager

Google has harnessed Perspective for automated moderation before. In 2019 it released a browser extension called Tune that let social media users avoid seeing messages with a high chance of being toxic, and it’s been used by many commenting platforms (including Vox Media’s Coral) to supplement human moderation. But as we noted around the release of Perspective and Tune, the language analysis model has historically been far from perfect. It sometimes misclassifies satirical content or fails to detect abusive messages, and Jigsaw-style AI can inadvertently associate terms like “blind” or “deaf” — which aren’t necessarily negative — with toxicity. Jigsaw itself has also been criticized for a toxic workplace culture, although Google has disputed the claims.

Unlike AI-powered moderation on services like Twitter and Instagram, however, Harassment Manager isn’t a platform-side moderation feature. It’s apparently a sorting tool for helping manage the sometimes overwhelming scale of social media feedback, something that could be relevant for people far outside the realm of journalism — even if they can’t use it for now.

Today’s Storystream

Feed refreshed 32 minutes ago The tablet didn’t call that play by itself

R
The Verge
Richard Lawler32 minutes ago
Teen hacking suspect linked to GTA 6 leak and Uber security breach charged in London.

City of London police tweeted Saturday that the teenager arrested on suspicion of hacking has been charged with “two counts of breach of bail conditions and two counts of computer misuse.”

They haven’t confirmed any connection with the GTA 6 leak or Uber hack, but the details line up with those incidents, as well as a suspect arrested this spring for the Lapsus$ breaches.


R
The Verge
Richard LawlerTwo hours ago
Green light.

Good morning to everyone, except for the intern or whoever prevented us from seeing how Microsoft’s Surface held up to yet another violent NFL incident.

Today’s big event is the crash of a NASA spaceship this evening — on purpose. Mary Beth Griggs can explain.


D
David PierceTwo hours ago
Thousands and thousands of reasons people love Android.

“Android fans, what are the primary reasons why you will never ever switch to an iPhone?” That question led to almost 30,000 comments so far, and was for a while the most popular thing on Reddit. It’s a totally fascinating peek into the platform wars, and I’ve spent way too much time reading through it. I also laughed hard at “I can turn my text bubbles to any color I like.”


Welcome to the new Verge

Revolutionizing the media with blog posts

Nilay PatelSep 13
T
Thomas Ricker10:44 AM UTC
The Simpsons pays tribute to Chrome’s dino game.

Season 34 of The Simpsons kicked off on Sunday night with an opening credits “couch gag” based on the offline dino game from Google’s Chrome browser. Cactus, cactus, couch, d’oh! Perfect.


T
Youtube
Thomas Ricker7:29 AM UTC
Table breaks before Apple Watch Ultra’s sapphire glass.

”It’s the most rugged and capable Apple Watch yet,” said Apple at the launch of the Apple Watch Ultra (read The Verge review here). YouTuber TechRax put that claim to the test with a series of drop, scratch, and hammer tests. Takeaways: the titanium case will scratch with enough abuse, and that flat sapphire front crystal is tough — tougher than the table which cracks before the Ultra fails — but not indestructible.


E
Twitter
Emma RothSep 25
Rihanna’s headlining the Super Bowl Halftime Show.

Apple Music’s set to sponsor the Halftime Show next February, and it’s starting out strong with a performance from Rihanna. I honestly can’t remember which company sponsored the Halftime Show before Pepsi, so it’ll be nice to see how Apple handles the show for Super Bowl LVII.


E
Twitter
Emma RothSep 25
Starlink is growing.

The Elon Musk-owned satellite internet service, which covers all seven continents including Antarctica, has now made over 1 million user terminals. Musk has big plans for the service, which he hopes to expand to cruise ships, planes, and even school buses.

Musk recently said he’ll sidestep sanctions to activate the service in Iran, where the government put restrictions on communications due to mass protests. He followed through on his promise to bring Starlink to Ukraine at the start of Russia’s invasion, so we’ll have to wait and see if he manages to bring the service to Iran as well.


E
External Link
Emma RothSep 25
We might not get another Apple event this year.

While Apple was initially expected to hold an event to launch its rumored M2-equipped Macs and iPads in October, Bloomberg’s Mark Gurman predicts Apple will announce its new devices in a series of press releases, website updates, and media briefings instead.

I know that it probably takes a lot of work to put these polished events together, but if Apple does pass on it this year, I will kind of miss vibing to the livestream’s music and seeing all the new products get presented.


E
External Link
Emma RothSep 24
California Governor Gavin Newsom vetoes the state’s “BitLicense” law.

The bill, called the Digital Financial Assets Law, would establish a regulatory framework for companies that transact with cryptocurrency in the state, similar to New York’s BitLicense system. In a statement, Newsom says it’s “premature to lock a licensing structure” and that implementing such a program is a “costly undertaking:”

A more flexible approach is needed to ensure regulatory oversight can keep up with rapidly evolving technology and use cases, and is tailored with the proper tools to address trends and mitigate consumer harm.