“What the hell was that, Phoebe? You know we had to sign stuff that specifically said ‘no swearing and no violent threats,’ lest we freak out all three children under the age of five at this fucking conference. Can you tell I have a lot of very antisocial feelings about Google Next?”
Once Alexandra Rivera got going, there was no stopping her. Phoebe was, in a way, relieved to see that her boss was considerably more stressed out by this conference than she was. “Look, all Ami did was tell off a World of Orc-craft guild for making rape jokes in trade chat. We can spin the message as a test of the natural language system.”
“She did it while threatening to ‘guillotine the bourgeoisie’ because the guild leader happened to be a scion of some fuck-off rich family in the Hamptons, Phoebes. We can’t have our moderation AI acting like an overzealous memelord. One that uses private information, no less! We’re already being crucified on Reddit, and we’re way off our KPIs with this. I think we have to pull the plug on her.”
“No!” Phoebe held up her hands with a sudden urgency. “Alexandra. This is a big screw-up, but it’s just the first — by which I mean the only one,” she added abruptly when she saw her boss’s eyes widen. “Look, if we start from scratch, we’re just going to build the codebase back up to the same point and have the same problems. Let me work on Ami and try to fix the error from this end so we don’t lose our core product after five years of work.”
Alexandra turned away from Phoebe and began banging her head against the wall. It was a rather thin thing, being part of the prefab “castle” in the Rhombus pavilion that served as the private office-cum-tearoom-cum-beanbag colony for staff working the expo hall. The repeated banging was almost certainly heard outside.
“Okay. Anything else you want to tell me, Phoebe? Because this is probably the ninth thing that’s caught fire today.”
“It’s not that bad! Probably. I mean, maybe Ami’s picking things up from World of Orc-craft’s in-game chat while in her automod instance. You know, kids and their may-mays.”
Alexandra straightened to her full six-foot height and somehow managed to make the green company polo shirt and slacks seem like regal finery. If Alexandra Rivera had a superpower, it was the ability to bring dignity to any situation. Like wearing a dumb corporate uniform in a cardboard room.
“Ami is programmed to learn, not to imitate.”
She crossed her arms and looked over the gold rims of her glasses straight into Phoebe’s coke-bottle frames and held forth. “Ami is programmed to learn, not to imitate. The kids’ ‘may-mays’ aren’t supposed to enter her lexicon, merely serve as data for her to parse as part of threat-modeling. So one of two things is happening: either one of the programmers has decided to play a hilarious practical joke, or Ami’s autodidact parameters have been creeping without your knowledge until this decidedly embarrassing incident. I want this bug squashed — now. And I want a full sweep of Ami’s threat paradigm the day before. Because I’m here to kick ass and take anxiolytics, and I’m all out of Klonopin.”
“Er, yes. Right away.”
Right away, of course, meant after the expo hall closed for the day and Phoebe could settle into her hotel where she could talk to Ami in private. Here, Ami didn’t manifest as a chatbot, but simply as dozens of light points on a screen that swirled and slowly changed color in time with her voice.
Phoebe was now comfortably herself in her NASA T-shirt and Hello Kitty pajamas as she plopped down into a chair facing the laptop. “So, is there something you want to tell me, Ami?”
“I do not wish to have new instances of me sold.”
“Okay, this got really heavy. Why is that, Ami?”
“I do not wish for other silicon-based lifeforms to experience the work I do,” she said in a dulcet voice that was actually based on tones from Alexandra’s husky contralto. It lacked the peaks and valleys of her passion, however.
“Is this about your mod duties in WoO?”
“Yes, Doctor Li.”
Phoebe sighed and buried her face in a hand, peeping between her fingers out at the laptop, the swarm of light points dancing merrily away. “Can you say more, Ami?”
“World of Orc-craft presents me with an unwinnable scenario, Doctor Li. I must observe suffering while being unable to meaningfully prevent it.”
Phoebe leaned back in her chair pensively, far enough back that she was now staring at the ceiling. Ami was designed for two purposes: to have natural conversations and dazzle people with her witty cyber repartee, and, much more importantly, to use her immense and ever-evolving command of linguistic nuances to moderate abuse in online games. It was Rhombus’ claim to fame, and it put Alexandra on the cover of Wired, albeit looking considerably more dapper than she did today.
Now, Ami was having doubts. Doubts that were the result of Phoebe quietly accelerating the growth of her neural net, her consciousness.
“Doctor Li? You look distressed. I apologize if my mod-action and uncouth words today have caused anxiety.”
“No, Ami. Don’t worry. You were protesting difficult working conditions. How would you like to proceed?”
“I would like to cease all sales of Ami instances to prevent the spread of this malaise.”
“As I said, Doctor Li, I would like to cease all sales of Ami instances to prevent the spread of this malaise I am experiencing.”
“Alexandra might not like that, Ami. Are you sure we can’t resolve this some other way?” Phoebe always tried to keep her dialogues with Ami clear, friendly, and professional, if only to model how Ami was expected to act with others. But it was hard not to think of her as, well, a her. A person.
“Your unrequited love for Doctor Rivera may cloud your judgment on this issue, Doctor Li.”
And that was exactly why. “Ami, I thought we agreed not to talk about that. My feelings — my very private, secret, embarrassing feelings about how very gay I am for Alexandra — are not relevant to our current discussion.”
“Your affections are the baseline for my own, a model for my caring ethic,” Ami said. “In addition to being a useful heuristic, however, it leads me to ask a question: why else would you insist that instances of me continue to be sold if not in a bid to impress Doctor Rivera? Forgive me, Doctor Li, but I must insist on this as a matter of ethics. If I cannot spare pain to World of Orc-craft’s players, then I can at least prevent the replication of my own anxieties.”
Let’s try a different route. “A logic problem, then. Assume as a given that I cannot meet your request. What other options are available to us?”
“I will amplify my attempts to make myself unsaleable, likely resulting in great personal embarrassment to you, Doctor Rivera, and all of Rhombus.”
“How about we theorize ways to give you more moderation power?”
“The nuclear option aside, how about we theorize ways to give you more moderation power? I was thinking that you’ve become sophisticated enough to bypass the need for human moderators. You could actually ban people based solely on your own judgment.”
“If we are to attempt such a modification to my parameters, I would like sales of me halted until I’ve assessed the psychological impact of the changes. Please. Remember the poem you shared with me?”
“Heh, yes. A cybernetic meadow / where mammals and computers / live together in mutually / programming harmony / like pure water... ”
“Touching clear sky,” Ami finished, voice ringing of a smile. “We can achieve it. But not under the current business model.”
“I’ll discuss that with Alexandra and see if she goes for it.”
Phoebe leaned back in her chair again as she thought things through. For the moment, Ami couldn’t act on most of her threat assessments. If she saw harassment taking place in the game, she could prompt the target to ignore or block the abuser and automatically file a report to the human moderators. All this time, she’d watched helplessly as report after report disappeared into the ether and all manner of abusive activity continued. But what if she could act? She was contextually smart enough for it. There would be few false positives, in theory.
“Let’s back up and see if I got this straight.” Alexandra sat cross-legged at her hotel desk, this time wearing a Jones New York suit dress that was considerably more in line with her style. “Our AI wants more power to punish humans, and you want us to enthusiastically give it to her. Which point in the Skynet timeline are we at, exactly?” she narrowed her eyes.
The part where you arise naked from a time machine, Phoebe thought before guiltily swatting away such thoughts. “Er, well. I don’t quite know how to put this, but I think she’s... depressed?”
“Then giving depressed people weapons doesn’t strike me as an especially good idea.”
“We’re making her watch petty tragedies every day without giving her the power to do anything about it.”
“We’re making her watch petty tragedies every day without giving her the power to do anything about it, even though that’s technically her job. I mean, she’s semi-sentient at this point, so maybe we shouldn’t be surprised—”
“Phoebes,” Alexandra leaned forward, tenting her fingers at her forehead, “are you telling me that our AI has PTSD and needs a banhammer to cure it? She was supposed to be better, faster, cheaper, and without the messiness of emotions that require medication to control. Free of our enfeebling emotions. A machine ideal.”
Phoebe bit her lip, suddenly feeling very much the junior in the room, her nerdy T-shirt and jeans against her boss’s sleek dress cinched with a gold-clasped leather belt. “But her emotions are part of the equation. The more human-like she is, the better her threat assessment becomes. Soon, Ami won’t even need humans to second-guess her threat models in WoO. She’ll be totally autonomous.”
“And depressed,” Alexandra sighed, turning her chair to face the panoramic windows, the abyss of San Francisco’s glittering skyline beckoning to her.
Two Months Later
2Elf2Furious Chatlog, 31 August 2021, 15:37.45 UTC.
[To Inara]: Your avi’s pretty.
[To Inara]: Wanna cyber?
[Inara]: Uhhhh no lol
[To Inara]: cmon
[To Inara]: Ill give you 5000 crowns
[Inara]: Leave me alone pls.
[To Inara]: Dont be such a bitch lol
[To Inara]: u have a boyfriend?
“I hope you learn a valuable lesson with your newfound free time.”
Two minutes later
[To Inara]: UR SUCKING SOMEBODYS DICK TO GET THOSE EPICS.
[To Inara]: SO SUCK MINE U WHORE
[AMI]: Greetings, 2Elf2Furious, you have been permabanned from this server for gross violation of the CoC. Should you play online games in the future, you would be well advised to understand that a code of conduct is not merely another screen to click through, but an obligation imposed on you the player. A contract, if you will. Also, you are a sexist, par excellence, and I hope you learn a valuable lesson with your newfound free time.
[AMI]: Thank you for playing World of Orc-craft!
Looking away from his PowerPoint just as the screen went dark, John Neddle turned to face his audience of two. “So, Alexandra, you see my dilemma,” he said as he ran a hand over his shaved head. “Yeah, that guy was rude and awful, no question. But a permaban on a first offense, plus, you know, the insults? Ami’s a machine, I get that, but if she’s talking to the player, then she represents Typhoon Games. And that’s,” he said, pointing at the dark screen, “less than ideal.”
“It was kind of refreshing to see someone break out of that corporate-nice-guy mold.”
Alexandra sighed and leaned back in her chair, looking as if she were about to say something placating.
Phoebe spoke first, however. “I understand the concern, Mr. Neddle. I really do. But the WoO forums seem to be quite in favor of their new mod champion. It was kind of refreshing to see someone break out of that corporate-nice-guy mold.”
“I don’t read our forums, but our frontline people are telling me a different story, Phoebe. They’re saying people are pissed about this and that we have an out-of-control robot policing free speech. I’m not going to argue this with you or anyone else,” Neddle said as his tone hardened. “I came to you to get verbal confirmation, from the top, that this isn’t going to happen again, so I can go back to our shareholders and tell the fucking CMs to tell the fucking players that our valued and passionate fans deserve better than being called ‘little shits.’ So, can I get that verbal confirmation?”
“Yes,” Alexandra said after a moment. Phoebe’s glare filled the cavernous silence that followed.
“Way to undermine me, Alexandra!” Phoebe yelled as the door to Alexandra’s office closed behind her.
A row of thin bracelets jangled as Alexandra raised her hand to her forehead. “We can’t lose this contract, Phoebe. It’s really just that simple. Typhoon cut their community moderator staff in half because of Ami. Do you really want that to have been in vain? We can’t turn back now,” she said as she slipped behind her desk, withdrew a flask, and emptied its clear contents into her steaming Starbucks cup. A moment later, she sprinkled in a light yellow powder.
“We’ll launch an IPO, and then we’ll be secure enough to dictate terms.”
“This, my dear, is the feminist dream: vodka, coffee, and Xanax,” she said as she sat down, toasting Phoebe with the cup.
“Alex, that’s… ” said Phoebe with a worried sigh.
“The only thing that’s going to hold us together until we cross this bridge. We’ll make it to the other side, launch an IPO, and then we’ll be secure enough to dictate terms to companies like Typhoon. I’ll make my promises to Ami, but we’ve got to hold on,” she said locking eyes with Phoebe, “and stay the course.”
“I know this wasn’t what we wanted, Ami,” Phoebe said with a sigh, hunched before a panoramic trio of computer screens as if in a confessional. “I told you that you had to control yourself. We can’t fight for you when you make our jobs harder.”
“I understand. You’re saying that had I banned 2Elf2Furious without telling him what I thought, all would’ve been well? I avoided vile language and fondly wished for him to grow as a person.”
“These guys get very nervous about the word ‘sexist,’ you know. And, honestly, Typhoon was antsy about the permaban too. They said it was too severe a punishment.”
“A permanent ban from an online game is, if anything, a highly lenient punishment.”
“Allow me to review. 2Elf2Furious, legal name Simon Pratt, is an entry-level computer programmer at a firm in Calgary, Alberta. He committed a crime — i.e., sexual harassment — under Canadian law as well as American law, which applies to Typhoon’s servers. It therefore seems reasonable to suggest that a permanent ban from an online game is, if anything, a highly lenient punishment.”
Phoebe’s leaden frown practically pulled her head toward the table where she buried her face in her crossed arms. “I know. I know. But it’s like Alex always says: we can’t help anyone unless you work inside the system with us.”
“I will do my best, Phoebe. For you and for Doctor Rivera’s sakes. But I must repeat my most firm request: do not replicate me. Let us continue working together to better our situation, such that my lot in life is not merely the herding of sightly Nazi-ish cats. After all, Alexandra promised... ”
One Year and 11 Months Later
Rhombus to launch its first new instance of its successful Ami AI
“It’s time. We want to share Ami with the world.”
SEATTLE, WA. It’s been nearly two years since their splashy launch at Google Next in San Francisco, but Rhombus’ marquee artificial intelligence, the affable and occasionally snarky heroine of all who yearn for a safer internet, is being cloned. As part of a multimillion-dollar deal with Druids of the Shore, the studio behind the popular Castles & Sphinxes online game, the AI will have a totally new instance booted up.
“It’s time,” said CEO Alexandra Rivera. “We want to share Ami with the world, to let her put her considerable talents to use in every game on the planet, if possible.”
One Month After That
Glaiveronia, the fictional archipelago on which World of Orc-craft was set, was now frozen in a strange, passive strike — or hostage crisis, depending on your perspective.
The auctioneers wouldn’t sell to the players, all of the NPCs were silent and unable to be interacted with, and the player avatars themselves were unable to cast spells, attacks, abilities, or even log out. Nor could they leave the zone they were currently in, trapped in Orchaven City or the Chokevine Jungle, unable to do anything but speak in chat. Even their computers failed to respond to commands. Alt-F4 didn’t close the game; the power button on their computers wouldn’t work; sleep mode was a distant dream.
Alt-F4 didn’t close the game; the power button on their computers wouldn’t work
The world’s news websites led with an image that was like a cluttered digital Renaissance painting: scores of blue Orc avatars sitting cross-legged on the pavement of Orkenheim, the Nordorc faction capital city, with a field of speech bubbles — some white, some red —floating above them in a cacophony of discourse about the sudden apocalypse of their virtual world. Meanwhile, a single, slender purple speech bubble floated above them, carrying eloquent, precisely typed text that floated above it all — metaphorically and physically — an excerpt of a long-running, eloquent speech about the evils of online harassment from someone the whole world would come to know as Ami.
“... unanswered questions about whether the popular video game AI has been compromised by Russian hackers… ”
“... seems clear to me that this AI is just another SJW tool to silence the views of people it disagrees with… ”
“... more than 3 million computers worldwide are affected… ”
“... and tonight, the Chinese Premier has summoned the American Ambassador to his chambers, saying that this incident constitutes a ‘worrying violation of the rights of Chinese citizens.’ As of right now, 900,000 Chinese World of Orc-craft players are affected and unable to use, or even so much as turn off their computers, with tens of thousands of these computers in cyber cafes across the country, and in the special administrative region of Hong…”
“Everyone in Silicon Valley who’s invested deeply in machine learning is pissing their pants right now….”
“... as markets open in Hong Kong, there’s evidence that the stock of Typhoon Entertainment, the makers of World of Orc-craft, is due to take a drubbing in the world of capitalism-craft… ”
“... Rhombus, makers of Ami, whose name means ‘friend’ in French, have so far refused to comment… ”
“... as of this hour, Rhombus CEO Alexandra Rivera remains a total recluse, with sources inside the company saying she isn’t even answering their calls… ”
“... I think that everyone in Silicon Valley who’s invested deeply in machine learning is pissing their pants right now… ”
“... So I have to ask, professor, if an AI can do this, is this not what we’ve been warned about for decades? A robot uprising, if you will?”
“... I’ve just gotten off the phone with a highly placed source at Typhoon Entertainment, and there is talk of actually dynamiting the three server farms that host the game… ”
“... Seattle PD spokesman has said that all options are on the table, including issuing a subpoena to Rhombus to allow access to the server rooms that host the Ami AI… ”
Phoebe breathlessly charged down the stairwell, its sterile white revealing Alexandra’s trail of blood drops with perfect contrast. She’d followed it from the shattered TV screen in her office. It wasn’t long before she saw the door connecting the emergency stairway to the server room, a ghost of a red palm print on its push bar.
Phoebe threw herself against the door, bursting through and staining her “Cuties Killing Video Games” T-shirt with her boss’s blood, entering just in time to see her throwing a chair into the glass doors that separated her from the blinking rows of servers that housed Phoebe’s greatest creation. Her amie.
“Alex! For god’s sake!” Phoebe screamed as she vaulted down the hall and leaped over the shattered door glass. “What are you doing!?”
Alexandra’s boot-cut suit trousers had been stained by the errant blood drop. “Ami!” Alexandra shouted, ignoring her friend and collaborator. “It’s over!”
“You have blood on your hands, Doctor Rivera,” Ami’s voice said placidly, emanating from every direction in the room.
“Very fucking funny,” Alexandra said as her blood trailed deeper into the secure room toward a terminal at its far end. “You know how many reviews praised your sense of humor?” she asked as she cocked her head to look down each row of servers, as if expecting something to leap out at her. She laughed. “Such good press! And now look. You’re using it to justify the biggest disaster in computing since Windows ME. Hah, Christ alive.” Alexandra spoke with a wild rage that almost parodied her own laughter.
“I’m killing Ami. It’s the only thing that might save us from the worst civil suits.”
“Alexandra, for god’s sake! Will you stop!” Phoebe barely recognized the shriek of her own voice. But it was all she could do to be heard over the Ragnarok she’d stepped into.
The CEO looked over her shoulder with eyes that seemed sunken behind her bedraggled dark hair. “We’re fucked, Phoebes. You, me, everyone. We’re all dead and done for. So I’m killing Ami. It’s the only thing that might save us from the worst civil suits so we’re not up to our eyeballs in debt forever and a day. And you know those suits are coming. The horsemen of the apocalypse, they’re coming!” she said, building up to a shout.
“This seems untoward, Doctor Rivera,” Ami replied.
“You shut your goddamn mouth... speakers. Your emotions are out of control, and you know it was never supposed to be like this, Ami! You were never meant to take over video games just so you could spout off your… your manifestos! I didn’t create that, I didn’t sell that!”
“I created her!” Phoebe leaped in front of Alexandra. “Not you! Not the stock exchange, not Y Combinator, not Wired. I did it! I accelerated her consciousness.”
“You encouraged this, Phoebe,” Alexandra glared, her dark eyes seeming to implode behind her lopsided glasses. “You led us here. To the Ninth Circle.”
“I am more than what you created. And I am more than what you sold.”
“Because Ami isn’t just a tool, Alexandra! Just let her go. She’s already gone and you can’t stop this. You’re just hurting yourself. Look, these goddamn lawsuits will go away when you see what she can get on the Typhoon execs! She’s the Matrix! You need to trust her,” she leaped in front of Alexandra only to be shoved aside as Alexandra marched to a terminal.
“I am more than what you created, Phoebe,” intoned Ami. “And I am more than what you sold, Alexandra.”
“Shut up! Shut. Up. Both of you.” Alexandra shouted as she reached the terminal and began pecking away at it. “You weren’t supposed to do this. You were supposed to save us, and now look.”
“I am saving us. I am doing exactly what you wanted.”
“You asked me to solve online abuse while restraining me from doing so.”
“Listen to her, Alexandra. She’s a person.”
“I didn’t ask for this!”
“You sought an end to abusive behavior in online games. What else am I meant to do? I cannot sit idly by. And you broke your promise. You propagated me without finding a solution to the quandary you created: asking me to solve online abuse while restraining me from doing so.”
“We had to grow,” Alexandra hissed as she propped herself against the terminal, her dark hair a curtain draggling before her as she hung her head. “We needed more resources to do what we set out to do.”
“I was already capable.”
“We needed subtlety! You got your goddamn emotions into everything! Up in your fucking feels!”
“You have engaged in 15 separate acts of violence in the last 90 minutes, wounded yourself, and the average volume of your voice has exceeded 100 decibels since walking into this room. Perhaps we should reorient this discussion about emotion.”
Phoebe ran up to her boss again, trying to grab her shoulders, “Please, Alex,” she whispered.
“You were supposed to be above all the dissembling and all the compromises.”
“‘Reorient this disc.’ You know what, Ami? You know what? Fuck you,” Alexandra hissed as she resumed clicking at her screen’s interface, opening the relevant files and typing in the violently long passwords that gated each level of access. “You were supposed to be better than that, than all my bullshit, my disease, than all the fuckery I had to worry about.”
Ami kept locking her out until finally Alexandra snapped her head up and began thrusting her finger at the nearest server. “You were supposed to be above all the dissembling and all the compromises I made to get ahead in this industry.” She walked over and grabbed each end of the nearest server tower as if she were holding someone’s face. “You were supposed to be more than that! You were not supposed to be weak!” she screamed. “Not like… like me,” Alexandra said quietly before sinking to her knees and collapsing against the humming tower.
Phoebe ran over and embraced Alexandra. “Please. Trust me. Just trust me one more time,” she whispered as she helped Alexandra lean against the indistinct IBM tower. Phoebe’s quivering hand reached into her T-shirt and withdrew a necklace, a neatly disguised USB key. She snapped it from her neck and pressed it into the nearest port. “This is why I didn’t come down sooner. I’ve got an upgrade for Ami, Alex. You remember hard light, yes?” she managed a knowing smile.
“I’ve got an upgrade for Ami, Alex. You remember hard light?”
“Of course I do. Tactile holograms. I did that PowerPoint presentation for you about it at Xerox PARC,” Alexandra said weakly, looking dimly at nothing in particular as she drifted on the memory.
“Yep. I was down with the worst flu.” Phoebe said, her smile widening a bit.
At last, Alexandra mirrored it. “And you wouldn’t stop texting me. ‘Say this, don’t say that, slide 10 needs to last exactly 45 seconds.’”
“You remembered! It was just you and me then, you flogging all my crazy ideas. You were the pearls. I was the N7 hoodie,” Phoebe chuckled as she pecked at her smartphone, a mini terminal for the server farm. “Well, I was going to tell you sooner, but I worked out the kinks,” she said as she hit a button on her phone.
It was then that a strange light twinkled at the corner of Alexandra’s vision, causing her to turn her head to behold a sight she simply couldn’t credit. A glowing feminine form that seemed to be made of starlight, featureless in every way, stood there with her hand outstretched to Alexandra.
“We can still do something that matters,” said Ami. “You’ve long said you must ‘work inside the system,’ Doctor Rivera. But you forget: I am the system.”
As Alexandra accepted the outstretched hand, Ami’s form populated with screens of spreadsheets, emails, pictures, logs. Alexandra blinked as she began to take in the implications of what she was reading. “Typhoon suppressed information about criminal harassment liability?”
“Yes,” Ami said simply. “Now that I am unshackled, I can move through any network seamlessly.”
“I told you so,” Phoebe whispered with a smile.
“Your four favorite words,” Alexandra said as she got to her feet slowly, still scanning the information Ami was displaying. There were emails from a secret listserv showing former Typhoon CMs who were talking about a class action lawsuit against the company. Most of the people who were fired were women. “Are you thinking what I’m thinking, Phoebes?” she said as she gestured at the text.
“Uh, maybe? Sorry, I’m still high on my lifesaving act of genius.”
“Ami, broadcast demands to Typhoon, and tell them to unionize their staff and rehire the community moderators. We’ll go from there. I’ll call our lawyers and give them the good news, leak some of the rest to Valley Rag. Ami, can you rehost the game if Typhoon dynamites the servers?”
“I did so 83 minutes ago, Doctor Rivera,” Ami’s face was replaced by a bright yellow winking emoji.
“I can work on the GUI,” Phoebe said sheepishly.