It came as a total surprise: the most impressive demonstration at Google’s I/O conference yesterday was a phone call to book a haircut. Of course, this was a phone call with a difference. It wasn’t made by a human, but by the Google Assistant, which did an uncannily good job of asking the right questions, pausing in the right places, and even throwing in the odd “mmhmm” for realism.

The crowd was shocked, but the most impressive thing was that the person on the receiving end of the call didn’t seem to suspect they were talking to an AI. It’s a huge technological achievement for Google, but it also opens up a Pandora’s box of ethical and social challenges.

For example, does Google have an obligation to tell people they’re talking to a machine? Does technology that mimics humans erode our trust in what we see and hear? And is this another example of tech privilege, where those in the know can offload boring conversations they don’t want to have to a machine, while those receiving the calls (most likely low-paid service workers) have to deal with some idiot robot?

In other words, it was a typical Google demo: equal parts wonder and worry.

But let’s start with the basics. Onstage, Google didn’t talk much about the details of how the feature, called Duplex, works, but an accompanying blog post adds some important context. First, Duplex isn’t some futuristic AI chatterbox, capable of open-ended conversation. As Google’s researchers explain, it can only converse in “closed domains” — exchanges that are functional, with strict limits on what is going to be said. “You want a table? For how many? On what day? And what time? Okay, thanks, bye.” Easy!

Mark Riedl, an associate professor of AI at Georgia Tech with a specialism in computer narratives, told The Verge that he thought Google’s Assistant would probably work “reasonably well,” but only in formulaic situations. “Handling out-of-context language dialogue is a really hard problem,” Riedl told The Verge. “But there are also a lot of tricks to disguise when the AI doesn’t understand or to bring the conversation back on track.”

One of Google’s demos showed perfectly how these tricks work. The AI was able to navigate a series of misunderstandings, but did so by rephrasing and repeating questions. This sort of thing is common for computer programs designed to talk to humans. Snippets of their conversation seem to show real intelligence, but when you analyze what’s being said, they’re revealed as preprogrammed gambits. Google’s blog post offers some fascinating details on this, spelling out some of the verbal ticks Duplex will use. These include elaborations (“for Friday next week, the 18th.”), syncs (“can you hear me?”), and interruptions (“the number is 212-” “sorry, can you start over?”).

It’s important to note that Google is calling Duplex an “experiment.” It’s not a finished product, and there’s no guarantee it’ll be widely available in this form, or widely available at all. (See also: the real-time translation feature Google showed off for its Pixel Buds last year. It worked flawlessly onstage, but was hit-and-miss in real life, and available only to Pixel phone owners.) Duplex works in just three scenarios at the moment: making reservations at a restaurant; scheduling haircuts; and asking businesses for their holiday hours. It will also only be available to a limited (and unknown) number of users sometime this summer.

One more big caveat: if a call goes wrong, a human takes over. In its blog post, Google says Duplex has a ”self-monitoring capability” that allows it to recognize when conversations have moved beyond its capabilities. “In these cases, it signals to a human operator, who can complete the task,” says Google. This is similar to Facebook’s personal assistant M, which the company promised would use AI to deal with customer service scenarios, but ended up outsourcing an unknown amount of this work to humans instead. (Facebook closed this part of the service in January.)

All this gives us a clearer picture of what Duplex can do, but it doesn’t answer the questions of what effects Duplex will have. And as the first company to demo this tech, Google has a responsibility to face these issues head-on.

The obvious question is, should the company notify people that they’re talking to a robot? Google’s vice president of engineering, Yossi Matias, told CNET it was “likely” this would happen. Speaking to The Verge, Google went further, and said it definitely believes it has a responsibility to inform individuals. (Why this was never mentioned onstage isn’t clear.)

Many experts working in this area agree, although how exactly you would tell someone they’re speaking to an AI is a tricky question. If the Assistant starts its calls by saying “hello, I’m a robot” then the receiver is likely to hang up. More subtle indicators could mean limiting the realism of the AI’s voice or including a special tone during calls. Google tells The Verge it hopes a set of social norms will organically evolve that make it clear when the caller is an AI.

Joanna Bryson, an associate professor at the University of Bath who studies AI ethics, told The Verge that Google has an obvious obligation to disclose this information. If robots can freely pose as humans the scope for mischief is incredible; ranging from scam calls to automated hoaxes. Imagine getting a panicked phone call from someone saying there was a shooting nearby. You ask them some questions, they answer — enough to convince you they’re real — and then hang up, saying they got the wrong number. Would you be worried?

But Bryson says letting companies manage this themselves won’t be enough, and there will need to be new laws introduced to protect the public. “Unless we regulate it, some company in a less conspicuous position than Google will take advantage of this technology,” says Bryson. “Google may do the right thing but not everyone is going to.”

And if this technology becomes widespread, it will have other, more subtle effects, the type which can’t be legislated against. Writing for The Atlantic, Alexis Madrigal suggests that small talk — either during phone calls or conversations on the street — has an intangible social value. He quotes urbanist Jane Jacobs, who says “casual, public contact at a local level” creates a “web of public respect and trust.” What do we lose if we give people another option to avoid social interactions, no matter how minor?

One effect of AI phone calls might be to make us all a little bit ruder. If we can’t tell the difference between humans and machines on the phone, will we treat all phone conversations with suspicion? We might start cutting off real people during calls, telling them: “Just shut up and let me speak to a human.” And if it becomes easier for us to book reservations at a restaurant, might we take advantage of that fact and book them more speculatively, not caring if we don’t actually show up? (Google told The Verge it would limit both the number of daily calls a business can receive from Assistant, and the number of calls Assistant can place, in order to stop people using the service for spam.)

There are no obvious answers to these questions, but as Bryson points out, Google is at least doing the world a service by bringing attention to this technology. It’s not the only company developing these services, and it certainly won’t be the only one to use them. “It’s a huge deal that they’re showcasing it,” says Bryson. “It’s important that they keep doing demos and videos so people can see this stuff is happening [...] What we really need is an informed citizenry.”

In other words, we need to have a conversation about all this, before the robots start doing the talking for us.

Next Up In Tech

• Boeing employees’ frightening internal messages released in 737 Max investigation
• Microsoft’s Windows Terminal is getting retro-style CRT effects, search, and more
• ‘Black Book’ suggests the feds have some unexpected surveillance tools, including a gravestone camera
• Amazon suspiciously says browser extension Honey is a security risk, now that PayPal owns it
• CES 2020 winds down: Intel shows off a DG1 graphics card and Roombas get arms
• Some NYC subway riders are accidentally getting double-charged because of Apple Pay