When I first saw Google’s new Duplex technology, it was simultaneously thrilling and horrifying. It’s the kind of technological evolution that you initially see as furthering humanity and having the potential to spawn countless business opportunities.
However, as its potential sinks in, you get a creepy vibe as you realize how bad actors might exploit it with malicious intent. As it turns out, what Google Duplex will do may not be quite as promising and scary as its first demo suggested. Nonetheless, this article will explore four privacy and security concerns we should consider as future AI-based personal assistants continue to advance.
First, if you haven’t heard about or seen Google Duplex, you have to watch this video. Check it out now; I’ll wait.
In short, Duplex is a proof-of-concept extension for Google Assistant, the personal assistant technology found on many devices; similar to Apple’s Siri and Amazon’s Alexa. We’ve all become used to talking to personal assistants on our phones and home speakers, asking them to do different simple tasks for us — from setting an alarm to recommending new music.
Over the years, these assistant technologies have begun to sound more and more human, and can respond to more and more types of requests. However, they still remain fairly limited. They can’t carry on a real conversation, they often sound recognizably “robotic,” and they remain very narrow in what they understand and do. Google Duplex seems different.
When watching that demo, you were probably astounded by how human the Duplex assistant sounded. During the demo, Duplex makes two automated calls for its owners to make appointments. The recipients on the calls have no clue they are talking to an automated program, and the call sounds like a normal conversation. The automated assistant tries to make an appointment on a certain day and time, but learns no slots are available, and has to adjust and expand accordingly. It’s something we all do naturally when speaking to one another, but it’s exceptional for a computer program. When the human recipients’ questions start to get complex and switch context, Duplex still keeps up and answers accordingly, even uttering human interjections like, “mm-hmm.” In computing, the Turing test is a challenge to measure a machine or program’s ability to behave indistinguishably from a human. On the surface, Google Duplex passed that test during their IO conference demo.
From a technological standpoint, Duplex is astounding and offers boundless opportunities. However, as you realize the demo victims didn’t know they were talking to a robot, it becomes equally easy to imagine some darker scenarios too. As you’ll learn, Google Duplex will likely be more limited than you might have imagined from its first demo. Nonetheless, as these technologies progress, there’s also the potential for increased misuse. Let’s examine four possible short and long-term considerations for this AI technology:
Duplex begins with deceit – One of Duplex’s biggest strengths is also a significant weakness or danger. The whole point of Duplex is to act as naturally human as possible. Google researchers found as soon as humans realize they’re taking a call from a robot, they’re more likely to hang up. Making Duplex sound human is a core requirement to actually completing some non-digital tasks that require cooperation from another human, such as setting an appointment in absence of digital scheduling mechanisms.
However, this also means Duplex starts its interactions with duplicity in mind. During the first demo, the human caller had no clue they were talking to a robot. While I don’t think Google has bad intentions in mind, this initial deception does prove it’s possible for an AI to trick humans. Once this type of technology becomes widely available, you can easily imagine bad actors bending it to more malicious purpose.
However, Google seems to have adjusted to this perception problem. A few days ago, they showed a new, real-world demo. During this demo, Duplex mentioned it was an automated assistant right out of the gate. Though this step makes Google Duplex safer, the cat is already out of the bag. Now that the world has seen an AI convince someone that it’s a fellow human, I believe it’s only a matter of time before someone designs a call AI that is meant to deceive.
Privacy concerns – To do its job, technologies like Duplex likely have to record you, similar to how Siri and Alexa do. We all hope that well-meaning companies only hold these recordings temporarily—just long enough for the AI to analyze and understand the speech. But who really knows? Companies have elected to store all the data they collect before. Not only could these recordings contain things we say that we might want to keep private, but they also hold core characteristics about us. Specifically, our distinct voice, which could be used against us (as you will learn later).
Automated systems recording our voice without our consent isn’t only a privacy concern, but it could have serious legal implications as well. Some states have wiretapping laws which don’t allow you to record another person on a call without consent. During the original demo, Duplex did not say it was recording the call. However, Google also corrected that oversight with their second demo. Now, Duplex warns you its recording, saying things like, “I’m Google’s automated booking service, so I’ll record the call.” Nonetheless, that still leaves the potential privacy implications of a corporation having recording of millions of peoples’ voices, as well as any sensitive thing they say. It’s nice for companies like Google to say they will “do no evil,” but history has shown we can’t always take corporations at their word. Before this technology gets too widespread, I hope we see some regulations limiting how long and how much of a recording these types of solutions can store.
Automated vishing – One of the biggest threats to information security doesn’t even involve technology. Social engineering is the act of conning a victim into giving you information you shouldn’t have. One of the world’s most infamous hackers, Kevin Mitnick, made a name for himself by socially engineering victims over the phone; an attack the security industry calls vishing (a portmanteau of voice phishing).
If you attend some security conferences like DEF CON, you often see presentations where social engineers vish victims live from the podium. If you haven’t seen social engineering before, watch this great video showing you how easy it is for someone to change your credentials with a simple phone call. In short, vishing is one of the most difficult threats to protect yourself from since technology typically can’t help. It’s all about using psychology to exploit natural human weakness and tendencies.
One of the few things really saving the industry from vishing attacks is they don’t scale. Every social engineering call requires the time of a human, who can only make one call at a time. However, imagine if AI assistants were crafted for vishing calls instead. Now an AI might make hundreds or thousands of calls at a time. The first thought I had while watching Duplex convincingly making an appointment as a human was that hackers will be all over this.
The good news is I don’t think attackers will be able to directly leverage Google Duplex for evil. First, it seems like it might be a closed system, only available to Google Assistant. Second, Google researchers claim they have only designed it for “closed domain” tasks. Even though Duplex seemed to carry on a dynamic and interactive conversation about “appointment scheduling,” it doesn’t mean it knows how to talk about anything else. I don’t think attackers will literally be able to use Duplex to make vishing calls right away.
However, Google Duplex does prove that you can build a recurrent neural network (RNN) that can speak like a human and do specific phone tasks quite well. Considering Duplex is built upon Google’s open source TensorFlow project, I predict it won’t be long until bad actors build malicious Duplex-like variants on their own.
Spear-vishing: Future “AI assistants” could sounds like you – Duplex has proven that AI technology is strong enough that a motivated black hat could write a malicious AI phone assistant that made vishing calls … but let’s take that to the next level. AI and machine learning are also allowing researchers to mimic our voices and images as well. In fact, last year a company called Lyrebird unveiled a voice imitation algorithm that could mimic a real person simply based on having a small audio snippet of their voice. Remember how technologies like Duplex record the human’s voice on a call? Now imagine that becoming a huge library of audio which could be used to imitate the voices of others. Automated vishing calls from a computer AI are bad enough, but what if these calls could pretend to come from your friends, family, or even your boss?
Spear-phishing has become a huge threat. Often, sophisticated hackers learn a bit about you, and use that knowledge to design emails that you’re more likely to respond to. One example is crafting an email that appears to come from your boss. Now imagine a call that comes from your boss. It sounds like your boss, talks with his or her cadence, but is actually an AI assistant using a voice imitation algorithm. Such a call offers limitless malicious potential to bad actors.
Personally, I believe technology is benign—it’s neither good nor evil. Rather, how we decide to use or mitigate it resolves whether or not that technology ultimately saves or destroys humanity. Google Duplex looks pretty amazing and cool. However, we need to realize the negative potential of products like Duplex — or more specifically products leveraging AI in new ways. Ultimately, I believe we’ll find a way to leverage these technologies to improve society. However, we should carefully consider new regulations, security design, and other safeguards that limit the potential damage that powerful technologies can achieve. Don’t be a dupe and allow yourself to be deceived by an unchecked malicious Duplex.
