Amazon Wants Alexa to Hear Your Whispers and Frustration

Amazon announced new listening features for Alexa, part of a suite of upgrades to the voice assistant that will expand its ability to understand the world around it.
Image may contain Paper Towel Pill Medication Paper Towel and Tissue
Amazon

(Whispers) Amazon Alexa will soon notice if you talk to it sotto voce—and whisper its response back to you.

The new feature, announced by Amazon today alongside new devices including a microwave and a wall clock at an event in Seattle, is one of several upgrades that will expand the virtual assistant’s ability to listen to and understand the world around it. Alexa will able to confer with you in whispers before the end of the year, making Amazon’s voice-operated assistant less awkward to use when someone is, say, sleeping nearby. Amazon will also make its assistant capable of listening for trouble such as breaking glass or a smoke alarm when you’re away from home, a feature called Alexa Guard.

Meanwhile, inside Amazon’s labs, the company is experimenting with giving Alexa a rudimentary form of emotional awareness, enabling it to listen for the sound of frustration in a person’s voice.

“We’re going beyond recognizing words,” says Rohit Prasad, the vice president who heads work on the artificial intelligence inside Alexa’s guts.

Alexa was an oddity when it launched late 2014 inside the cylindrical Echo speaker. It’s now the leader in a rapidly expanding voice assistant market. Amazon is cagey with numbers but says it sold tens of millions of Echos and other Alexa-enabled devices last holiday season, and rivals like Google and Apple have competing smart speakers of their own.

Alexa evolved out of advances in an approach to artificial intelligence called machine learning, which Amazon used to train algorithms to recognize speech from across the room with surprising accuracy. Tech giants have ramped up investment in machine learning research over recent years to create new products, profits, and streams of valuable data. Amazon’s plans for Alexa depend on Prasad’s team figuring out how to apply the fruits of that research to make the assistant smarter.

The new Alexa Guard feature coming later this year is an example of that. To activate it, you’d call out “Alexa, I’m leaving” or a similar phrase to an Echo or other device on your way out the door. If an Echo device in a home hears the sound of breaking glass or a smoke alarm while you’re gone, it will send a notification to your phone with a link to a recording of the sound that triggered the warning.

To avoid having to create a live audio link between a person’s home back to Amazon’s computer systems, Prasad’s team had to create a new machine learning system that lurks inside an Echo device and constantly listens for alarms or smashing sounds. It was trained in part by using audio samples from public domain video, although Prasad says development also involved some destruction. “We did break a lot of glass in our internal testing,” he says.

Amazon’s audio algorithms are also getting better at tracking subtleties of speech. Prasad’s team trained algorithms to detect the characteristically sibilant sounds of whispered speech to enable the whispering upgrade coming later this year. He says Alexa will also get better at analyzing the prosody of what people say. When combined with better text analysis, that will make tasks such as creating shopping lists easier, because the assistant can understand that “add paper towels, peanut butter, and bananas” to my shopping list refers to three separate items, not just one.

Greg Roberts, a managing director who tracks personal technology at consultants Accenture, says Amazon and its rivals can use AI upgrades to drive increasing usage and adoption of their assistants. When Accenture surveyed users of Echo-like devices late last year, nearly 70 percent already reported using their smartphone less and their voice assistant more. “Similar to what we saw when smartphones first arrived, people move to the interface that’s easier to use,” Roberts says.

There's also evidence that some consumers are wary of advances in the ability of devices like the Echo to listen to them. “Privacy concerns have already been a barrier to adoption,” says Werner Goertz, a research director at analyst Gartner. “The industry’s efforts have not been sufficient to remove this misapprehension.” Goertz spoke to WIRED after disabling the Alexa installed in his hotel room to stop it from hearing its name and butting into the conversation.

The frustration-detection feature Amazon is testing in the lab illustrates the tension between using AI to improve functionality, and privacy. For some consumers, Amazon knowing about their feelings in addition to their purchases and music choices might seem a step too far.

Prasad declines to say when that emotional awareness technology might be deployed but argues it could help Alexa learn from its mistakes and would be used only to improve the assistant. His group is planning to launch a system later this year that does that using other cues, such as when a person interrupts and rephrases their query or command. “We put customer trust first,” Prasad says.


More Great WIRED Stories