One day in 2017, Alexa went rogue. When Martin Josephson, who lives in London, came home from work, he heard his Amazon Echo Dot voice assistant spitting out fragmentary commands, seemingly based on his previous interactions with the device. It appeared to be regurgitating requests to book train tickets for journeys he had already taken and to record TV shows that he had already watched. Josephson had not said the wake word – “Alexa” – to activate it and nothing he said would stop it. It was, he says, “Kafkaesque”.
This was especially interesting because Josephson (not his real name) was a former Amazon employee. Three years earlier, he had volunteered to sit in a room reciting a string of apparently meaningless phrases into a microphone for an undisclosed purpose. Only when Amazon released the Echo in the US in 2014 did he realise what he had been working on. He bought a Dot, the Echo’s cheaper, smaller model, after it launched in 2016, and found it useful enough until the day it went haywire. When the Dot’s outburst subsided, he unplugged it and deposited it in the bin. “I felt a bit foolish,” he says. “Having worked at Amazon, and having seen how they used people’s data, I knew I couldn’t trust them.”
The Dot wasn’t supposed to behave like a dadaist drill sergeant. Then again, voice assistants often do things that they are not supposed to do. Last year, an Amazon customer in Germany was mistakenly sent about 1,700 audio files from someone else’s Echo, providing enough information to name and locate the unfortunate user and his girlfriend. (Amazon attributed this “unfortunate mishap” to human error.)
In San Francisco, Shawn Kinnear claimed that his Echo activated itself and said cheerfully: “Every time I close my eyes, all I see is people dying.” In Portland, Oregon, a woman discovered that her Echo had taken it upon itself to send recordings of private conversations to one of her husband’s employees. In a statement, Amazon said that the Echo must have misheard the wake word, misheard a request to send a message, misheard a name in its contacts list and then misheard a confirmation to send the message, all during a conversation about hardwood floors. Not great, Alexa.
Technology frequently inspires ambivalence: we know that Facebook and Google know too much about us, yet we continue to use their services because they’re so damn convenient. Voice assistants, however, are unusually polarising. People who consider them sinister and invasive (myself included) regard enthusiasts as complacent, while those who find them useful and benign see the sceptics as paranoid technophobes. There is one question freighted with bigger issues about our relationship with the tech industry: should you let Alexa into your home?
In January, Amazon’s senior vice-president of devices, David Limp, revealed that the company had sold more than 100m Alexa-enabled devices. Last year in the US, where one in five adults own a home voice assistant, Alexa had a 70% US market share, compared with the Google Assistant’s 24%. It is therefore best-placed to become what Shoshana Zuboff, in her bestseller, The Age of Surveillance Capitalism, calls the “One Voice”: the dominant ecosystem that would give its operator “the ability to anticipate and monetise all the moments of all the people during all the days”. Zuboff calls the birth of Alexa “a threshold event”.
Alexa, however, has grown up in an era of increasing scepticism about the power and morality of the “big five” tech companies: Amazon, Apple, Facebook, Google/Alphabet and Microsoft. Events such as the Edward Snowden leaks and the Cambridge Analytica scandal have tarnished Silicon Valley’s utopian promises. CEOs are less likely to trivialise privacy concerns. Tech journalists are more likely to be critics than cheerleaders. Politicians are more willing to hold companies to account.
This year has been particularly tricky. Over the past six months, Bloomberg, the Guardian, Vice News and the Belgian news channel VRT have gradually revealed that all the big five have been using human contractors to analyse a small percentage of voice-assistant recordings. Although the recordings are anonymised, they often contain enough information to identify or embarrass the user – particularly if what they overhear is confidential medical information or an inadvertent sex tape. The revelations were the last straw for many Alexa sceptics. “We live in a techno-dystopia of our own making. If you still have an Alexa or any other voice assistant in your home, you were warned,” wrote the Gizmodo writer Matt Novak.
Having worked at Amazon and another big-five company, Josephson thinks this resistance to these companies is justified. “They have zero interest, in my opinion, in wondering what the impact of those products will be. To treat them as the right people to wield that power is a ludicrous situation that we wouldn’t allow in any other industry. They, frankly, are not safe guardians of the data that they’re collecting every day without us knowing.”
Voice control first seized the public imagination in the 60s, via HAL 9000, the sentient computer in 2001: A Space Odyssey, and the Starship Enterprise’s endlessly helpful computer in Star Trek. The latter was a major reference point for the teams that developed Amazon’s Echo and the Google Assistant. “The bright light, the shining light, that is still many years away, many decades away, is to recreate the Star Trek computer,” Limp told a conference audience in 2017.
In the real world, voice recognition didn’t become commonplace until Apple launched its phone-based voice assistant, Siri, in 2011. Alexa goes much further by colonising the user’s home. Three decades ago, the prescient computer scientist Mark Weiser called this kind of frictionless ecosystem “ubiquitous computing”. In 2015, Google’s Eric Schmidt foresaw a day when the internet will be “part of your presence all the time”, making your real-life behaviour as mappable as what you do on your laptop or phone. You will never be offline.
The Gizmodo editor Adam Clark Estes was initially excited by the first major hardware innovation since Apple’s iPad four years earlier. The more he learned about the technology, however, the less he liked it. His own Echo would wake up unprompted. Recordings began showing up as evidence in court cases. The FBI refused to confirm or deny that it was using Alexa for surveillance purposes. “It became increasingly clear to me that the privacy watchdogs were right,” he says. “It is, at base, a wiretapping device.”
Voice assistants epitomise the tension between efficiency and privacy. The technology is still deeply imperfect; in more than one out of 10 transcripts analysed by one of Bloomberg’s sources, Alexa woke up accidentally. Accurately interpreting voice commands by taking account of different languages, accents, tones, contexts and degrees of ambient clutter requires far more computational power than a single device can contain. Therefore, most of the work is performed in the cloud, which is how human monitors are able to collect and analyse voice recordings. “You are building an infrastructure that can be later co-opted in undesirable ways by large multinationals and state surveillance apparatus, and compromised by malicious hackers,” says Dr Michael Veale, a lecturer in digital rights and regulation at UCL Faculty of Laws at University College London.
Not all voice assistants are equal. Apple, whose profits don’t rely primarily on data collection, uses more in-device computation and encryption at both ends. “Apple is the best at privacy,” says Estes. “At the same time, I think everyone agrees that Siri sucks.” Amazon Echo and Google Assistant are much more reliable because these companies’ business models depend on knowing so much more about you in order to microtarget advertising. They like to frame data collection as a means to improve services while playing down the immense commercial benefits.
Google and Amazon could, of course, choose to improve the technology by paying people to test it – the kind of work Josephson performed in 2012 – instead of treating their customers as a free research-and-development database. Or they could advertise the fact that humans may be studying the recordings. “It should be on the box,” says Dr Jeremy Gillula, the project director at the Electronic Frontier Foundation, a group that campaigns against the misuse of technology. The generous interpretation of tech companies’ motives is a heedless utopianism, but Gillula argues that if they were truly naive about the implications of their technology, then they wouldn’t go to such lengths to conceal them. “I doubt they thought no one would care. I think they were trying to keep it quiet because if users knew what was going on they might stop buying the devices. It was a calculated business decision.”
In response, a spokeswoman for Amazon says: “Customer trust is at the centre of everything we do and we take customer privacy very seriously. We continuously review our practices and procedures to ensure we’re providing customers with the best experiences and privacy choices. We provide customers with several privacy controls, including the ability to review and delete their voice recordings. To help improve Alexa, we manually review an extremely small sample of Alexa requests to confirm Alexa understood and responded correctly. Customers can opt out of having their voice recordings included in that review process.”
Yet it has been demonstrated time and time again that, in their advertising and shrewdly worded privacy policies, tech companies routinely obscure the extent and nature of their data harvesting. “Google and Amazon have shown us that they’re inclined to take as much as they can until someone catches them with their hand in the cookie jar,” says Estes. “I hate to be dramatic, but I don’t think we’re ever going to feel safe from their data-collection practices. Government regulation is the only thing that is going to halt more damage.”
The US government has been reluctant to act. In July 2015, the Electronic Privacy Information Center (Epic), a long-established nonprofit organisation dedicated to “democratic values in the information age”, called for an investigation into “always on” devices, including voice assistants. They have yet to receive a response. “Market-based solutions don’t work in this area because companies have been allowed to conceal the defects of their own products,” says Marc Rotenberg, Epic’s executive director. He adds: “If you or I were to place a device in someone else’s home with the ability to capture their private communications, we would be in violation of the Federal Wiretap Act.”
If you’re an authoritarian country, why not just run the audio stream straight to a government surveillance agency?
In July, a Democratic congressman, Seth Moulton, introduced the Automatic Listening Exploitation Act, which would penalise companies whose voice assistants and smart doorbells (such as Amazon’s Ring) record conversations without permission, but Gillula considers it “pretty weak legislation”. In Europe, which has bitter first-hand experience of mass surveillance under totalitarian regimes, the regulators have sharper teeth. Recently, Johannes Caspar, Hamburg’s formidable commissioner for data protection, forced Google to suspend the transcription of voice recordings throughout the EU, while Apple and Facebook simultaneously halted transcription everywhere in the world. (Amazon merely allowed users to opt out of human monitoring.) The UK Information Commissioner’s Office is investigating whether human oversight breaches the General Data Protection Regulation. “We have significant regulation in this area already, but the key is enforcement,” says Veale. “Like all European data-protection authorities, the Information Commissioner’s Office is underfunded and overstretched, and needs significant investment and support to protect citizens’ fundamental rights.”
Without effective regulation, there is no defence against more invasive exploitation of voice assistants. By definition always on, even when they are not awake, the devices are constantly listening, although not always transmitting. Gillula says that there are no technical obstacles to enabling dormant devices to, for example, track users’ television viewing by responding to high-pitched signals embedded in shows and advertisements, or identify who is in the house at any given time. “That essentially becomes constant surveillance,” Gillula says. “I am hopeful that the companies would never go down this dystopian path, but I could see them saying: ‘Oh, it’s a feature: know when your kids are home!’ An appealing feature is how most of these things start.”
Where they end up has galvanised not just privacy watchdogs but writers. The dystopian implications of voice assistants are appearing in science fiction, including the Spike Jonze movie Her, Black Mirror and Years and Years. Behind them all looms the “never-sleeping ear” of George Orwell’s telescreen in Nineteen Eighty-Four: “You had to live – did live, from habit that became instinct – in the assumption that every sound you made was overheard.” The Echo Show, a smart assistant with a screen and camera, was widely compared to the telescreen when it was introduced two years ago. “Alexa, show me the dystopia,” quipped Vice. In Joanna Kavenna’s satirical novel Zed, the writer explores the “weird dysfunctional-servant aspect” of real-life voice assistants (which all have female voices as their default setting) and the sociopolitical consequences of the One Voice: in Zed, the tech monopoly, Beetle, is omnipresent and unaccountable. “The democratic idea is that we’re meant to have transparent corporations and governments, while people have privacy,” she says. “We have the inverse. People are uneasy because it’s not being debated at a mature, democratic level.”
Kavenna worries that, far from legislating to protect user privacy, US states will seek to access voice-assistant recordings in the name of crime prevention and national security. Last year, a judge in New Hampshire made headlines by ordering Amazon to submit Echo recordings of a double murder to investigators. “It puts them in a very complicated position between their customers and the government,” Kavenna says. “We’re very dependent on the political regime.”
Rotenberg agrees: “If you’re an authoritarian country, why not just run the audio stream straight to a government surveillance agency and argue that it’s to reduce crime in the home? It’s scary to contemplate, but conceivable.”
Today, voice assistants are not the most pressing threat to privacy only because they are optional. A facial recognition scanner can spy on you in a public space, but Alexa, like a vampire, must be invited into your home. The only truly effective power you can wield over this technology is not to use it. But making an informed choice is compromised by misleading marketing and undermined by Amazon’s efforts to embed Alexa into countless products, including cars, televisions, headphones, microwaves, thermostats and clocks, while signing deals with housebuilding companies and hotel chains, all with an eye to becoming the One Voice. “Voice control is being forced down consumers’ throats whether they like it or not,” says Estes.
We are not there yet. Tech companies rely on the myth of technological inevitability to occlude the business decisions they have made and the possibility of other models. To challenge them is to fight the future. Nonsense, says Kavenna. “Is it an inevitable consequence of tech? No, it’s an ideological application of tech. If you build a building, you don’t have to build a panopticon. It doesn’t have to be what we have now. They’ll often claim that if you’re opposed to this, you’re a neo-luddite. That myth has allowed a lot of people to become incredibly rich.”
None of the people I spoke to owns a home voice assistant, nor would they advise anyone to get one, but they all agree that it would be possible to develop a device that delivers the most popular services while respecting the user’s privacy. If the entire industry were to follow Apple’s lead in making human monitoring opt-in rather than opt-out, that would be a strong start. Then, as processing power increases, more tasks could be performed inside the device. But, of course, that would mean forfeiting that juicy, monetisable data. “A lot of stuff that you really want to do shouldn’t require the internet, and once you cut off access to the cloud, then the privacy concerns fade away,” says Estes. “There’s always the potential for things to get better – and there’s always going to be a fight to get to that point.”
Jeremy Gillula is so convinced that a safe, fully user-controlled voice assistant is possible that he is designing his own, using open-source software.
“I’m getting there,” he says cheerfully. “I’m not 100% satisfied yet, but it will turn the lights on and off.”