The Trouble With HomePod Reviews
by Jean-Louis Gassée
With its HomePod speaker, Apple has once again reshuffled existing genres. As an almost singular representative of the new consumer computational audio devices, HomePod’s slippery algorithms defeat quick and easy reviews.
As a testament to Apple’s place in the pantheon of electronics, reviews for a new product from Cupertino come fast and — often — furious. HomePod, Apple’s contribution to the “smart speaker” genre, exacerbates the venomenon, pardon, phenomenon. Tempting as it is to call out reviewers’ contradictions, ignorance of basics, and reflexive opinions passed as authoritative facts, one might try a more empathetic view of the basic challenges in evaluating speakers, including the not-so-new computational audio category now entering the consumer space.
(Throughout this article, I’m purposefully ignoring the Machine Learning/Artificial Intelligence aspects that I discussed in the Voice UI Monday Note two weeks ago. Here, we’re only concerned with the local audio processing performed by HomePod’s A8 processor.)
While doing research for today’s note, I found my own “Sound Holiday Thoughts” audio review from December 2013 (that was Monday Note 300…today’s is 486). In the prologue, I rejoice at the fortunate consequences of being expelled from a secondary school near Paris and sent to a robust boarding school in Britany where I became acquainted with the first transistor sold “over-the-counter”, the germanium-based OC71. Later, as I ventured into building radios, amplifiers and the complicated work of loudspeakers, I chanced upon the writings of a curmudgeonly audio engineer named Peter Aczel, editor-publisher of Audio Critic magazine. Aczel (sadly, recently deceased) competently and often mercilessly debunked myths about audio amplifiers, cables, “CD conditioning”, and other mendacious ways to part people from their money as they searched for audio nirvana (“a $400 LED clock claimed to improve the sound of a system when it is plugged into the same power line as the stereo”).
Throughout his career, Aczel remarked upon the difficulty in evaluating speakers. On this topic, three important factors stay with me to this day.
First, we must keep in mind the influence of the room and our position in it, how reflective material can make music sound brittle, how carpets and curtains will deaden the sound — or provide welcome balance for an overly bright room.
Second, all speaker comparisons must be double-blind, where neither the person running the test nor the evaluators know which device is on at the moment.
Third, and most important, output level differences matter. Contestant speakers must be carefully equalized to within 1db, because in any comparison the louder speaker will always sound better.
While reading blogosphere articles that compare HomePod to Sonos, Amazon Echo, and Google Home, I can’t help but wonder how many setups follow Aczel’s three cardinal rules. The closest I could find was David Pogue’s four speakers behind a curtain test, a home re-enactment (with crucial improvements) of Apple’s “listening session for reviewers” that Pogue had attended.
“…when I tweeted about the [Apple] test, a couple of people were suspicious of the setup, which of course was entirely controlled by Apple. What was the source material? What was the wireless setup?…So I decided to set up my own test at home.”
Pogue’s arrangement isn’t perfect: He first said he equalized the speakers “by ear”, a no-no given how unreliable loudness perception is. And it wasn’t double blind: As the administrator of the home test, Pogue knew which speaker was playing. This might sound like a petty complaint, but it matters when one considers the unconscious signals that can be sent to the audience.
Despite these imperfections, David Pogue’s review is of a higher quality than most, and was certainly better than Apple’s own listening session, which didn’t even make an attempt at “single” blindness:
“They even had a halo light rigged to turn on behind whichever speaker was playing, so you’d know which was which.”
Nonetheless, Pogue’s first review story got some flak, which he had the smarts to publicize in a follow-up to his first post. In particular, readers mentioned listener positions as a source of trouble:
“…if you’re at the same end of the row as a certain speaker, of course it’s closer to you, and therefore louder!”
Perhaps more important, yet harder to know: Did the fabric curtain interfere with the speakers’ adaptation to the room’s acoustics? Pogue says he did let the HomePod “listen” to the room before hanging the curtain. One can’t help wonder how the adaptation worked during the test after the curtain was installed.
This brings us back to the computational audio topic mentioned at the beginning.
Audio and computers are not new mates. In particular, I recall composer and conductor Pierre Boulez’ work at IRCAM in Paris in the early ’80’s, and the Jupiterian feelings I experienced hearing my voice transformed into a musical instrument in a symphony by the 4X workstation.
After I left Apple in 1990, I was treated to another computational audio performance when I explored the possibility of becoming director of Stanford’s CCRMA (pronounced “karma”, of course). There, the beloved Max Mathews accompanied his opera singer wife Marjorie, by conducting a computer orchestra with his radio baton:
It was beautiful and seductive, and I declined the position with painfully mixed feelings — I knew I wasn’t cut for a university research role.
28 years later, the core idea behind the enhanced sound quality boasted by today’s computational audio devices is deceptively simple: Your “smart speaker” device contains microphones that “listen” to the room, that record and analyze how the sound emitted by the speaker is reflected, dampened, sharpened by the room’s acoustics. This information is used to instantaneously adjust the playback. For example, if an unwanted resonance emanates from the front left, the speaker will send less sound energy in that direction. As more people enter the room, curtains are opened, other noise sources are introduced, the speaker will tweak the equalizer and adjust the spatial sound pattern.
In other words, the sound is “tuned” to fit the room and its occupants.
How well the device can do this depends on the number of microphones (you need at least two for spatial location) and speakers (the HomePod has seven). In theory, more speakers are better, because it means the device will have more control as it adjusts the balance of sound energy. In reality, the smaller speakers — the tweeters — are the most important; lower frequencies impart little directional information, although the computer can still decide to reduce low frequency output in general if the room “booms” too much.
It is a lovely theory and an elegant solution, but one that makes it difficult to evaluate and compare the speakers in any reliable way. The undisclosed, proprietary algorithms that these devices use border on PFM (definition #2 in the Urban Dictionary). The subtle variations as room conditions and listening positions change make the canonical double-blind tests difficult to translate into everyday customer experience.
This is where we find a new type of difficulty when evaluating this new breed of smart speakers, and why we must be kind to the early HomePod reviewers: The technical complexity and environmental subjectivity leads to contradictory statements and inconsistent results. We’ll have to wait and see what actual humans do. As former Microsoft senior exec Steven Sinofsky puts it in a massive but eminently readable Twitter thread now available in Medium:
“Apple’s brilliance is in focusing mostly on two audiences — end-users and developers — tending to de-emphasize the whole “techie” crowd, even IT.”
Put less elegantly, in consumer advertising lore we call early adopters “dogs”: They eagerly snarf any new food. This tells us nothing; we need to wait and see if they come back to the pail.
That process, the spreading of Word-of-Mouth, will be complicated by software updates supposed to appear in the next few months — and for ever after that.
As a good dog, I bought a HomePod but will wait a little more for a Third Impression to form. I have no guess regarding the size of the business for Apple but I have a clear idea of the HomePod’s place in the Apple ecosystem — one that makes me wonder about the company’s missed opportunity for a simpler marketing position, one that promises less and delivers more.
— JLG@mondaynote.com
