Tech —

Send Wi-Fi companies floor plans, receive the ultimate mesh networking test

When Wi-Fi companies have suggestions, why not let them participate directly?

The old showbiz adage continues to hold true (even in Wi-Fi testing): you can't please everyone. Shortly after our last round of mesh Wi-Fi testing, in which a six-pack of Plume devices surprised the field, e-mails arrived from both the Google Wifi and AmpliFi HD teams. The results weren't representative of their devices, they said, and perhaps I placed the devices badly. Both companies suggested placing an access point (AP) downstairs instead of all three APs being upstairs.

While I doubted this pretty strongly—such a setup would require a multi-hop "tree" topology, which neither device is really designed well for—I set my own ego aside. At the very least, these pleas highlighted a weakness common to any three-piece mesh kit: they're deceptively difficult to place well.

But blindly following Google's and AmpliFi's recommendations to move an access point downstairs would have weakened the devices' previous coverage pattern upstairs. That arrangement means the upstairs and downstairs access points have to cover half of the house from one location rather than each covering about a third of the house the way I'd had them arranged.

Instead, I responded to both companies with a compromise. Why not leave placement to the experts? I sent my floor plans over to Google and AmpliFi—as well as offering the same opportunity to Eero, Orbi, and Plume—and allowed these companies to make location recommendations as detailed as they'd like. If my role in testing was purely reading the measurements, would the results be different?

Testing methodology

In my ongoing work with Wi-Fi at Ars, I've spent a fair amount of time on RF "heat maps" and only a little on throughput testing. As a general rule, I'm not going to do that anymore. It turns out that heat maps aren't always relevant, and sometimes they're even misleading. As I mentioned in a lengthy explainer on Wi-Fi, we've all been trained to "hunt for more bars," but that frequently doesn't correspond well to actual performance.

So in our tests this time around (Mesh Showdown II: Return of the Wi-Fi?), we're looking at iperf3 throughput—both upload and download—in each of seven sites selected throughout the test house.

Test Sites

Upstairs, Bedroom 1 and Office 1 are both easy shots to the router. For three-piece kits with optimal placement, they're also easy shots to a satellite AP in Bedroom 2 or a satellite AP in the Kitchen, respectively.  Bedroom 2 is a moderately long-range shot to the router, in which a client can either make the long-range connection to the router directly or can connect locally to the AP in the same room and let the AP manage the long-range shot.

Downstairs has its own three test sites: Office 2, Downstairs Bath on the left, and Bedroom 3 on the right. It doesn't look like there's a lot of difference between Office 2 and Downstairs Bath (they're only a few feet apart), but Downstairs Bath is in the deepest part of the RF shadow the foundation slab casts. A signal straight from the router to the Downstairs Bath would need to get through a concrete foundation slab and several feet of earth.

Finally, there's the car. This is an extremely long-range torture test, made while sitting in the car with its doors and windows closed, engine and air conditioner running, and its car stereo playing music streaming from my phone over Bluetooth. Furthermore, my wife's not-so-mini-van is parked in the carport, meaning a shot to the router has to traverse a total distance just under 50 feet including exiting the car, going through the minivan, and penetrating one exterior and two interior walls along the way.

The tests: Hard Mode and Easy Mode

If those e-mails referenced up top aren't evidence enough, know that it's always tough meeting expectations for testing whether we're talking companies or readers. Some readers want the thing that just works without any hassle and don't care about the biggest numbers you can get, but other readers want to know the best performance you can possibly wring out of the system if you fiddle with it and hold your mouth just right.  So this time out, I'm testing each device both ways. I strongly recommend paying more overall attention to Hard Mode. There's not much point in bragging about big numbers if you can't get consistency, in my opinion. The whole point of Wi-Fi is hands-off convenience—if you're gonna fiddle with it, you might as well just run a wire.

For this go-round, I'm focusing on single-device (the test laptop is the only device connected to the wLAN) throughput as measured by iperf3. Unlike wired devices, Wi-Fi devices (access points, routers, and clients) tend to differ rather dramatically in download versus upload throughput, so I'm testing both ways. The client device (an Acer C720 Chromebook running Ubuntu 16.04, using a USB3-connected Archer T4U 802.11ac two-stream wNIC) is connected to the wLAN once, and then this client is required to do its own band-steering and AP-steering ("roaming") as I walk through the test sites in a specific order: Office 2, Downstairs Bath, Bedroom 3, Bedroom 1, Bedroom 2, Office 1, Car. Absolutely no assistance is given to the client device during Hard Mode testing. If it fails to roam to the best access point, tough. If it roams without assistance to a better access point during the three tests than the one it started out on, also tough. It still only gets three runs at the prize in total, and the median run of the three is selected.

If a device doesn't seem to have connected in the best way it could have for any of its three runs at a particular test site during Hard Mode testing, I'll return to that site after the Hard Mode run is over and make a best effort to finagle the setup into doing better. This best effort is usually restricted to manually disconnecting and reconnecting the test laptop from the wLAN a few times to try to get a more reasonable connection, but in some cases (and I am looking balefully at AmpliFi HD, here) it might get a bit more heroic.

Results

The biggest numbers for all the competitors come on the downstream side (where your laptop or phone is downloading something, rather than uploading it). This is because the AP is generally going to have a more powerful transmitter than your portable device does. On the upstream side, you'll tend to lower numbers due to the lower transmit power on your mobile device... but there's still a lot of room for variation, since the access points can have varying degrees of input sensitivity.

Be careful reading these graphs. I deliberately omitted the "Mbps" from the labels on the left axis, because you can't really add all of the colored blocks together and claim that Orbi has "1433 Mbps download." Each colored block represents a separately taken test reading, and none of these devices would be capable of anything like these speeds if tests were being run simultaneously from all test sites. I've added in the actual value in Mbps for Office 1 on each chart. Office 1 is a straight shot to the base router from roughly 25' away, so this is representative of a real-world "nearly best case" connection. Yes, you could get even higher numbers camped out about 7 feet from the router‚—but let's face it, that's a pretty narrow use case.

Unfortunately, these numbers don't compare to our previous testing very cleanly. This is partly because I got more rigorous with test locations since vendors were complaining, and partly because the last article only ran iperf3 in one direction (which, somewhat surprisingly, is upstream). The simplest invocation—iperf3 -c ip.address.of.server—uploads data rather than downloading it. This pretty heavily favored Plume, since Plume and AmpliFi HD are the only competitors with nearly equal throughput in either direction as tested.

More details are coming, but one important quick takeaway is that "Easy Mode" means different things for different competitors. Plume didn't actually get an "Easy Mode," because it roamed properly and rapidly everywhere. Orbi's "Easy Mode" was the addition of a second satellite AP, which was ridiculous overkill in the 3,500-square-foot test house. Google Wifi, AmpliFi HD, and Eero all received quite a bit more cajoling in Easy Mode. This could mean running a few extra tests at a site where roaming happened late or manually unplugging and re-plugging my USB3 test wNIC when roaming went badly. In AmpliFi HD's case, Easy Mode even involved some rather heroic measures and a lot of cursing.

Google Wifi and AmpliFi HD both unintentionally hamstrung themselves with the company recommendations to place an AP downstairs. Bedroom 3, Office 2, and the car are all nearly invisible for both kits on Hard Mode—and placement is the major reason. Downstairs clients tended to prefer to make their own connection directly to an upstairs AP, even when poked and prodded at. Remember, that connection was farther away since, rather than having an upstairs AP directly above Office 2 and Bedroom 3, there was now only one upstairs AP farther away from both. There were also quite a few connections from upstairs clients to the downstairs AP, which muddied things up even worse.

Channel Ars Technica