Tech —

Microsoft was leading the world in AR; now it’s at risk of being left behind

HoloLens was inspirational, but it’s ARKit and ARCore that are going to win developers.

The HoloLens headset.
The HoloLens headset.

Stop me if you think you've heard this one before: a Redmond-based software company is an early player in a brand-new market, then finds itself displaced and left behind after late-coming competitors bring to market similar products in a way that captures the mainstream audience.

We saw this with smartphones: Microsoft's various Windows Mobile offerings built a small but loyal following, but the company was too slow to acknowledge the widespread appeal of touch-first interfaces and didn't recognize the power of Android's zero-cost licensing. Microsoft eventually built a good product and was even carving out double-digit market share in Europe, but it subsequently floundered, and the company has now all but disappeared from the market.

The surprise reveal of Microsoft's HoloLens back in 2015 caught the world off guard. The product hadn't leaked, and it offered something genuinely new: a self-contained headset that mixed 3D computer graphics with the real world, offering the kind of augmented reality experience that had hitherto been the sole preserve of science fiction. Unlike virtual reality headsets, which completely occlude your view, HoloLens lets you see the whole world in all its glory, obscuring only those parts that were hidden behind virtual objects. It borrowed from work done with Kinect, containing complex systems for mapping and tracking the room and objects around you. The setup enabled full, untethered operation without needing any base stations or object markers. As a melding of the virtual and the real, the HoloLens was thoroughly convincing.

But is Microsoft going to be able to turn this leadership in innovation into leadership in market share? That's where things look challenging.

Based on what it has learned from HoloLens, Microsoft has built what it calls the Windows Mixed Reality platform. "Mixed Reality" is important here; Microsoft's platform spans a wide range of devices, from vision-occluding virtual reality headsets, to augmented overlay devices like HoloLens, to through-the-camera systems on tablets. Redmond's platform includes the basic building blocks needed for building both augmented and virtual reality applications: motion and object tracking, 3D motion input, and (of course) 3D graphics. With the news yesterday that SteamVR titles would also run on Microsoft's Mixed Reality platform, the company looks, in some regards, to be well-positioned for this new mixed-reality world.

There is, however, a big gap in Microsoft's product line-up and a big question mark over its potential marketplace uptake. First, the question mark: does the world want headsets? Microsoft's platform is driving the cost of those headsets down—they should start at about $300, compared to $500-$600 for first-generation, PC-based headsets. The platform is also making the headsets much simpler—no base stations to stick on the walls, just plug in the headset and go. But it still means buying an extra accessory for your PC.

The alternative to a headset is a through-the-camera AR system—the kind of thing done by Pokémon Go, wherein computer-generated graphics are overlaid on a feed from a world-facing camera, using a phone (or tablet) as a kind of viewport into a virtual world. Headsets are superior in all manner of ways—they're more natural to use, they leave your hands free for manipulating objects in 3D space, they can offer bonuses such as positional audio and retinal tracking, and they're much more immersive—but they have one huge downside: you have to buy a damn headset. You don't have to buy anything extra for a through-the-camera system, by contrast, because you already have a smartphone that does everything you need for a through-the-camera system.

<em>Pokémon Go</em> wasn't the first to put virtual objects like this adorable Pidgey in the real world, but it's probably the best-known software to do so.
Enlarge / Pokémon Go wasn't the first to put virtual objects like this adorable Pidgey in the real world, but it's probably the best-known software to do so.

Apple's ARKit and Google's ARCore give developers the tools they need to develop through-the-camera AR software for iOS and Android phones. These tools give developers a target of billions of devices without requiring any end-user hardware purchase.

The problem here for Microsoft isn't the technology. The fundamental trick for smartphone AR—using camera input to correct the accumulated errors that come from double integration of accelerometer data and hence offer six-degrees-of-freedom motion tracking without the need for fixed external references—is the same one that Microsoft is using for its cheap VR headsets. Microsoft has demoed through-the-camera AR, the Windows Mixed Reality platform supports it, and—while there will inevitably be differences between ARKit, ARCore, and Windows Mixed Reality—all three platforms are going to have broadly similar capabilities.

Rather, the problem is simply that Apple and Google have reach on smartphones; Microsoft doesn't. This represents an enormous gap in the reach of the Windows Mixed Reality platform.

Microsoft's position in the headset space is certainly stronger, though still complex. VR has more stringent demands on performance, due to motion sickness concerns, so while smartphone-based VR systems (Samsung's Gear VR and Google's Daydream) exist, they're somewhat more constrained in terms of what they can do than PC-based systems. Moreover, even smartphone-based systems require some kind of accessory to be purchased, due to the need to mount a phone to your face.

While these accessories are cheaper than full VR headsets (with Samsung in particular offering regular promotions that bundle the Gear VR for free with a new phone), they still represent something of a roadblock to wider adoption.

The need for an accessory is something of an equalizer between the phone and PC; the better performance arguably tilts things in the PC's favor. As such, we wouldn't expect Microsoft to be completely shut out of the virtual reality future. But the company's reach into the augmented reality space looks limited.

Stuck in the middleware with you

Redmond is not entirely without options, of course. Building a must-have phone is, at this point, unlikely. But there is a tried-and-true way of diminishing the value of platforms: middleware. That is, building a software library that provides services to applications and hides or abstracts away the specifics of the underlying platform.

Microsoft, as a platform vendor, has traditionally been hostile to middleware; the company saw platforms such as Java and the Web as threats that undermined the value of the Windows platform by offering a new platform that allowed developers to write software that could run just as well on Windows, Macs, or Unix machines. But that hasn't stopped the company developing and buying its own middleware: .NET and the Xamarin stack both fit this category. Expanding these to offer AR middleware—something that let developers span both ARKit and ARCore and which just so happened to work with Windows Mixed Reality, too—could ensure that Microsoft remains relevant in this new space, even in spite of its weakness in the phone market.

It wouldn't tremendously surprise us if this is the path that the company ultimately takes, but making a success of this approach would require Microsoft to do another thing that it has often struggled with: move fast. It's inevitable that this augmented reality middleware will be built, as developers are sure to want to avoid rebuilding their software for ARKit and ARCore. We know that big players already have plans in this space, too, with Facebook promising to build some kind of AR middleware. It's not inevitable, therefore, that Microsoft's middleware will be the middleware of choice: the company needs to bring something to market fast, and it needs to make sure that it delivers something good.

If Microsoft doesn't, well, it's all too easy to see history repeating itself, with those initial HoloLens forays failing to provide any lasting market penetration.

Here's our firsthand view of the then-awe-inspiring HoloLens in action from 2016. Filmed by Esy Casey, produced and edited by Nathan Fitch

Channel Ars Technica