Eye tracking is the next phase for VR, ready or not

Virtual reality seems like it's undergone a dramatic refocusing over the past year: away from PCs, and toward do-it-all self-contained headsets like the groundbreaking Oculus Quest.

A very Oculus Quest-like standalone VR headset ended up on my face at CES in Las Vegas, one of a number of VR demos I tried at the show. The Pico Neo 2 Eye may never be a headset you end up buying, but it's an interesting sign of the near future. It's a standalone VR headset, like Facebook's excellent Oculus Quest. But yes, it also tracked my eyes.

Tobii, a Swedish technology company focusing on eye tracking, has been working on integrating its hardware into a variety of VR and AR headsets, and even computers, for years (Dell's Alienware series uses Tobii tech). The HTC Vive Pro Eye, released last year, was the first major VR headset to have Tobii's eye tracking built-in, but it was designed for business use, and was extremely expensive. The Pico Neo 2 Eye is the first smaller-scale eye-tracking VR headset, using Tobii's tech and a Qualcomm Snapdragon 845 chip -- a combination I first saw a preview of two years earlier.

The Oculus Quest, amazing as it is, uses an even older Qualcomm Snapdragon 835 chip. The Pico VR headset has better graphics capability, on paper. Qualcomm's got even more advanced VR chips on the horizon: a newer XR2 chip promises not only better graphics, but a whole lot of simultaneous cameras that can track the outside world... and look inwards, tracking your face and eyes.

According to Tobii, eye tracking is an essential part of VR and AR going forward, but it's not going mainstream just yet. "We are working with the majority of manufacturers on incorporating eye tracking," Tobii's CEO and co-founder, Henrik Eskillson, told me. "The first devices were 2019, more devices in 2020. We expect it to hit the real consumer volumes, based on engagements we've had, in 2021. And then even more scale, and become more or less the de facto standard, [in] 2022, 2023." Enterprise and higher-end AR headsets like the HoloLens 2 and Magic Leap already have eye tracking, but at this point no mainstream AR/VR products do.

Eye tracking can be extremely helpful. Whatever I look at can be registered, which can certainly help apps focus on tuning controller input to match what I'm interested in targeting. Eye tracking and hand tracking could also combine in some interesting ways down the road. 

There's also a truly practical reason for eye tracking: it helps intelligently reduce graphics processing to only your fovea (called foveated rendering), creating an experience that looks the same, but taxes the processor a lot less. That means only the parts of the image you're looking directly at get the fullest resolution and image quality, everything else can save GPU cycles. 

A demo I tried with Tobii and the Pico Neo 2 Eye showed a room where graphics effects are applied, with and without eye tracking/foveated rendering. The frame rate ends up a lot smoother with eye tracking, which would also mean less lag, and less nausea. 

My thicker glasses sometimes don't work well with Tobii's inner rings of eye-tracking VR cameras, but hopefully future headsets will improve how the tech works with glasses like mine.

But also, with eye tracking, there are looming unsolved issues. Companies could know what you're interested in, for ad targeting, or anything else. Eye tracking could approach a type of mind reading. That brings up all kinds of privacy and data tracking questions. Do you want an ad tracker knowing exactly which parts of a webpage or video your eyes are focused on? How is that data shared, or connected with personally identifiable information? Will emotional states be able to be recorded -- or will companies think these eye movements indicate emotions? It's just another front in the war on digital behavior tracking. 

Oculus Quest uses a fixed type of foveated rendering that just makes the center of the display higher-res than the periphery. It also allows Facebook to skip eye tracking for now, and the looming question of how a company like Facebook will deal with eye tracking data. Facebook now uses Oculus VR activity for ad targeting if you're logged into Facebook on an Oculus headset. How would eye tracking be handled? Qualcomm, a company driving the ability to track increasing amounts of information via simultaneous cameras on its XR platforms, has a vision for handling the collection of this data, but it's not clear yet how that will play out, either.

There's also a matter of cost. The Pico Neo 2 costs significantly more with eye tracking (around $900) vs. the version without (around $700). And while eye tracking can help improve graphics performance, there's no definite word on how this will impact overall battery life on standalone headsets.

Still, I keep wondering when eye tracking will arrive on home VR headsets, and how it will be handled. That moment could be coming soon enough, but it could also refocus concerns on privacy and user tracking, too.