Microsoft’s DreamWalker VR turns your daily commute into a totally different one

Researchers at Microsoft have developed new VR technologies that they claim allow users to remain fully immersed in a virtual world even while traversing public places in the real world on foot.

Microsoft describes the project, titled DreamWalker, as "a method for allowing people to safely navigate a given route in real-world environments, such as a daily walk to work, while seeing themselves strolling a different VR world, such as a city of their choosing." It was developed by researchers Jackie Yang, Eyal Ofek, Andy Wilson, and Christian Holz.

The company's research division published a blog post about the new research yesterday, and researchers will present the method at the ACM Symposium on User Interface Software and Technology tomorrow.

The real-world path and corresponding virtual environment are planned in advance based on geolocation data, then updated on the fly as required in an uncontrolled outdoor environment. Any potential obstacles the user encounters while traversing real space are recorded by real-time sensing technologies in the VR apparatus, including a dual-band GPS sensor, two RGB depth cameras, and "a Windows Mixed Reality-provided relative position trace." Those obstacles may be replaced by obstacles in the virtual world, such as road blocks. Additionally, a video-game quest marker-like arrow will direct the user in what is deemed to be a safe and efficient traversal direction.

"Discovered obstacles that may move or appear in users' paths are managed by introducing moving virtual obstacles, or characters, such as pedestrians walking near users, to block them from any potential danger," the blog post explains. "Other options for controlling users' paths may include pets and dynamic events such as vehicles being parked, moving carts, and more, limited only by the imagination of the experience creator."

The system tries its best to introduce these virtual objects outside of the user's field of view to minimize unrealistic popup, not dissimilar to how a 3D video game environment uses streaming and frustum culling to maximize performance or introduce new assets to the scene.

DreamWalker actually uses another Microsoft Research-developed tech called Mise-Unseen—which "allows covert changes to be made to a VR environment while a user's gaze and attention are fixed on something else"—to do this thanks to eye-tracking tech.

While DreamWalker had a predecessor project called VRoamer that attempted to deliver safe experiences in uncontrolled indoor environments, most applications of VR being researched today or sold to consumers or businesses require a controlled space for safe use. DreamWalker is not ready for the public yet, but it's a step toward unexpected applications of VR that may have previously only been considered for AR—though there's a lot of crossover in AR and VR research and product development across the industry.

When Google Glass was first introduced many years ago, some commentators worried about the privacy and safety implications. Several Pokémon Go-related accidents introduced more questions about how technology companies can act responsibly when selling AR or VR products or experiences and what municipalities and legislators need to be thinking about moving forward. And with Apple expected to launch AR glasses as soon as next year, these questions are only going to become more pressing. It was enough to think about safety when using AR in public spaces, but the potential challenges and applications for VR in public will need further consideration, too.

Microsoft's researchers have published a paper with more technical details if you want to learn more about how it works.