Google’s new Mixed Reality tech aims to create a more communal experience by removing the obfuscating elements of VR headsets.
Google Research and Google Daydream Labs say that the new the new tech will overlay VR headsets with the users face, giving the experience a much more human touch. Currently VR is great for the person inside, but spectators can have a hard time trying to gauge what’s actually happening.
The new tech reportedly uses a combination of machine learning, 3D computer vision, and advanced rendering techniques to ostensibly “remove” the headset obscuring the users face.
Via Google Blog: VR user captured in front of a green-screen is blended with the virtual environment to generate the MR output: Traditional MR output has the user face occluded, while Google’s result reveals the face. Note how the headset is modified with a marker to aid tracking.
Avneesh Sud, Software Engineer, Google Research, said: “If you’re watching someone else use VR, it’s hard to tell what’s going on and what they’re seeing. And if you’re in VR with someone else, there aren’t easy ways to see their facial expressions without an avatar representation.”
“Mixed reality is a way to convey what’s happening inside and outside a virtual place in a two-dimensional format. With this new technology, we’re able to make a more complete picture of the person in VR.”
The way the technology works is by creating a 3D dynamic model of the users face to accurately portray any facial expressions or emotions, and then layer it onto the front of the digital headset. This allows spectators to see the users face onscreen despite it being hidden under a VR helmet.
The virtual world is then rendered on a green screen, whilst the video feed of the rest of the body is captured with a camera. For anyone watching on another screen the result is a complete version of the user and the world they’re inhabiting, with their face free of obstruction.
Content from our partners
The technology is a long way from public release however, and the requirement of a video camera, a green screen, and a VR headset itself is a high barrier to entry. The tech is also not currently running on Google Daydream and instead uses a modified HTC Vive.
Sud said: “Our initial work focused on mixed reality is just one potential application of this technology,” adding “It’s just the beginning for this technology.”
