BMW Group's US Technology Office in Mountain View presents an update on research in collaboration with Meta's Reality Labs Research, which demonstrates, for the first time, the ability to accurately project fixed virtual reality (VR) and mixed reality (MR) content to the occupants of a fast-moving car, even when turning, passing bumps and accelerating. By integrating Meta Quest's tracking system with BMW's sensor data, the researchers activated a series of fixed gaming, entertainment, productivity and meditation experiences "locked in the car."
Claus Dorrer , Head of the Technology Office of the BMW Group in the USA at Mountain View: "It's too early to say exactly how or when this technology will get into the hands of customers, but we envision a number of potential use cases for XR devices in vehicles - from helping drivers locate their car in a crowded parking lot to warning of road hazards and displaying important information about the vehicle's condition. The prospects for future AR glasses and VR devices - for both passengers and drivers - are promising. The research collaboration with Meta will allow us to discover what immersive in-vehicle XR experiences could look like in the future and pioneer the seamless integration of such devices into cars."
Prototype results overcome key technical challenges
VR devices use cameras and motion sensors to understand their exact position, so when the user moves their head or looks around, the virtual content remains constant. However, in fast-moving vehicles, the sensors are confused by conflicting signals from the device's movement in the car and the vehicle's movement in the real world. Simply put, the devices' motion sensors think they are moving, but their cameras think they are stationary when looking inside the car.
This mismatch means that today's VR devices cannot display stable virtual content when travelling in a fast-moving vehicle, especially if the vehicle is accelerating, turning quickly or passing bumps in the road.
To solve this challenge, BMW and Meta researchers collaborated to integrate information from the sensor array of a BMW car into the tracking system of a Meta Quest Pro. This enabled the tracking system to simultaneously calculate the position of the device relative to the car and the position of the moving car relative to the real world, allowing for the accurate presentation of virtual objects in a moving car and the demonstration of immersive VR and MR experiences for passengers.
Richard Newcombe , Vice President for Scientific Research, Reality Labs Research : "Our research prototype shows that we can enable fun and comfortable experiences for passengers that are connected to the car itself, including VR and MR games, entertainment, productivity, and even meditation opportunities. The technology has the potential to transform the way we can safely interact with our surroundings while traveling, and as we move to trusted content when transitioning to AR glasses, we hope that passengers will be able to see things like markers for landmarks, restaurants, points of interest and more."
Combining the motion data generated by the vehicle with the capabilities of the XR device sensor allowed us to abstract the vehicle's motion and more accurately track the movement of devices within a vehicle. This in turn allowed us to render content that is synchronized with vehicle motion in real-time, both inside and outside the vehicle at high speeds, regardless of whether the vehicle is on a flat, straight road or on a rough road with sharp turns.
The results of the prototype result in a full range of motion of a person in the car (known as six degrees of freedom) and an XR experience that simultaneously calculates the position of a device relative to the car and the position of the moving vehicle relative to the real world, incorporating the car's sensors.
The BMW Group is a pioneer in digital technologies, such as access to the vehicle via the digital key, and in promoting industry standardisation for such innovations. The company's global network of technology offices has played a key role in promoting such innovation projects. The BMW Group is open to new partners in order to achieve the goal of creating an industry standard for connecting XR consumer electronics to vehicles. For possible development in series production, the BMW Group is committed to the principle of privacy by design. Throughout the digital feature development process, the BMW Group takes steps to ensure that customer data is protected and processed securely in accordance with data privacy requirements through established procedures - in all markets where the company operates.
