Latest Trends in AR and VR Sensors

New trends in VR device sensors

Demand for AR, VR, and MR devices is expected to grow significantly with the growth of the metaverse market. Currently, gaming and industrial applications are the main focus. According to a study published by International Data Corporation (IDC) in June 2022, the VR headset market is expected to grow from approximately 10 million units in 2021 to more than triple that by 2026. Sensor technology is an important part of this prevalent trend. Sensor technology is not essential in head-mounted display to watch movies or browse videos. However, it becomes critical for AR/VR devices to play games or do work. This article covers recent trends in sensor technology used in AR, VR, and MR devices, as well as potential future developments.

The difference between 3DoF and 6DoF

Head-mounted displays (HMDs) project images according to the user’s movement. This is made possible by a system in VR devices that detects the orientation and rotational speed of one’s head.

VR device systems that detect head orientation are classified as ‘3DoF’ or ‘6DoF,’ depending on the movement and direction detected by the sensor. DoF (Degrees of Freedom) refers to how much a sensor can perceive. The higher the degrees of freedom (axes) that these sensors can detect, the more immersive the VR experience.

3DoF means that there are three axes that can be detected: X, Y, and Z axes. The system is able to recognize whether the user is looking up/down, left/right, or diagonally down/up. 3DoF-compatible HMDs are mainly used for viewing 360° images. Compared to 6DoF, it has fewer functions and allows users to experience VR at a lower cost. In addition, less space is required since there is no need for sensing hands or feet movements. It also allows multiple people to view the VR at the same time. Because of these features, 3DoF-compatible HMDs are being used in amusement parks, museums, and corporate training programs.

In contrast, 6DoF has six axes of freedom that the sensor can detect. As shown in the illustration below, in addition to head movement, the system also detects the user’s movement in the X, Y, and Z-axis directions. Specifically, it is able to recognize back/forth, left/right, and up/down movements.

Due to the increased number of sensors, 6DoF VR devices are more costly than 3DoF VR devices. It also requires a larger amount of space than is required for 3DoFs since it is able to detect both body and head movements. However, 6DoF VR devices can provide a more immersive experience for the user. The ability to provide an environment that more closely resembles reality makes them suitable for realistic simulation training that involves hand and body movements.

3DoF is sufficient if the main purpose is to view and operate in 360° video recordings and computer graphics. However, a device equipped with 6DoF is required to enjoy a VR experience with the freedom to move (i.e. walking, crouching, and moving your hands).

Outside-in and inside-out tracking technology

The function that captures the movement of the head and neck using sensors installed in a HMD is called “head tracking”. When a user looks up, down, left, or right, the sensor recognizes the movement and displays an image that matches it. This allows the user to get a 360° view within the computer graphic VR environment. 3DoF devices detect the orientation and rotational direction of the head by using the acceleration, gyro, and magnetometer sensors installed in the HMD.

On the other hand, in 6DoF devices, sensors are required to detect the user’s head position and movement. There are two types of systems:

1. Outside-in: a separate tracking camera is installed above the TV or somewhere in the room to detect head and body movements.
2. Inside-out: a sensor is built into the HMD or goggles.

For the outside-in type, the cameras and sensors are located outside, so the user must be within a limited area that allows them to be detected. On the other hand, the inside-out type uses cameras and sensors installed in VR device to capture, measure, and analyze external images to determine the head orientation and direction of movement. Therefore, there are no space restrictions and users can move freely around a room.

Position tracking

Most VR devices currently on the market allow the user to control a VR avatar by using a hand-held controller. Conventional VR devices with optical sensors emit infrared rays outward, which are received by an external sensor to determine the user’s position. This requires the installation of an external sensor in the room, as well as a cord to connect the sensor to the headset. On the other hand, there are now devices that do not require external sensors or cords, enabling VR experiences with just a headset and controller. By making both the HMD and the controller compatible with 6DoF, it is possible to automatically combine the user’s real-world movements with their VR avatar.

Such devices use the Time-of-Flight (ToF) method that measures the distance to an object by the time difference between light, infrared rays, ultrasonic waves, etc. emitted from the headset or controller, and the time it takes for the light to be reflected back from the object. In recent years, this technology is being used in systems for detecting obstacles in self-driving vehicles.

Hand tracking revolutionizes VR device operations

Currently, mainstream VR devices use a hand-held controller to control avatars. However, hand tracking technology is being developed as a method to enable a more immersive experience.

Hand tracking is a method where a camera attached to an HMD recognizes the user’s hands and reflects their position in the VR or MR avatar’s hands. This eliminates the awkwardness of operating an avatar with a controller. Being able to express movements down to one’s fingertips, such as waving or playing rock-paper-scissors, also enables communication through fine hand movements.

In addition, if the hand tracking function eliminates the need for a controller, the hassle of charging and pairing controllers will also be eliminated, making the VR experience more convenient. It would also eliminate hand fatigue caused by holding the controller and reduce manufacturing costs. In the future, users may be able to intuitively control VR spaces, such as pausing and fast-forwarding a video through hand gestures.

Hand tracking utilizes multiple cameras and AI technologies to accurately determine the position and movement of one’s fingers. Furthermore, technology that reproduces the sensation of touch by using a glove controller (haptics technology) is in development. There is potential for various hand-related technological developments in VR.

Eye tracking to reflect a person’s line of sight in VR

Along with the use of hands in VR operations, eye tracking sensors that utilize human eye movements are also advancing. Eye tracking sensors are installed on the inside of a VR device to capture the user’s eye movements to reflect the user’s line of sight in VR.

One of the drawbacks of the VR experience is ‘VR sickness,’ similar to motion sickness. However, eye tracking can help alleviate this issue. By reflecting eye movements in the virtual space, users can gaze at other avatars, make eye contact, or aim at a target in a shooting game using only their line of sight. Current eye tracking sensors use a near-infrared camera to capture the reflection points of light on the cornea and eyeball to estimate the direction and position of gaze.

Eye tracking enables realistic visual worlds

When a person focuses on one thing, the surrounding area is blurred. However, with ordinary VR devices, all of the projected images are clearly in focus. This causes a phenomenon in which the viewer feels a difference from reality, reducing their sense of immersion. The eye tracking function is effective in eliminating this sense of discomfort.

Further developments in eye tracking will also enable a technology called ‘forbidden rendering,’ which uses a near-infrared camera to detect the user’s eye movements and project high-resolution images only in the vicinity of the field of view.

Rendering only the areas that the user is focused on at a high resolution in addition to rendering at a lower resolution as one moves outwards, could significantly reduce the computational burden on the device and greatly improve the overall graphics quality. Eye tracking technology is expected to enable visual expressions closer to real life, leading to improved and higher visual quality VR experiences.

Dexerials’ products for eye tracking sensors

Dexerials’ Anisotropic Conductive Film (ACF) and Anti-reflection Film (ARF) contribute to the development and improved quality of eye tracking sensors.

Eye tracking sensors are often placed around the eye of the device. When sensors use film substrates, they are usually optically transparent. FPCs (flexible printed circuit boards) are sometimes bonded on optically transparent films for the signal wiring of infrared LEDs, and an increasing number of customers are adopting Dexerials’ ACFs for this purpose. Eye tracking sensor modules can be thinner, smaller, and lighter when bonded by ACF. In addition, Dexerials’ anti-reflection and moth-eye type anti-reflection films can be used to suppress the light reflection inside the housing.

VR devices with more than 10 cameras and sensors

In the future, there will be VR and MR devices that use multiple sensors and camera modules to combine external images with images in VR. Such devices may be equipped with 10 or more cameras and sensor modules. Dexerials’ ACFs and UV- and thermosetting adhesives used to attach lenses can contribute to the development of VR and MR devices.

Furthermore, through the improvement of various tracking technologies, future VR and MR devices will be able to precisely recognize the fine movements of a person’s head, neck, body, gaze, hands, etc., while generating images, sounds, and tactile sensations. Dexerials will continue to contribute to the development of new devices for the metaverse.