LAVER LABS

Edited version of demo for AR navigation in multiple underground floors in Gangnam Station (Links to original: Multi-floor route; Single-floor route)

Gangnam Station is considered to be most difficult to apply augmented reality (AR) technology in Korea; its low ceilings, multitudinous columns and maze-like arrays of stores as well as high contrast lighting interfere with the device’s “eyes” in identifying its exact location. Waves of people in Gangnam Station, which boasts the heaviest foot traffic flow and density in Korea, as well as the frequently changing store signs and facility interiors also pose a great challenge.

Extreme environment test amid waves of people, low ceilings and maze-like arrays of stores in Gangnam Station

Mapping/positioning technology responding to an extreme environment

1) Positioning technology responding to rapidly changing dynamic situations
NAVER LABS is researching and developing visual localization (VL), which can track six degrees of freedom (6DoF) poses with image sensors in the smartphone alone. While this VL tech allows you to measure the accurate location in indoor spaces, it is very difficult to position the location with images alone in Gangnam Station. This is because low ceilings, arrays of stores and crowds of people masking the background in Gangnam Station interfere with accurate positioning.

Example of creating a human mask to exclude people getting in the way of the camera and recognize background information

To solve this problem, we also applied semantic segmentation, which determines which area of the image each pixel belongs to. We created a mask image of “humans” which interfere with positioning, excluded them during positioning, and implemented more solid visual localization in the rapidly changing surrounding environment.

Furthermore, low ceilings and repetitive indoor structures in Gangnam Station are all recognized as similar to the machine’s eye. So, the traditional approach (i.e. NetVLAD) greatly decreases the accuracy of location information in terms of which field of view the image has. To overcome this issue, we developed image retrieval technology, which can additionally collect more detailed information for some part of the image and improve positioning accuracy.

(Top) Traditional NetVLAD-based results; (Bottom) NAVER LABS’s image retrieval-based matching results. The matching success rate improved greatly after using NAVER LABS’s tech.

2) Mapping technology in combination with different sensors
It is quite difficult to map narrow and long stairs or escalators, which you can often find in Gangnam Station. For instance, it is similar to not being able to know where your car is when you are driving through a long tunnel. The GIF image below shows how mapping can sometimes fail when using only traditional LiDAR data to map areas.

Example of a LiDAR raw scan on escalators, which shows almost no change in LiDAR data even moving

To overcome limitations in narrow and long sections of spatial environments, we had to use visual feature information, including signs and lines on walls, together from the camera. To do so, we newly developed an algorithm that combined LiDAR and camera data and were able to successfully map Gangnam Station despite some spatial obstacles.
 

3D point cloud video of multiple floors in Gangnam Station mapped by M1X/COMET

AR tracking technology taking a step closer to reality

There were other user environment variables to consider in complex spaces like Gangnam Station, such as people not holding their smartphone up in the air and other unexpected occurrences while on the move. We had to take into consideration cases where the camera looked at the floor or the ceiling when the smartphone’s camera angle changed or even deviated from the route. Below is the test video of AR navigation tracking under extreme conditions. It operates stably even in tough conditions where it is difficult to maintain tracking.

Test video of AR navigation tracking in real-life conditions

This demo conducted through an MOU with the Seoul Metropolitan Government implemented Gangnam Station’s various POIs so that you can easily get to stores, exits, and facilities.

By demonstrating Korea’s most difficult Gangnam Station, NAVER LABS was able to prove, once again, its indoor VL/AR technology in real everyday spaces. Building on the demonstration of our improved technology in indoor spaces, we will continue to expand NAVER LABS’s technological space by connecting even outdoor spaces.