TLDR: We introduce the first deep neural network that can accurately localize an image using the same 2D semantic maps that humans use to orient themselves. OrienterNet leverages free and global maps from OpenStreetMap and is thus more accessible and more efficient than existing approaches.
Abstract
Humans can orient themselves in their 3D environments using simple 2D maps. Differently, algorithms for visual localization mostly rely on complex 3D point clouds that are expensive to build, store, and maintain over time. We bridge this gap by introducing OrienterNet, the first deep neural network that can localize an image with sub-meter accuracy using the same 2D semantic maps that humans use. OrienterNet estimates the location and orientation of a query image by matching a neural Bird's-Eye View with open and globally available maps from OpenStreetMap, enabling anyone to localize anywhere such maps are available. OrienterNet is supervised only by camera poses but learns to perform semantic matching with a wide range of map elements in an end-to-end manner. To enable this, we introduce a large crowd-sourced dataset of images captured across 12 cities from the diverse viewpoints of cars, bikes, and pedestrians. OrienterNet generalizes to new datasets and pushes the state of the art in both robotics and AR scenarios.
1) Given an input image, OrienterNet first infers a mental map of the scene as a neural Bird’s-Eye View (BEV). 2) Given a coarse GPS prior location, we query OpenStreetMap and compute a neural map. 3) Matching the BEV against the map yields a probability volume over 3-DoF camera poses.
Demo
BibTeX Citation
@inproceedings{sarlin2023orienternet,
author = {Paul-Edouard Sarlin and
Daniel DeTone and
Tsun-Yi Yang and
Armen Avetisyan and
Julian Straub and
Tomasz Malisiewicz and
Samuel Rota Bulo and
Richard Newcombe and
Peter Kontschieder and
Vasileios Balntas},
title = {{OrienterNet: Visual Localization in 2D Public Maps with Neural Matching}},
booktitle = {CVPR},
year = {2023},
}