DRS-VPT · inside the transformer

Every pixel asks the map where am I?

Hover the image. In each global attention layer the patch under your cursor attends to all 5,000 map tokens; the map shows where that attention goes. Switch to features to see what the map tokens have become at that depth, with the image painted in the same colours.

query image

loading scene…

map · drag to orbit, scroll to zoom

global attention layer

What you are seeing

The transformer alternates global blocks, where image and map tokens attend to each other, with per-frame blocks that keep them apart. The scrubber walks the 12 global blocks; attention is averaged over heads.

Reading the colours

What happens with depth

Data

The scenes come from public datasets, used here for non-commercial research. Each keeps its source's licence; cite the original dataset if you use it.

Trafalgar Square: the map is fused from the SfM depth of MegaDepth photos of the National Gallery (Z. Li and N. Snavely, MegaDepth: Learning Single-View Depth Prediction from Internet Photos, CVPR 2018); none of those photos is shown. Query photo: “National Gallery Facade” by Nathan Stazicker, CC BY-SA 3.0, resized. The model predicts its intrinsics, and there is no ground-truth pose for it.
Stairs: 7-Scenes (J. Shotton, B. Glocker, C. Zach, S. Izadi, A. Criminisi and A. Fitzgibbon, Scene Coordinate Regression Forests for Camera Relocalization in RGB-D Images, CVPR 2013), © Microsoft Corporation, under the Microsoft Research License Agreement (non-commercial use only).
KITTI scenes: the KITTI odometry benchmark (A. Geiger, P. Lenz, C. Stiller and R. Urtasun, Vision meets Robotics: The KITTI Dataset, IJRR 2013), under CC BY-NC-SA 3.0. Test samples and their random scan transforms follow the VP2P evaluation split (sequences 09 and 10).