DRS-VPT: Directly Relocalizing in a Scan with Vision Point Transformers

Fu, Lanke Frank Tarimo; Fallon, Maurice

ECCV 2026 · Malmö, Sweden

DRS-VPT: Directly Relocalizing in a Scan with Vision Point Transformers

Lanke Frank Tarimo Fu^● Maurice Fallon^●

^● Oxford Robotics Institute, University of Oxford · {fu, mfallon}@robots.ox.ac.uk

Paper (PDF) arXiv BibTeX Attention explorer Code

Left: the inputs, a camera image and a LiDAR scan with unknown pose. Middle: the camera pose read out from each transformer block, then refined coarse to fine; once it converges, the scan is painted by the image. Right: the scan projected into the image at each pose.

Abstract

We present DRS-VPT, a feed-forward transformer architecture for foundational image-to-scan registration. Given query images and a reference 3D point cloud, the model predicts the scan pose and point map alongside the poses and point maps of each camera, all expressed in the first camera's frame. It additionally predicts a coarse-to-fine pyramid of per-point and per-pixel features for direct reprojective alignment of the scan to the first image. This formulation unifies downstream tasks such as camera–LiDAR calibration in autonomous driving and indoor camera-to-map relocalization. A single DRS-VPT model achieves state-of-the-art performance for image-to-LiDAR registration in autonomous driving, competitive indoor relocalization without training map-specific weights, and strong zero-shot transfer to unseen environments. We also show qualitatively that the model learns complex scan-to-image projection properties such as occlusion of back-facing points.

99.73%

KITTI accuracy (2 m / 5°)
camera–LiDAR alignment

5×

lower translation error vs.
ICP / Kabsch refinement

66.7%

recall, zero-shot
KITTI→nuScenes transfer

single model — driving
+ indoor, no per-scene weights

Demo

Input cloud · unposed · no color

Raw Argoverse camera frame used as the DRS-VPT query image.

Input image

scroll or pinch to zoom Output · Aligned · Saliency

The reference LiDAR sweep projected onto the query image, coloured by depth.

Projected overlay

Method

DRS-VPT architecture diagram: DINOv2 and Sonata encoders feed an alternating global/local attention transformer, which drives pose, point-map and coarse-to-fine alignment-feature heads, refined by unrolled differentiable Gauss-Newton alignment. — **Fig. 2 — DRS-VPT architecture.** Query images and a reference 3D point cloud are encoded with DINOv2 and Sonata. Alternating global cross-modal and per-frame local attention fuse the tokens, from which the model predicts poses, dense point maps, and coarse-to-fine alignment features for both modalities — refined by differentiable direct image–scan alignment into the final cloud pose.

End-to-end walkthrough on a real scene: image and unaligned cloud input, per-level image/cloud features, learned weights and feature similarity, the regressed output pose, and the refined pose after differentiable alignment. — **The pipeline on a real scene.** From an image and an unaligned, colourless point cloud, DRS-VPT extracts per-level image and cloud features and confidence weights, computes their cross-modal similarity, and regresses an initial pose — which the differentiable alignment stage then refines into the final registered pose.

Raw camera image for the feature-space scene. — **The shared image–cloud feature space.** Each image pixel and each cloud point is projected into the same learned feature space. Hover the cloud (bottom-left) to probe a point's feature; the top-right panel overlays the image with similarity to that feature in the same embedding — the cross-modal correspondence DRS-VPT's confidence weighting is built on. Bottom-right shows the image's own top-3 PCA components for reference.

Results

State-of-the-art alignment, competitive relocalization, strong transfer

A single DRS-VPT model is evaluated, unmodified, across four settings — wide-baseline camera–LiDAR alignment, indoor camera-to-map relocalization, zero-shot indoor transfer, and cross-dataset driving transfer.

A driving-scene LiDAR sweep and camera image from Argoverse, with the model's learned scan point weight overlaid on both the raw cloud and the projected points, from low to high. — **Wide-baseline camera–LiDAR alignment.** Given a raw LiDAR sweep and an unregistered camera image, DRS-VPT learns to up-weight the scan points that are most informative for alignment (yellow = high weight) — concentrated on structure visible to the camera.

Camera–LiDAR alignment under wide initialization

Translation t [m] and rotation r [°] error (mean / median); accuracy is % of predictions within 2 m / 5°. Scan perturbed by up to ±10 m with no pose initialization.

Method	KITTI t [m]	KITTI r [°]	KITTI Acc	nuScenes t [m]	nuScenes r [°]	nuScenes Acc
CorrI2P	3.78 / –	5.89 / –	72.42	3.04 / –	3.73 / –	49.00
VP2P	1.91 / 1.23	5.17 / 2.69	64.13	0.89 / –	2.15 / –	88.33
CoFiI2P	0.32 / 0.27	1.27 / 1.06	99.41	1.21 / 0.75	2.54 / 1.70	90.87
TrafficLoc	0.17 / 0.14	0.82 / 0.55	99.18	2.26 / 0.23	5.68 / 1.63	90.06
DRS-VPT (Ours)	0.09 / 0.05	0.34 / 0.25	99.73	0.33 / 0.13	0.67 / 0.33	97.76

Indoor camera–map localization — 7-Scenes

Translation error [cm] / rotation error [°] per scene. Baselines train a scene-specific model per map; DRS-VPT ingests the 3D map dynamically at inference with one shared model.

Method	Chess	Fire	Heads	Office	Pumpkin	Kitchen	Stairs	Recall↑
InLoc	3/1.05	3/1.07	2/1.16	3/1.05	5/1.55	4/1.31	9/2.47	66.3
DSAC*	2/1.10	2/1.24	1/1.82	3/1.15	4/1.34	4/1.68	3/1.16	85.2
PixLoc	2/0.80	2/0.73	1/0.82	3/0.82	4/1.21	3/1.20	5/1.30	75.7
ACE	2/0.68	2/0.87	1/0.67	3/0.84	4/1.15	4/1.34	5/1.25	75.6
DRS-VPT (Ours)	3/0.84	2/1.01	2/1.00	3/0.88	7/1.67	5/1.30	27/3.26	66.4

Zero-shot indoor relocalization — 12-Scenes

Never seen during training. Median translation [cm] / rotation [°] error and recall at 5 cm / 5°, aggregated per scene.

	apt1	apt2	office1	office2	Overall
Trans. (cm)	4	5	4	7	–
Rot. (°)	1.92	2.23	1.82	2.78	–
Recall (%)	56.8	48.9	62.0	33.7	53.6

Cross-dataset generalization — KITTI → nuScenes

Baselines trained only on KITTI, evaluated zero-shot on nuScenes. DRS-VPT trained on the full multi-dataset corpus with nuScenes explicitly excluded.

Method	t [m]	r [°]	Recall (%)
TrafficLoc	16.55	95.55	1.9
CoFiI2P	17.46	93.40	2.9
DRS-VPT (Ours)	0.538	1.14	66.74

Bar chart comparing in-distribution KITTI accuracy against out-of-distribution nuScenes accuracy for TrafficLoc, CoFiI2P, and DRS-VPT: prior methods collapse from ~99% to under 3%, while DRS-VPT holds at 66.7%. — Prior methods score >99% in-distribution on KITTI but collapse to under 3% recall on unseen nuScenes; DRS-VPT's multi-dataset training holds up at 66.7% under the same shift.

Emergent behavior

Learned occlusion awareness, without ever being supervised on it

Without any explicit visibility labels or occlusion supervision, DRS-VPT's confidence weights learn to suppress scan points that are geometrically present but hidden from the camera's current viewpoint — purely as a byproduct of the reprojective alignment objective.

An Oxford Spires facade: camera image, LiDAR input, LiDAR normals, and a side-on view of the learned scan point weight showing the front-facing wall weighted high and the points behind it weighted near zero. — **Emergent camera–point visibility (Oxford Spires).** Viewed edge-on (bottom-right), the learned weight cleanly separates the front-facing wall (high weight) from the points immediately behind it (near-zero weight) — the model discovers back-face occlusion on its own.

Qualitative

Learned saliency across sensors and scenes

The same confidence-weighting behavior holds across very different sensors and environments — structure-from-motion facades, legged-robot LiDAR sweeps, and RGB-D indoor maps — with high saliency consistently landing on geometrically distinctive structure.

Learned feature saliency and confidence weighting on an ETH3D facade: input image, per-point feature colouring, and confidence-weighted point cloud across pyramid levels.

Learned feature saliency and confidence weighting on a GrandTour legged-robot LiDAR sequence: input image, per-point feature colouring, and confidence-weighted point cloud across pyramid levels.

Learned saliency and occlusion weighting (ETH3D facade, left; GrandTour legged-robot LiDAR, right). The model assigns high saliency to geometrically distinctive structure, and its confidence weights suppress occluded scan points — without ever being told which points are visible.

Qualitative result on a 7-Scenes staircase: camera input, RGBD map input, map normals visualization, and the learned scan point weight from two viewpoints.

Indoor RGB-D map (7-Scenes, Stairs). Camera input, the RGB-D map it is registered against, a surface-normals visualization of that map, and the learned per-point scan weight — concentrated on the stair edges most useful for alignment.

Training corpus

Trained across sensors, scales, and environments

One foundational model, trained across ten multi-modal datasets spanning structure-from-motion reconstructions, LiDAR scans, and dense laser/depth-camera maps — indoor, outdoor, and large-scale driving.

MegaDepth Oxford Spires KITTI Argoverse nuScenes PandaSet ETH3D ScanNet++ 7-Scenes WildRGBD GrandTour

Cite

BibTeX

@inproceedings{fu2026drsvpt,
  title     = {{DRS}-{VPT}: Directly Relocalizing in a Scan with Vision Point Transformers},
  author    = {Fu, Lanke Frank Tarimo and Fallon, Maurice},
  booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
  year      = {2026},
  eprint    = {2609.12557},
  archivePrefix = {arXiv}
}