Lingbot-Map

LingBot-Map is a feed-forward 3D foundation model for streaming 3D reconstruction. It takes a sequence of images or a video and produces camera poses and a dense point cloud reconstruction without per-scene optimization. It's built for researchers and engineers working on SLAM, visual odometry, robotics mapping, or 3D scene understanding who need fast, long-horizon reconstruction rather than slow offline bundle adjustment.

The model is implemented around a Geometric Context Transformer architecture that combines anchor context, a pose-reference window, and trajectory memory to keep pose estimates stable over long sequences. It uses paged KV cache attention (via FlashInfer, with an SDPA fallback) to run efficiently on streams exceeding 10,000 frames, reporting around 20 FPS at 518×378 resolution.

Key features

• Geometric Context Transformer: unifies coordinate grounding, dense geometric cues, and long-range drift correction in a single streaming model.
• Feed-forward streaming inference: no iterative per-scene optimization; predictions come from a single forward pass per frame with paged KV cache attention for memory efficiency.
• Keyframe interval control: --keyframe_interval reduces KV cache memory by only caching every N-th frame while still producing predictions for all frames, useful beyond the ~320-frame training window.
• Windowed inference mode: --mode windowed handles sequences over 3,000 frames by processing overlapping windows with configurable size and keyframe overlap.
• Sky masking: an ONNX sky segmentation model filters sky points from outdoor reconstructions, with automatic model download and mask caching.
• Interactive viser viewer: demo.py launches a browser-based 3D viewer for quick inspection of reconstructed scenes and camera trajectories.
• Offline rendering pipeline: demo_render/batch_demo.py runs inference on very long sequences (tens of thousands of frames) and renders a headless point-cloud flythrough video, bypassing the interactive viewer's length limits.
• Memory and speed tuning: flags like --offload_to_cpu, --num_scale_frames, and --camera_num_iterations let you trade reconstruction accuracy for lower VRAM use or faster inference.

Ideal use cases

LingBot-Map fits well when you have a video or ordered image sequence (indoor walkthrough, outdoor drive, aerial footage, robot camera feed) and need camera poses plus a 3D point cloud without running a slow per-scene optimizer. It's suited to long sequences: the windowed inference mode and offline renderer are built specifically to handle sequences from a few thousand up to tens of thousands of frames, such as a 13-minute, ~25,000-frame indoor walkthrough.

It's also a reasonable base for benchmarking against existing streaming or optimization-based 3D reconstruction methods, since the repo ships evaluation pipelines for datasets like KITTI, Oxford Spires, VBR, TUM-D, 7-scenes, ETH3D, Tanks and Temples, and NRGBD.

It is not a good fit if you need mesh-quality, texture-complete 3D assets (it outputs point clouds, not meshes), or if you're working with unordered, non-sequential photo collections rather than a continuous trajectory. It also requires a CUDA GPU for practical performance; CPU-only inference isn't the intended path, and the batch rendering pipeline adds extra native dependencies (Kaolin, Open3D) that increase setup complexity.

Installation

Requires a CUDA 12.8-capable GPU and Python 3.10.

conda create -n lingbot-map python=3.10 -y
conda activate lingbot-map

# PyTorch (CUDA 12.8)
pip install torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu128

# Install the package
pip install -e .

# FlashInfer for paged KV cache attention (recommended; falls back to SDPA if absent)
pip install --index-url https://pypi.org/simple flashinfer-python

# Optional: visualization extras
pip install -e ".[vis]"

Download a model checkpoint from Hugging Face (robbyant/lingbot-map) or ModelScope, then run the interactive demo:

python demo.py --model_path /path/to/lingbot-map.pt \
    --image_folder example/courthouse --mask_sky

This opens a viser viewer at http://localhost:8080. For sequences over 3,000 frames, use windowed mode:

python demo.py --model_path /path/to/lingbot-map.pt \
    --video_path video.mp4 --fps 10 \
    --mode windowed --window_size 128 --overlap_keyframes 16 --keyframe_interval 2

For sequences too long for the interactive viewer, use the offline rendering pipeline, which needs extra dependencies:

pip install -e ".[vis,render]"

That extra installs Open3D and PyYAML for headless point-cloud flythrough rendering via demo_render/batch_demo.py, and also needs NVIDIA Kaolin matching the PyTorch/CUDA versions above.