Common Use Cases for LingBot-Map: Interactive Streaming and Batch Rendering

LingBot-Map is primarily used for real-time interactive 3D reconstruction via browser-based visualization and offline batch rendering of long video sequences, with specialized workflows for sky masking, windowed inference, and custom camera path generation.

LingBot-Map is a feed-forward 3D foundation model engineered for streaming 3D reconstruction of image sequences. As implemented in the Robbyant/lingbot-map repository, it unifies coordinate grounding, dense geometric cues, and long-range drift correction through its Geometric Context Transformer architecture. Common use cases for lingbot-map span from robotics and AR/VR applications requiring immediate scene understanding to research visualization demanding high-fidelity offline video production for trajectories exceeding 25,000 frames.

Interactive Streaming Reconstruction

The repository provides real-time visualization through demo.py, which targets use cases requiring immediate feedback, such as robotics debugging or AR/VR scene exploration.

Real-Time Point Cloud Visualization

Running the interactive demo loads a checkpoint and consumes an image folder or video frame-by-frame. According to the source code in demo.py (lines 54-66), the script initializes a Viser viewer at http://localhost:8080 and streams reconstructed point clouds via WebSocket. This workflow leverages FlashInfer (or PyTorch SDPA as fallback) via the paged KV-cache implementation in lingbot_map/layers/flashinfer_cache.py (lines 15-18 and 24-27), enabling stable inference at approximately 20 FPS on 518×378 resolution inputs.

Sky Masking for Outdoor Scenes

For outdoor reconstructions, users apply automated sky removal to prevent distant sky pixels from contaminating depth estimates. The --mask_sky flag triggers the ONNX-based segmentation model in lingbot_map/vis/sky_segmentation.py, referenced in demo.py (lines 59-66) and demo_render/batch_demo.py (lines 71-78). Masks are applied before point-cloud aggregation, improving geometric accuracy in open environments.

Long-Sequence Batch Rendering

For production workflows and extended trajectories (e.g., 25,000+ frames), demo_render/batch_demo.py provides a headless rendering pipeline. This use case supports windowed inference and custom camera choreography for research visualizations and virtual tours.

Windowed Inference for Extended Trajectories

Sequences exceeding the training RoPE range (approximately 320 frames) require the --mode windowed flag with a specified --window_size. As implemented in demo_render/batch_demo.py (lines 50-57 and 92-99), this splits the stream into overlapping windows and resets the KV cache per window to prevent drift while maintaining geometric consistency across the full trajectory.

Keyframe-Based Memory Management

To limit KV-cache growth during extremely long videos, users employ --keyframe_interval N. This configuration, supported in both entry points (lines 24-27 and 33-40), stores only every N-th frame in trajectory memory. Non-keyframes still produce predictions but do not expand the cache, enabling stable inference on sequences exceeding 10,000 frames without memory exhaustion.

Cinematic Camera Path Design

Cinematic renderings are configured through YAML presets in demo_render/config/. Users edit files such as indoor.yaml or outdoor_drive.yaml to define camera segments (follow, birdeye, static, pivot), then pass --config demo_render/config/indoor.yaml to batch_demo.py (lines 77-85 and 86-105) to generate high-quality MP4 fly-throughs with custom camera paths.

Model Selection and Checkpoints

The repository provides three optimized checkpoints hosted on HuggingFace and ModelScope:

• lingbot-map: General-purpose checkpoint for standard sequences
• lingbot-map-long: Optimized for extended trajectories requiring long-term consistency
• lingbot-map-stage1: Stage-one training checkpoint for specialized fine-tuning

Users select checkpoints via the --model_path argument to match specific use case requirements for short versus long trajectories, as documented in the model download section (lines 33-40).

Practical Code Examples

The following commands demonstrate the primary use cases for lingbot-map:

# 1️⃣ Interactive demo on the courthouse scene (with sky masking)

python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/courthouse \
    --mask_sky

# 2️⃣ Batch rendering of a long indoor walkthrough (≈25 000 frames)

python demo_render/batch_demo.py \
    --video_path /data/demo_videos/indoor_travel.MP4 \
    --output_folder /data/outputs/indoor_travel/ \
    --model_path /path/to/lingbot-map.pt \
    --config demo_render/config/indoor.yaml \
    --mode windowed --window_size 128 \
    --keyframe_interval 10 --overlap_keyframes 8 \
    --sky_mask_dir /data/outputs/sky_masks \
    --camera_vis default --keyframes_only_points \
    --save_predictions

# 3️⃣ Enable windowed inference with a custom keyframe interval (for any sequence)

python demo.py \
    --image_folder example/oxford \
    --model_path /path/to/lingbot-map-long.pt \
    --mask_sky \
    --mode windowed --window_size 128 \
    --keyframe_interval 2

Core Architecture Supporting Streaming Use Cases

The implementation of these use cases relies on specific architectural components that enable efficient long-sequence processing.

Geometric Context Transformer

The backbone architecture in lingbot_map/models/gct_base.py implements the Geometric Context Transformer, which unifies coordinate grounding, dense geometric cues, and long-range drift correction via an anchor context, pose-reference window, and trajectory memory (lines 29-31). This design specifically supports the streaming architecture required for both interactive and batch processing workflows.

Paged KV-Cache Attention

Memory efficiency for streaming use cases relies on lingbot_map/layers/flashinfer_cache.py, which implements paged KV-cache attention. This mechanism, integrated into the inference pipeline (lines 15-18 and 24-27), provides selective attention over the trajectory history while maintaining the ~20 FPS performance critical for real-time applications.

Summary

• Interactive visualization via demo.py provides real-time browser-based point cloud inspection at ~20 FPS using Viser
• Batch rendering via demo_render/batch_demo.py generates high-quality offline videos for sequences exceeding 25,000 frames
• Windowed inference (--mode windowed) enables processing beyond the 320-frame training limit by managing KV-cache resets
• Keyframe sampling (--keyframe_interval) controls memory growth for extremely long videos without sacrificing reconstruction quality
• Sky masking improves outdoor reconstruction accuracy by filtering sky pixels before point-cloud aggregation
• Three specialized checkpoints (lingbot-map, lingbot-map-long, lingbot-map-stage1) optimize for different sequence lengths and use cases

Frequently Asked Questions

What is the difference between the lingbot-map and lingbot-map-long checkpoints?

The lingbot-map checkpoint is optimized for general-purpose reconstruction of standard-length sequences, while lingbot-map-long is specifically trained for extended trajectories requiring enhanced long-range consistency. According to the repository documentation (lines 33-40), the long variant better handles drift correction over thousands of frames when using windowed inference modes.

How does windowed inference handle sequences longer than 320 frames?

When --mode windowed is specified, the implementation in demo_render/batch_demo.py (lines 50-57 and 92-99) splits the input stream into overlapping windows of the specified size (e.g., 128 frames). The KV cache resets at each window boundary to prevent attention drift beyond the training RoPE range, while overlapping regions ensure geometric continuity between segments.

Can I use LingBot-Map without FlashInfer installed?

Yes, the repository gracefully falls back to PyTorch's Scaled Dot-Product Attention (SDPA) when FlashInfer is unavailable. The lingbot_map/layers/flashinfer_cache.py implementation (lines 15-18 and 24-27) detects the FlashInfer dependency and automatically switches to the fallback mechanism, though this may impact the maximum achievable frame rate for real-time streaming use cases.

What file paths control the camera movement in batch rendering?

Camera trajectories are defined via YAML configuration files stored in demo_render/config/. The batch_demo.py script (lines 77-85 and 86-105) parses these files to execute camera segments including follow, birdeye, static, and pivot movements. Users can reference indoor.yaml or outdoor_drive.yaml as templates for creating custom cinematic paths.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}