How to Configure Offline Rendering for Very Long Sequences (>10,000 Frames) in LingBot-Map

To configure offline rendering for sequences exceeding 10,000 frames in LingBot-Map, use the batch_demo.py driver with --mode windowed, --keyframe_interval, and --overlap_keyframes flags to implement sliding-window inference that resets the KV cache and maintains bounded memory usage.

The Robbyant/lingbot-map repository provides a headless offline rendering pipeline specifically designed to process video sequences far longer than the interactive viewer can handle. While the demo.py viewer is limited by GPU memory constraints and the model's RoPE training window of approximately 320 frames, the batch driver implements sliding-window inference and efficient keyframe caching to render tens of thousands of frames without memory exhaustion.

Prerequisites: Installing CUDA Extensions and Dependencies

Before processing very long sequences, you must build the custom CUDA extensions required for voxelisation and frustum culling. According to the repository's installation instructions, these extensions are not included in the base package and must be compiled separately.

Run the following setup sequence:

# Install core dependencies and rendering extras

pip install -e .
pip install -e ".[vis,render]"  # Installs open3d, pyyaml

pip install onnxruntime-gpu     # Required for GPU sky masking

pip install --index-url https://pypi.org/simple \
    kaolin -f https://nvidia-kaolin.s3.us-east-2.amazonaws.com/torch-2.8.0_cu128.html

# Build CUDA extensions (required once)

cd demo_render/render_cuda_ext && python setup.py build_ext --inplace && cd ../..

The setup.py file in demo_render/render_cuda_ext/ compiles two critical modules: voxel_morton_ext for fast voxel hashing and frustum_cull_ext for camera frustum culling during the final rendering pass.

Understanding the Batch Rendering Architecture

The offline pipeline centers on demo_render/batch_demo.py, which orchestrates argument parsing, windowed inference, prediction storage, and final MP4 compositing. Unlike the interactive demo, this driver reuses the same FlashInfer cache implementation found in lingbot_map/layers/flashinfer_cache.py but manages it through explicit windowing rather than continuous accumulation.

For sequences longer than 10,000 frames, you must address two architectural constraints:

1. RoPE Training Limit: The Geometric Context Transformer was trained on RoPE windows of approximately 320 frames; beyond this, positional encoding accuracy degrades.
2. KV-Cache Growth: Without intervention, the paged KV cache grows proportionally with the number of keyframes, eventually exceeding GPU memory.

The batch driver solves both by implementing sliding-window inference that periodically resets the KV cache and RoPE state.

Configuring Sliding-Window Inference for Long Sequences

To enable windowed processing, invoke batch_demo.py with the --mode windowed flag. This activates the logic that splits the input stream into overlapping chunks, resets the KV cache between windows, and stitches predictions into a coherent output.

Use the following flags to tune the windowing behavior:

• --window_size 128: Sets the number of KV-cache slots per window. A typical configuration uses 8 scale frames plus 120 keyframe slots (total 128), keeping memory usage predictable regardless of total video length.
• --overlap_keyframes 8: Shares 8 keyframes between consecutive windows to maintain pose alignment and temporal continuity across window boundaries.
• --keyframe_interval 13: Stores only every 13th frame in the KV cache. Non-keyframes still produce depth predictions but do not expand the cache, which is crucial for very long sequences.

A typical configuration for a 25,000-frame indoor walkthrough appears as follows:

python demo_render/batch_demo.py \
    --video_path /data/indoor_walk.mp4 \
    --output_folder /data/outputs/indoor_walk/ \
    --model_path /path/to/lingbot-map.pt \
    --config demo_render/config/indoor.yaml \
    --mode windowed \
    --window_size 128 \
    --keyframe_interval 13 \
    --overlap_keyframes 8 \
    --save_predictions

Optimizing Memory with Keyframe Management

The FlashInfer cache (lingbot_map/layers/flashinfer_cache.py) implements a paged memory system where only keyframes allocate persistent KV slots. For sequences exceeding 10,000 frames, aggressive keyframe sub sampling is essential to prevent out-of-memory errors.

Enable --keyframes_only_points to restrict point-cloud unprojection to keyframe indices only. This produces a sparser but computationally manageable point cloud, reducing the memory footprint of the final rasterisation pass. The flag is particularly effective when combined with --keyframe_interval values between 10 and 15.

Defining Camera Trajectories via YAML

Long sequences require pre-planned camera paths defined in YAML configuration files. The batch driver reads these from demo_render/config/, with indoor.yaml serving as the recommended starting point for extended indoor walks.

A configuration file specifies camera segments, field of view, and transition frames:

# demo_render/config/indoor.yaml

camera:
  fov: 60.0
  transition: 30
  segments:
    - mode: follow
      frames: [0, 3000]
      back_offset: 0.25
      up_offset: 0.1
      look_offset: 0.5
    - mode: birdeye
      frames: [3000, 3500]
      reveal_height_mult: 3.0
    - mode: follow
      frames: [3500, -1]
      back_offset: 0.3
      up_offset: 0.08
      look_offset: 0.4

The segments array defines the trajectory: follow mode tracks the camera path with configurable offsets, while birdeye provides overview shots. The -1 in the final frame range indicates "until end of sequence."

Optional Sky Masking for Outdoor Scenes

For outdoor sequences or large indoor spaces with visible sky, enable the --mask_sky flag to activate an ONNX segmentation model (skyseg.onnx) via onnxruntime-gpu. This filters sky points from the depth unprojection, dramatically improving visual quality by eliminating distant, noisy points.

When using this option, specify the output directories for mask caching:

python demo_render/batch_demo.py \
    --video_path long_outdoor.mp4 \
    --output_folder ./out/ \
    --model_path ./lingbot-map.pt \
    --config demo_render/config/outdoor_large.yaml \
    --mode windowed --window_size 128 \
    --keyframe_interval 12 \
    --mask_sky \
    --sky_mask_dir ./cache/sky_masks \
    --sky_mask_visualization_dir ./cache/sky_viz

Complete Configuration Examples

Processing an Image Folder Instead of Video

If your long sequence exists as individual frames rather than an MP4, use --image_folder:

python demo_render/batch_demo.py \
    --image_folder ./dataset/university/ \
    --output_folder ./out/university/ \
    --model_path ./lingbot-map.pt \
    --config demo_render/config/indoor.yaml \
    --mode windowed --window_size 128 \
    --keyframe_interval 5 \
    --mask_sky

Minimal Configuration for Memory-Constrained Systems

For systems with limited VRAM, reduce the window size and increase the keyframe interval:

python demo_render/batch_demo.py \
    --video_path my_long_seq.mp4 \
    --output_folder ./out/ \
    --model_path ./lingbot-map.pt \
    --config demo_render/config/default.yaml \
    --mode windowed --window_size 96 \
    --keyframe_interval 20 \
    --overlap_keyframes 4 \
    --keyframes_only_points \
    --save_predictions

Debugging Long Sequences

Add --frame_tag and --frame_tag_position top_right to overlay a <current>/<total> counter on the rendered output. This assists in debugging specific temporal regions without reprocessing the entire sequence.

Summary

• Use demo_render/batch_demo.py as the entry point for sequences exceeding the interactive viewer's capacity.
• Enable --mode windowed to reset the KV cache and RoPE state every N frames, preventing memory exhaustion and maintaining pose accuracy beyond the 320-frame training limit.
• Tune --keyframe_interval and --window_size** to balance reconstruction density against GPU memory constraints.
• Build CUDA extensions (voxel_morton_ext, frustum_cull_ext) before first use to enable fast point-cloud rasterisation.
• Leverage YAML presets to define complex camera trajectories without modifying CLI commands.
• Persist predictions with --save_predictions to enable re-rendering with different camera paths without re-running inference.

Frequently Asked Questions

What is the maximum sequence length supported by LingBot-Map's offline renderer?

The offline renderer supports arbitrarily long sequences limited only by disk space for intermediate NPZ files. The sliding-window implementation in demo_render/batch_demo.py keeps GPU memory bounded by resetting the KV cache every --window_size frames (typically 128), allowing you to process tens of thousands of frames on a single GPU.

Why does the interactive demo fail on long videos while the batch pipeline succeeds?

The interactive demo.py maintains a continuous KV cache that grows with every frame, eventually exceeding GPU memory. Additionally, the model's RoPE positional encoding was trained on windows of approximately 320 frames; beyond this, accuracy degrades. The batch driver implements explicit windowing (--mode windowed) that resets these states periodically, with overlapping keyframes (--overlap_keyframes) ensuring temporal continuity.

How do I reduce GPU memory usage when rendering 10,000+ frames?

Increase the --keyframe_interval to store fewer frames in the KV cache (e.g., every 15-20 frames instead of every frame), reduce --window_size to 96 or 64 slots, and enable --keyframes_only_points to limit point-cloud generation to keyframe indices only. These settings trade point-cloud density for memory efficiency while maintaining trajectory coherence through the sliding-window overlap mechanism.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}