How to Use the Offline Batch Rendering Pipeline in LingBot-Map

The offline batch rendering pipeline in LingBot-Map processes long video sequences and large image folders through a four-stage workflow orchestrated by demo_render/batch_demo.py: data ingestion, model inference via streaming or windowed modes, prediction export as .npz files, and headless RGBD rendering into MP4 videos or GLB assets.

LingBot-Map (available at Robbyant/lingbot-map) provides a dedicated offline pipeline for processing sequences that exceed the memory constraints of the interactive demo.py viewer. By running demo_render/batch_demo.py from the command line, you can extract frames from video sources, run inference on canonical-sized crops, and render high-quality fly-through videos without requiring a display attached to the machine.

Understanding the Four-Stage Pipeline

The offline batch rendering pipeline operates through four distinct stages that transform raw media into final rendered outputs.

Stage 1: Data Ingestion and Preprocessing

The pipeline accepts input via either --video_path for video files or --input_folder for directories of images. When processing video, the extract_frames_from_video function uses OpenCV to extract frames on-the-fly, optionally caching them as PNGs to --save_frames_dir for reuse across runs. For image folders, list_image_paths discovers files and filters them by range, stride, or explicit --first_k and --last_k arguments. All frames are then processed by load_and_preprocess_images in lingbot_map/utils/load_fn.py, which crops inputs to a canonical size and pads them to multiples of the model's patch size.

Stage 2: Streaming vs. Windowed Inference

The run_inference function in demo_render/batch_demo.py supports two execution modes controlled by the --mode argument.

Streaming inference (--mode streaming) calls model.inference_streaming and maintains a KV-cache for continuous frame sequences. Control memory usage by adjusting --keyframe_interval, which trims the cache every N frames.

Windowed inference (--mode windowed) calls model.inference_windowed and splits the sequence into overlapping windows. Configure the effective overlap using --overlap_keyframes or specify absolute pixel dimensions with --overlap_size. Set the window length with --window_size.

Stage 3: Prediction Export and Caching

After inference, per-frame predictions are serialized via save_predictions_npz as a directory of .npz files. This layout enables fast parallel I/O when the rendering stage calls load_predictions_from_npz. Enable this export by including --save_predictions in your command. If you have previously saved predictions, you can bypass inference entirely by passing --load_predictions followed by the path to the .npz file.

Stage 4: Headless RGBD Rendering

The final stage invokes render_with_pipeline to execute the rgbd-render pipeline. A PipelineConfig object is first built from an optional YAML preset (--config) and then overridden by any CLI flags. The pipeline performs CUDA-accelerated voxelization via voxelize_frame (from demo_render/render_cuda_ext), constructs a virtual camera path using rgbd_render.camera.build_camera_path, and adds optional overlays including trajectory trails, frustum visualization, and sky-mask overlays. The OfflinePipeline class writes the final output as an MP4 video. Enable sky masking with --mask_sky, which utilizes the ONNX segmentation model in lingbot_map/vis/sky_segmentation.py.

Key Implementation Files

Understanding the source structure helps with debugging and extending the pipeline:

• demo_render/batch_demo.py: Main entry point handling argument parsing, scene discovery, and stage orchestration.
• lingbot_map/utils/load_fn.py: Contains load_and_preprocess_images for canonical cropping and resizing.
• lingbot_map/vis/sky_segmentation.py: ONNX-based sky segmentation for the --mask_sky option.
• demo_render/render_cuda_ext/: CUDA extensions for voxelize_frame and frustum culling.
• rgbd_render/pipeline/builder.py and rgbd_render/pipeline/offline.py: Build scene representations and manage headless MP4 encoding.
• demo_render/config/*.yaml: Presets like indoor.yaml and outdoor_drive.yaml defining default camera paths and rendering parameters.

Running the Offline Pipeline

Below are three typical usage patterns. Replace placeholder paths with your actual directories.

Batch Processing Multiple Scenes

Process a folder where each subdirectory contains a scene:

python demo_render/batch_demo.py \
    --input_folder /data/scenes \
    --output_folder /data/outputs \
    --model_path /path/to/lingbot-map-long.pt \
    --mode streaming \
    --keyframe_interval 2 \
    --mask_sky \
    --save_predictions

Processing a Single Video

Extract frames on-the-fly from a video file using windowed inference:

python demo_render/batch_demo.py \
    --video_path /data/video/indoor_travel.MP4 \
    --output_folder /data/outputs \
    --model_path /path/to/lingbot-map.pt \
    --config demo_render/config/indoor.yaml \
    --mode windowed \
    --window_size 128 \
    --overlap_keyframes 8 \
    --keyframe_interval 13 \
    --mask_sky \
    --save_predictions \
    --save_glb

Rendering from Cached Predictions

Skip inference and render directly from previously saved .npz files:

python demo_render/batch_demo.py \
    --load_predictions /data/outputs/indoor_travel.npz \
    --output_folder /data/outputs \
    --config demo_render/config/indoor.yaml

Configuration and Optimization

Managing Memory with Keyframe Intervals

The --keyframe_interval parameter controls how frequently the KV-cache is trimmed during streaming inference. Lower values reduce memory consumption but may decrease temporal consistency. For windowed mode, ensure --window_size fits within your GPU memory, and use --overlap_keyframes to maintain smooth transitions between windows.

YAML Presets and Camera Paths

Configuration files in demo_render/config/ define default camera trajectories, voxel sizes, and overlay settings. Pass a preset with --config, then override specific values via CLI flags. For example, indoor.yaml typically configures tighter camera paths and smaller voxel sizes compared to outdoor_drive.yaml.

Summary

• The offline batch rendering pipeline in LingBot-Map is orchestrated by demo_render/batch_demo.py and designed for headless processing of long sequences.
• Data ingestion supports both video files and image folders, with preprocessing handled by lingbot_map/utils/load_fn.py.
• Inference supports streaming (with KV-cache management) and windowed (with overlap control) modes.
• Predictions are cached as .npz files via save_predictions_npz, enabling fast re-rendering without recomputing inference.
• Rendering uses the rgbd_render pipeline with CUDA-accelerated voxelization (voxelize_frame) and outputs MP4 videos or GLB assets.

Frequently Asked Questions

What is the difference between streaming and windowed inference in LingBot-Map?

Streaming inference (--mode streaming) processes the entire sequence continuously while maintaining a KV-cache in memory, making it suitable for long videos with the --keyframe_interval parameter to limit cache growth. Windowed inference (--mode windowed) splits the input into discrete chunks of size --window_size with configurable overlap (--overlap_keyframes), which is more memory-efficient for very high-resolution inputs but requires careful tuning of overlap to prevent boundary artifacts.

How can I process videos that are too large to fit in GPU memory?

Use windowed inference with a reduced --window_size and increase the --overlap_keyframes to maintain temporal consistency between chunks. Alternatively, use streaming inference with a low --keyframe_interval (e.g., 2-4) to aggressively trim the KV-cache. The preprocessing step in load_and_preprocess_images automatically handles canonical cropping to reduce per-frame memory footprint.

Can I reuse depth predictions without rerunning the neural network?

Yes. Include --save_predictions during your initial run to export .npz files via save_predictions_npz. On subsequent runs, pass --load_predictions /path/to/file.npz to skip the inference stage entirely and proceed directly to rendering or GLB export. This is significantly faster when experimenting with different camera paths or rendering configurations.

What hardware is required for the CUDA voxelization step?

The voxelize_frame operation in demo_render/render_cuda_ext requires an NVIDIA GPU with CUDA support. The pipeline falls back to CPU implementations if the CUDA extensions are not built, but this drastically reduces performance when rendering long sequences. Ensure your environment has the CUDA toolkit installed and the extensions compiled before running the rendering stage.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}