What Are the Input Data Formats Expected by LingBot-Map? A Complete Technical Guide

LingBot-Map consumes sequences of RGB images as 8-bit unsigned integer arrays and, when running within the benchmark framework, expects per-frame dictionaries containing optional metadata such as camera poses, intrinsics, depth maps, and validity masks.

The Robbyant/lingbot-map repository implements a 3D scene reconstruction system that processes visual data through two distinct pathways: a streamlined demo mode for quick inference and a comprehensive benchmark mode for evaluation against SLAM datasets. Understanding the specific input data formats expected by LingBot-Map is essential for both running pre-trained models and integrating custom datasets into the evaluation suite.

Core Visual Input Specifications

RGB Image Requirements

The model's inference pipeline requires RGB images in 8-bit unsigned integer format with shape H×W×3. According to the source code in lingbot_map/utils/load_fn.py, the load_and_preprocess_images function handles the loading pipeline:

• Accepts PNG, JPEG, or any format compatible with Pillow
• Applies EXIF orientation correction automatically
• Converts to RGB color space
• Resizes to meet patch-size constraints (default multiple of 14)

The function returns a torch.Tensor of shape (N, 3, H, W) where N represents the number of frames in the sequence.

Video Input Processing

For video sources, demo.py utilizes OpenCV's cv2.VideoCapture to extract frames at a specified FPS. The implementation temporarily stores extracted frames as JPEG files before passing them through the standard image preprocessing pipeline located in lingbot_map/utils/load_fn.py.

Benchmark Framework Metadata Schema

The BaseDataset Contract

When evaluating on standard SLAM datasets (KITTI, TUM, 7-Scenes), the framework expects a dictionary structure defined in benchmark/benchmark/dataset/base.py. The abstract load_frame_data method must return a dictionary with the following schema:

{
    "rgb": np.ndarray,          # HxWx3 uint8 (required)

    "depth": np.ndarray,        # HxW float32 (optional)

    "mask": np.ndarray,         # HxW bool (optional)

    "pose": np.ndarray,         # 4x4 float32 C2W (optional)

    "intrinsics": np.ndarray,   # [fx, fy, cx, cy] or 3x3 matrix (optional)

}

Dataset-Specific Requirements

Concrete implementations vary by dataset:

• KITTI: Requires rgb, pose (4×4 camera-to-world matrix), and intrinsics (as [fx, fy, cx, cy]), implemented in benchmark/datasets/kitti.py
• TUM and 7-Scenes: Follow the same schema with optional pose fields when ground truth is unavailable
• Custom datasets: Must inherit from BaseDataset and implement load_frame_data, returning at minimum the rgb array

Data Pipeline Integration

The model consumed by both paths is the GCTStream network located in lingbot_map/models/gct_stream.py. In the demo path, demo.py calls load_images, which relies solely on the RGB tensor. In the benchmark path, the framework iterates through scenes and frames, feeding the complete metadata dictionary to the model. Optional fields (intrinsics, pose) enable pose refinement and depth scaling when available.

Code Examples

Loading Images from a Directory

from lingbot_map.utils.load_fn import load_and_preprocess_images

# Paths to image files

image_paths = ["frame_001.jpg", "frame_002.jpg"]

# Returns tensor of shape (N, 3, H, W)

images = load_and_preprocess_images(
    image_paths,
    mode="crop",          # or "pad" to handle aspect ratios

    image_size=518,       # target size

    patch_size=14,        # must be multiple of 14

)

Processing Video Input

python demo.py \
    --model_path checkpoint.pt \
    --video_path sequence.mp4 \
    --fps 15 \
    --image_size 518

Creating a Custom Dataset Class

from benchmark.benchmark.dataset.base import BaseDataset
import numpy as np
from PIL import Image
import pathlib

class CustomDataset(BaseDataset):
    def get_scenes(self):
        return ["scene_01"]
    
    def get_frame_list(self, scene):
        return list(range(100))  # 100 frames

    
    def load_frame_data(self, scene, frame_id):
        img_path = pathlib.Path(self.raw_data_root) / f"frame_{frame_id:04d}.png"
        rgb = np.array(Image.open(img_path).convert("RGB"), dtype=np.uint8)
        
        # Optional metadata

        pose = np.eye(4, dtype=np.float32)
        intrinsics = np.array([525.0, 525.0, 319.5, 239.5], dtype=np.float32)
        
        return {
            "rgb": rgb,
            "pose": pose,
            "intrinsics": intrinsics,
        }

Summary

• RGB images must be provided as 8-bit unsigned integer arrays (H×W×3) processed through load_and_preprocess_images in lingbot_map/utils/load_fn.py
• Image dimensions must align to the model's patch size (default 14 pixels) via cropping or padding modes
• Video inputs are supported through OpenCV frame extraction in demo.py
• The benchmark framework expects per-frame dictionaries with optional keys for depth, mask, pose, and intrinsics as defined in benchmark/benchmark/dataset/base.py
• Only the rgb field is mandatory for the GCTStream model inference; metadata fields enable advanced features like pose refinement when available

Frequently Asked Questions

Does LingBot-Map require camera calibration files?

No, the demo mode runs on images alone. However, for quantitative evaluation and pose refinement, you should provide intrinsics as a numpy array [fx, fy, cx, cy] or a 3×3 matrix within the benchmark framework's metadata dictionary.

What image resolutions work best with LingBot-Map?

The height and width must be multiples of the model's patch size, which defaults to 14 pixels. The load_and_preprocess_images function automatically handles this constraint using either center cropping or padding depending on the mode parameter.

Can I use depth maps or masks with my custom dataset?

Yes. Include depth (H×W float32 meters) or mask (H×W bool) keys in the dictionary returned by your BaseDataset.load_frame_data implementation. The model automatically utilizes these for depth-aware heads and masking invalid pixels during optimization.

How does the video processing pipeline handle temporary files?

The demo.py script extracts frames from video inputs using OpenCV at the specified FPS, saves them as temporary JPEG files, and then processes them through the standard image loader before cleanup.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}