# What Are the Input Data Formats Expected by LingBot-Map? A Complete Technical Guide

> Learn the expected input data formats for LingBot-Map. This guide details RGB image sequences, metadata, camera poses, depth maps, and validity masks for optimal use.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: how-to-guide
- Published: 2026-07-05

---

**LingBot-Map consumes sequences of RGB images as 8-bit unsigned integer arrays and, when running within the benchmark framework, expects per-frame dictionaries containing optional metadata such as camera poses, intrinsics, depth maps, and validity masks.**

The `Robbyant/lingbot-map` repository implements a 3D scene reconstruction system that processes visual data through two distinct pathways: a streamlined demo mode for quick inference and a comprehensive benchmark mode for evaluation against SLAM datasets. Understanding the specific input data formats expected by LingBot-Map is essential for both running pre-trained models and integrating custom datasets into the evaluation suite.

## Core Visual Input Specifications

### RGB Image Requirements

The model's inference pipeline requires RGB images in 8-bit unsigned integer format with shape H×W×3. According to the source code in [`lingbot_map/utils/load_fn.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/load_fn.py), the `load_and_preprocess_images` function handles the loading pipeline:

- Accepts PNG, JPEG, or any format compatible with Pillow
- Applies EXIF orientation correction automatically  
- Converts to RGB color space
- Resizes to meet patch-size constraints (default multiple of 14)

The function returns a `torch.Tensor` of shape `(N, 3, H, W)` where N represents the number of frames in the sequence.

### Video Input Processing

For video sources, [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) utilizes OpenCV's `cv2.VideoCapture` to extract frames at a specified FPS. The implementation temporarily stores extracted frames as JPEG files before passing them through the standard image preprocessing pipeline located in [`lingbot_map/utils/load_fn.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/load_fn.py).

## Benchmark Framework Metadata Schema

### The BaseDataset Contract

When evaluating on standard SLAM datasets (KITTI, TUM, 7-Scenes), the framework expects a dictionary structure defined in [`benchmark/benchmark/dataset/base.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/dataset/base.py). The abstract `load_frame_data` method must return a dictionary with the following schema:

```python
{
    "rgb": np.ndarray,          # HxWx3 uint8 (required)

    "depth": np.ndarray,        # HxW float32 (optional)

    "mask": np.ndarray,         # HxW bool (optional)

    "pose": np.ndarray,         # 4x4 float32 C2W (optional)

    "intrinsics": np.ndarray,   # [fx, fy, cx, cy] or 3x3 matrix (optional)

}

```

### Dataset-Specific Requirements

Concrete implementations vary by dataset:

- **KITTI**: Requires `rgb`, `pose` (4×4 camera-to-world matrix), and `intrinsics` (as `[fx, fy, cx, cy]`), implemented in [`benchmark/datasets/kitti.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/datasets/kitti.py)
- **TUM and 7-Scenes**: Follow the same schema with optional pose fields when ground truth is unavailable
- **Custom datasets**: Must inherit from `BaseDataset` and implement `load_frame_data`, returning at minimum the `rgb` array

## Data Pipeline Integration

The model consumed by both paths is the **GCTStream** network located in [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py). In the demo path, [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) calls `load_images`, which relies solely on the RGB tensor. In the benchmark path, the framework iterates through scenes and frames, feeding the complete metadata dictionary to the model. Optional fields (`intrinsics`, `pose`) enable pose refinement and depth scaling when available.

## Code Examples

### Loading Images from a Directory

```python
from lingbot_map.utils.load_fn import load_and_preprocess_images

# Paths to image files

image_paths = ["frame_001.jpg", "frame_002.jpg"]

# Returns tensor of shape (N, 3, H, W)

images = load_and_preprocess_images(
    image_paths,
    mode="crop",          # or "pad" to handle aspect ratios

    image_size=518,       # target size

    patch_size=14,        # must be multiple of 14

)

```

### Processing Video Input

```bash
python demo.py \
    --model_path checkpoint.pt \
    --video_path sequence.mp4 \
    --fps 15 \
    --image_size 518

```

### Creating a Custom Dataset Class

```python
from benchmark.benchmark.dataset.base import BaseDataset
import numpy as np
from PIL import Image
import pathlib

class CustomDataset(BaseDataset):
    def get_scenes(self):
        return ["scene_01"]
    
    def get_frame_list(self, scene):
        return list(range(100))  # 100 frames

    
    def load_frame_data(self, scene, frame_id):
        img_path = pathlib.Path(self.raw_data_root) / f"frame_{frame_id:04d}.png"
        rgb = np.array(Image.open(img_path).convert("RGB"), dtype=np.uint8)
        
        # Optional metadata

        pose = np.eye(4, dtype=np.float32)
        intrinsics = np.array([525.0, 525.0, 319.5, 239.5], dtype=np.float32)
        
        return {
            "rgb": rgb,
            "pose": pose,
            "intrinsics": intrinsics,
        }

```

## Summary

- **RGB images** must be provided as 8-bit unsigned integer arrays (H×W×3) processed through `load_and_preprocess_images` in [`lingbot_map/utils/load_fn.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/load_fn.py)
- Image dimensions must align to the model's **patch size** (default 14 pixels) via cropping or padding modes
- **Video inputs** are supported through OpenCV frame extraction in [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py)
- The **benchmark framework** expects per-frame dictionaries with optional keys for `depth`, `mask`, `pose`, and `intrinsics` as defined in [`benchmark/benchmark/dataset/base.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/dataset/base.py)
- Only the `rgb` field is mandatory for the **GCTStream** model inference; metadata fields enable advanced features like pose refinement when available

## Frequently Asked Questions

### Does LingBot-Map require camera calibration files?

No, the demo mode runs on images alone. However, for quantitative evaluation and pose refinement, you should provide intrinsics as a numpy array `[fx, fy, cx, cy]` or a 3×3 matrix within the benchmark framework's metadata dictionary.

### What image resolutions work best with LingBot-Map?

The height and width must be multiples of the model's patch size, which defaults to 14 pixels. The `load_and_preprocess_images` function automatically handles this constraint using either center cropping or padding depending on the `mode` parameter.

### Can I use depth maps or masks with my custom dataset?

Yes. Include `depth` (H×W float32 meters) or `mask` (H×W bool) keys in the dictionary returned by your `BaseDataset.load_frame_data` implementation. The model automatically utilizes these for depth-aware heads and masking invalid pixels during optimization.

### How does the video processing pipeline handle temporary files?

The [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) script extracts frames from video inputs using OpenCV at the specified FPS, saves them as temporary JPEG files, and then processes them through the standard image loader before cleanup.