How to Prepare Input Data from Images or Video Files for LingBot-Map

LingBot-Map requires input data as a batch of RGB tensors with shape (N, 3, H, W) where height and width are multiples of the patch size (14), achieved through the load_images() helper in demo.py that handles video frame extraction, image globbing, and preprocessing.

The Robbyant/lingbot-map repository provides a complete pipeline for preparing visual data from both static image collections and video files. This guide explains the end-to-end process for converting raw media into model-ready tensors, referencing the actual source implementation in demo.py and lingbot_map/utils/load_fn.py.

Input Data Requirements for LingBot-Map

LingBot-Map processes visual input as batches of RGB tensors with specific architectural constraints. The model expects tensors of shape (N, 3, H, W) where N is the batch size and H (height) and W (width) must be multiples of the model's patch size (default 14). According to the source code in lingbot_map/utils/load_fn.py, the default target image_size is 518 pixels, though the final dimensions adjust to ensure patch divisibility.

The Complete Preparation Pipeline

The repository centralizes data preparation in the load_images() function within demo.py. This helper orchestrates six distinct stages to transform raw media into normalized tensors.

Step 1: Source Type Detection

The pipeline first determines whether the input is a video file or an image folder. In demo.py lines 56-66, the function checks the video_path parameter:

if video_path is not None:
    # Treat as video source

else:
    # Treat as image folder

Step 2: Video Frame Extraction

When processing video files, the system uses OpenCV (cv2.VideoCapture) to extract frames at a specified sampling rate. Located in demo.py lines 69-84, this logic calculates an interval based on the source FPS and target FPS:

interval = max(1, round(src_fps / fps))

The function writes every interval-th frame as a JPEG into a sibling *_frames folder, creating a temporary image sequence for downstream processing.

Step 3: Image Collection and Globbing

For image folders, the helper collects files with supported extensions (.jpg, .png, .JPG) using glob.glob() as shown in demo.py lines 90-96:

for ext in ['.jpg', '.png', '.JPG']:
    paths.extend(glob.glob(os.path.join(image_folder, f"*{ext}")))

Step 4: Optional Sub-sampling and Rotation

Before loading pixels, the pipeline supports optional dataset curation through parameters defined in demo.py lines 98-114:

• first_k: Limits the dataset to the first K frames using paths[:first_k]
• stride: Skips frames with paths[::stride]
• rotate_clockwise_90: Applies 90° clockwise rotation using Pillow's Image.ROTATE_270

Step 5: Loading and Preprocessing

The collected paths pass to load_and_preprocess_images() in lingbot_map/utils/load_fn.py lines 65-99. This function performs several critical transformations:

1. Orientation correction: Fixes EXIF orientation metadata
2. Channel conversion: Removes alpha channels and converts to RGB
3. Size calculation: Computes target dimensions ensuring multiples of patch_size (14)
4. Resizing: Applies bicubic interpolation to reach target size
5. Spatial adjustment: Either crops (default) or pads to maintain pixel count while satisfying patch constraints

Step 6: Tensor Validation and Return

Finally, demo.py lines 119-124 validates the output shape and returns three objects:

return images, paths, resolved_folder

The returned images tensor has shape (N, 3, H, W) where H and W are confirmed as patch-size multiples, ready for the LingBot-Map model.

Code Implementation Examples

Processing Image Folders

To prepare a directory of images from the command line:

python demo.py \
    --model_path checkpoints/gct.pt \
    --image_folder /path/to/images \
    --mode streaming

This command globs all JPEG and PNG files, applies the default center-crop to 518×518 pixels, and produces a tensor of shape (N, 3, 518, 518).

Processing Video Files

To extract and prepare frames from a video:

python demo.py \
    --model_path checkpoints/gct.pt \
    --video_path /videos/scene.mp4 \
    --fps 15

The script creates a scene_frames/ directory, samples at 15 FPS, and applies the same preprocessing pipeline as image folders.

Programmatic Usage

For custom data pipelines, import the preprocessing utilities directly:

from lingbot_map.utils.load_fn import load_and_preprocess_images

# Raw file paths from video extraction or custom source

file_list = ["/tmp/frames/000001.jpg", "/tmp/frames/000002.jpg"]

# Generate model-ready tensor

tensor, _ = load_and_preprocess_images(
    file_list,
    mode="crop",          # Alternative: "pad" to preserve aspect ratio

    image_size=518,
    patch_size=14,
)

Key Source Files and Functions

• demo.py: Orchestrates the full pipeline including video frame extraction (cv2.VideoCapture), image globbing, and the top-level load_images() function.
• lingbot_map/utils/load_fn.py: Contains the core load_and_preprocess_images() implementation (lines 65-99) and an alternative load_and_preprocess_images_square() helper (lines 16-30) for datasets requiring square inputs.
• lingbot_map/utils/pose_enc.py: Handles pose encodings for downstream processing (post-preparation stage).
• lingbot_map/utils/geometry.py: Provides camera intrinsics handling for later reconstruction stages.

Summary

• LingBot-Map requires input tensors of shape (N, 3, H, W) where H and W are multiples of 14.
• The load_images() function in demo.py provides the end-to-end pipeline for both images and video.
• Video processing uses OpenCV to extract frames at configurable FPS into temporary folders.
• Preprocessing includes EXIF orientation correction, RGB conversion, and bicubic resizing with either cropping or padding.
• The load_and_preprocess_images() function in lingbot_map/utils/load_fn.py implements the core tensor preparation logic.

Frequently Asked Questions

What image formats does LingBot-Map support?

LingBot-Map supports JPEG and PNG files with extensions .jpg, .JPG, and .png. The load_images() function in demo.py explicitly globs these extensions when scanning directories, though the underlying load_and_preprocess_images() function in lingbot_map/utils/load_fn.py uses Pillow, which can handle additional formats if passed programmatically.

How does the video frame extraction determine which frames to sample?

The sampling logic in demo.py lines 69-84 calculates an interval by dividing the source video's FPS by the target FPS (interval = max(1, round(src_fps / fps))), then saves every interval-th frame. This ensures consistent temporal sampling without dropping below the requested frame rate.

Should I use "crop" or "pad" mode for preprocessing?

Use "crop" (the default) when you want to maintain the exact target resolution and remove peripheral pixels to achieve patch-size multiples. Use "pad" when you must preserve all image content and can tolerate black borders added to reach the required dimensions. The mode is passed to load_and_preprocess_images() in lingbot_map/utils/load_fn.py lines 65-99.

Why must image dimensions be multiples of 14?

The patch size of 14 corresponds to the Vision Transformer (ViT) patch embedding stride used in LingBot-Map's architecture. The model splits images into 14×14 pixel patches for processing, requiring input dimensions to be divisible by this value to avoid partial patches. The preprocessing code automatically adjusts the target size to the nearest valid multiple.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}