# How to Prepare Input Data from Images or Video Files for LingBot-Map

> Learn how to prepare image and video input data for LingBot-Map. Discover the efficient `load_images()` function for RGB tensor preprocessing and video frame extraction.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: how-to-guide
- Published: 2026-07-01

---

**LingBot-Map requires input data as a batch of RGB tensors with shape (N, 3, H, W) where height and width are multiples of the patch size (14), achieved through the `load_images()` helper in [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) that handles video frame extraction, image globbing, and preprocessing.**

The [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map) repository provides a complete pipeline for preparing visual data from both static image collections and video files. This guide explains the end-to-end process for converting raw media into model-ready tensors, referencing the actual source implementation in [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) and [`lingbot_map/utils/load_fn.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/load_fn.py).

## Input Data Requirements for LingBot-Map

LingBot-Map processes visual input as **batches of RGB tensors** with specific architectural constraints. The model expects tensors of shape **(N, 3, H, W)** where **N** is the batch size and **H** (height) and **W** (width) must be multiples of the model's **patch size** (default 14). According to the source code in [`lingbot_map/utils/load_fn.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/load_fn.py), the default target `image_size` is **518** pixels, though the final dimensions adjust to ensure patch divisibility.

## The Complete Preparation Pipeline

The repository centralizes data preparation in the `load_images()` function within [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py). This helper orchestrates six distinct stages to transform raw media into normalized tensors.

### Step 1: Source Type Detection

The pipeline first determines whether the input is a video file or an image folder. In [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) [lines 56-66](https://github.com/Robbyant/lingbot-map/blob/main/demo.py#L56-L66), the function checks the `video_path` parameter:

```python
if video_path is not None:
    # Treat as video source

else:
    # Treat as image folder

```

### Step 2: Video Frame Extraction

When processing video files, the system uses OpenCV (`cv2.VideoCapture`) to extract frames at a specified sampling rate. Located in [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) [lines 69-84](https://github.com/Robbyant/lingbot-map/blob/main/demo.py#L69-L84), this logic calculates an interval based on the source FPS and target FPS:

```python
interval = max(1, round(src_fps / fps))

```

The function writes every *interval*-th frame as a JPEG into a sibling `*_frames` folder, creating a temporary image sequence for downstream processing.

### Step 3: Image Collection and Globbing

For image folders, the helper collects files with supported extensions (`.jpg`, `.png`, `.JPG`) using `glob.glob()` as shown in [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) [lines 90-96](https://github.com/Robbyant/lingbot-map/blob/main/demo.py#L90-L96):

```python
for ext in ['.jpg', '.png', '.JPG']:
    paths.extend(glob.glob(os.path.join(image_folder, f"*{ext}")))

```

### Step 4: Optional Sub-sampling and Rotation

Before loading pixels, the pipeline supports optional dataset curation through parameters defined in [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) [lines 98-114](https://github.com/Robbyant/lingbot-map/blob/main/demo.py#L98-L114):

- **`first_k`**: Limits the dataset to the first K frames using `paths[:first_k]`
- **`stride`**: Skips frames with `paths[::stride]`
- **`rotate_clockwise_90`**: Applies 90° clockwise rotation using Pillow's `Image.ROTATE_270`

### Step 5: Loading and Preprocessing

The collected paths pass to `load_and_preprocess_images()` in [`lingbot_map/utils/load_fn.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/load_fn.py) [lines 65-99](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/load_fn.py#L65-L99). This function performs several critical transformations:

1. **Orientation correction**: Fixes EXIF orientation metadata
2. **Channel conversion**: Removes alpha channels and converts to RGB
3. **Size calculation**: Computes target dimensions ensuring multiples of `patch_size` (14)
4. **Resizing**: Applies bicubic interpolation to reach target size
5. **Spatial adjustment**: Either **crops** (default) or **pads** to maintain pixel count while satisfying patch constraints

### Step 6: Tensor Validation and Return

Finally, [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) [lines 119-124](https://github.com/Robbyant/lingbot-map/blob/main/demo.py#L119-L124) validates the output shape and returns three objects:

```python
return images, paths, resolved_folder

```

The returned `images` tensor has shape `(N, 3, H, W)` where `H` and `W` are confirmed as patch-size multiples, ready for the LingBot-Map model.

## Code Implementation Examples

### Processing Image Folders

To prepare a directory of images from the command line:

```bash
python demo.py \
    --model_path checkpoints/gct.pt \
    --image_folder /path/to/images \
    --mode streaming

```

This command globs all JPEG and PNG files, applies the default center-crop to 518×518 pixels, and produces a tensor of shape `(N, 3, 518, 518)`.

### Processing Video Files

To extract and prepare frames from a video:

```bash
python demo.py \
    --model_path checkpoints/gct.pt \
    --video_path /videos/scene.mp4 \
    --fps 15

```

The script creates a `scene_frames/` directory, samples at 15 FPS, and applies the same preprocessing pipeline as image folders.

### Programmatic Usage

For custom data pipelines, import the preprocessing utilities directly:

```python
from lingbot_map.utils.load_fn import load_and_preprocess_images

# Raw file paths from video extraction or custom source

file_list = ["/tmp/frames/000001.jpg", "/tmp/frames/000002.jpg"]

# Generate model-ready tensor

tensor, _ = load_and_preprocess_images(
    file_list,
    mode="crop",          # Alternative: "pad" to preserve aspect ratio

    image_size=518,
    patch_size=14,
)

```

## Key Source Files and Functions

- **[`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py)**: Orchestrates the full pipeline including video frame extraction (`cv2.VideoCapture`), image globbing, and the top-level `load_images()` function.
- **[`lingbot_map/utils/load_fn.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/load_fn.py)**: Contains the core `load_and_preprocess_images()` implementation (lines 65-99) and an alternative `load_and_preprocess_images_square()` helper (lines 16-30) for datasets requiring square inputs.
- **[`lingbot_map/utils/pose_enc.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/pose_enc.py)**: Handles pose encodings for downstream processing (post-preparation stage).
- **[`lingbot_map/utils/geometry.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/geometry.py)**: Provides camera intrinsics handling for later reconstruction stages.

## Summary

- LingBot-Map requires input tensors of shape **(N, 3, H, W)** where H and W are multiples of 14.
- The **`load_images()`** function in [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) provides the end-to-end pipeline for both images and video.
- Video processing uses OpenCV to extract frames at configurable FPS into temporary folders.
- Preprocessing includes EXIF orientation correction, RGB conversion, and bicubic resizing with either cropping or padding.
- The **`load_and_preprocess_images()`** function in [`lingbot_map/utils/load_fn.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/load_fn.py) implements the core tensor preparation logic.

## Frequently Asked Questions

### What image formats does LingBot-Map support?

LingBot-Map supports **JPEG and PNG** files with extensions `.jpg`, `.JPG`, and `.png`. The `load_images()` function in [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) explicitly globs these extensions when scanning directories, though the underlying `load_and_preprocess_images()` function in [`lingbot_map/utils/load_fn.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/load_fn.py) uses Pillow, which can handle additional formats if passed programmatically.

### How does the video frame extraction determine which frames to sample?

The sampling logic in [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) [lines 69-84](https://github.com/Robbyant/lingbot-map/blob/main/demo.py#L69-L84) calculates an interval by dividing the source video's FPS by the target FPS (`interval = max(1, round(src_fps / fps))`), then saves every *interval*-th frame. This ensures consistent temporal sampling without dropping below the requested frame rate.

### Should I use "crop" or "pad" mode for preprocessing?

Use **"crop"** (the default) when you want to maintain the exact target resolution and remove peripheral pixels to achieve patch-size multiples. Use **"pad"** when you must preserve all image content and can tolerate black borders added to reach the required dimensions. The mode is passed to `load_and_preprocess_images()` in [`lingbot_map/utils/load_fn.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/load_fn.py) [lines 65-99](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/load_fn.py#L65-L99).

### Why must image dimensions be multiples of 14?

The patch size of **14** corresponds to the Vision Transformer (ViT) patch embedding stride used in LingBot-Map's architecture. The model splits images into 14×14 pixel patches for processing, requiring input dimensions to be divisible by this value to avoid partial patches. The preprocessing code automatically adjusts the target size to the nearest valid multiple.