# How to Add New Dataset Support to LingBot-Map’s Benchmark Evaluation Framework

> Learn to add new dataset support to LingBot-Map's benchmark evaluation framework. Subclass BaseDataset, implement data loading, register your class, and update config for seamless integration.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: how-to-guide
- Published: 2026-07-31

---

**To add new dataset support to the LingBot-Map benchmark evaluation framework, subclass `BaseDataset` from [`benchmark/benchmark/dataset/base.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/dataset/base.py), implement the required data loading interface, register the class in [`benchmark/datasets/__init__.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/datasets/__init__.py), and reference it in your benchmark configuration file.**

LingBot-Map provides a flexible evaluation pipeline for mapping and localization algorithms. By implementing a standardized interface, you can integrate custom data sources—from indoor scans to drone footage—without modifying any core evaluation logic.

## Understanding the BaseDataset Interface

The benchmark framework discovers datasets through Python classes that inherit from the abstract `BaseDataset` defined in [`benchmark/benchmark/dataset/base.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/dataset/base.py). This contract standardizes how the evaluation pipeline accesses scene lists, frame sequences, and sensor readings.

### Required Methods

Your subclass must implement three mandatory methods and one optional method:

- **`get_scenes(self) → List[str]`** – Returns a list of scene identifiers, usually folder names. See the reference implementation in `KittiDataset.get_scenes` at [`benchmark/datasets/kitti.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/datasets/kitti.py) lines 78-86.
- **`get_frame_list(self, scene: str) → List[int]`** – Returns the frame indices to be processed for a given scene. Reference implementation: [`benchmark/datasets/kitti.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/datasets/kitti.py) lines 88-92.
- **`load_frame_data(self, scene: str, frame_id: int) → Dict[str, Any]`** – Loads per-frame data and returns a dictionary containing at least the key **`rgb`** (a `uint8` image). Optional keys include `depth`, `mask`, `pose`, `intrinsics`, and custom data. Reference: [`benchmark/datasets/kitti.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/datasets/kitti.py) lines 93-106.
- **`load_global_data(self, scene: str) → Dict[str, Any]`** (optional) – Returns scene-level data such as a ground-truth point cloud. Reference: [`benchmark/datasets/kitti.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/datasets/kitti.py) lines 53-57.

The base class also provides a default **`apply_sampling`** method that you can reuse or override if your dataset requires a custom sampling strategy.

### Optional Save Hooks

If your dataset outputs extra modalities (e.g., semantic masks or surface normals), implement methods named `__save_<key>_file__` with the signature `(self, output_dir: Path, base_name: str, data: np.ndarray)`. The framework automatically discovers these hooks during BSS-Saver initialization. See the documentation in [`benchmark/benchmark/dataset/base.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/dataset/base.py) lines 21-30.

## Step-by-Step Implementation Guide

### Step 1: Create the Dataset Module

Create a new Python file under `benchmark/datasets/`, for example [`my_dataset.py`](https://github.com/Robbyant/lingbot-map/blob/main/my_dataset.py). This file will contain your loader implementation.

### Step 2: Implement the Interface

Import `BaseDataset` and implement the required methods. Ensure `load_frame_data` returns a dictionary with at least the `rgb` key to satisfy the pipeline requirements.

### Step 3: Register the Dataset

Add an import statement to [`benchmark/datasets/__init__.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/datasets/__init__.py) to expose your class to the configuration loader:

```python
from .my_dataset import MyDataset  # noqa: F401

```

If [`__init__.py`](https://github.com/Robbyant/lingbot-map/blob/main/__init__.py) does not exist, create it with this import statement.

### Step 4: Configure the Benchmark

Reference your dataset in a YAML or JSON configuration file (consumed by [`benchmark/prepare.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/prepare.py)). Set `dataset.name` to the fully-qualified class name and provide constructor arguments under `dataset.args`:

```yaml
dataset:
  name: benchmark.datasets.my_dataset.MyDataset
  args:
    raw_data_root: /path/to/my_dataset_root
    sequences: [train, test]

```

The loader in [`benchmark/benchmark/core/loader.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/core/loader.py) instantiates your class via `importlib` and passes the supplied kwargs to your constructor.

## Complete Working Example

Below is a minimal implementation for a folder-structured image sequence dataset.

```python

# benchmark/datasets/my_dataset.py

import numpy as np
from pathlib import Path
from PIL import Image
from benchmark.benchmark.dataset.base import BaseDataset

class MyDataset(BaseDataset):
    """Simple loader for a folder-structured image sequence.

    Expected layout:
    └─ <raw_root>/
       ├─ scene_A/
       │  ├─ images/
       │  │  ├─ 000000.png
       │  │  ├─ 000001.png
       │  │  └─ …
       │  └─ poses.txt            # optional, one 4x4 matrix per line

       └─ scene_B/
          …
    """

    def __init__(self, raw_data_root: str, sequences: list | None = None):
        super().__init__(raw_data_root)
        # optional whitelist of scenes

        self._whitelist = sequences

    def get_scenes(self) -> list[str]:
        if self._whitelist is not None:
            return self._whitelist
        return sorted(p.name for p in self.raw_data_root.iterdir() if p.is_dir())

    def get_frame_list(self, scene: str) -> list[int]:
        img_dir = self.raw_data_root / scene / "images"
        return sorted(int(p.stem) for p in img_dir.glob("*.png"))

    def load_frame_data(self, scene: str, frame_id: int) -> dict:
        img_path = self.raw_data_root / scene / "images" / f"{frame_id:06d}.png"
        rgb = np.array(Image.open(img_path).convert("RGB"))
        # optional pose loading

        pose = None
        poses_path = self.raw_data_root / scene / "poses.txt"
        if poses_path.is_file():
            pose = np.loadtxt(poses_path)[frame_id]
        # intrinsics are hard-coded here but could be read from a calibration file

        intrinsics = np.array([500.0, 500.0, 320.0, 240.0], dtype=np.float32)
        return {"rgb": rgb, "pose": pose, "intrinsics": intrinsics}

```

Register the class in [`benchmark/datasets/__init__.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/datasets/__init__.py):

```python
from .my_dataset import MyDataset  # noqa: F401

```

Example benchmark configuration:

```yaml
dataset:
  name: benchmark.datasets.my_dataset.MyDataset
  args:
    raw_data_root: /data/my_dataset
    sequences: [scene_A, scene_B]

```

Running `python benchmark/prepare.py` or `python benchmark/run.py` will now process `scene_A` and `scene_B` using your custom logic.

## Summary

- **Subclass `BaseDataset`** from [`benchmark/benchmark/dataset/base.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/dataset/base.py) to ensure compatibility with the evaluation pipeline.
- **Implement four core methods**: Provide `get_scenes`, `get_frame_list`, `load_frame_data` (must include `rgb` key), and optionally `load_global_data` for scene-level ground truth.
- **Register in [`__init__.py`](https://github.com/Robbyant/lingbot-map/blob/main/__init__.py)**: Import your class in [`benchmark/datasets/__init__.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/datasets/__init__.py) to make it discoverable by name in configuration files.
- **Configure via YAML**: Reference the fully-qualified class name (e.g., `benchmark.datasets.my_dataset.MyDataset`) in the benchmark config under `dataset.name`, passing constructor arguments via `dataset.args`.
- **Leverage save hooks**: Implement `__save_<key>_file__` methods to output custom modalities beyond standard RGB and depth data.

## Frequently Asked Questions

### What file format should the RGB images be in?

The `load_frame_data` method must return a dictionary with an `rgb` key containing a NumPy array of shape `(H, W, 3)` with dtype `uint8`. Your loader can read any image format (PNG, JPEG, etc.) using libraries like PIL or OpenCV, provided you convert the result to this NumPy representation before returning.

### Do I need to modify the core benchmark code to add my dataset?

No. The framework uses dynamic loading via `importlib` in [`benchmark/benchmark/core/loader.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/core/loader.py) to instantiate your dataset class from the configuration file. As long as you subclass `BaseDataset` and register the import in [`benchmark/datasets/__init__.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/datasets/__init__.py), no changes to [`prepare.py`](https://github.com/Robbyant/lingbot-map/blob/main/prepare.py), [`run.py`](https://github.com/Robbyant/lingbot-map/blob/main/run.py), or other core modules are required.

### How do I handle datasets without pose information?

The `pose` key in the dictionary returned by `load_frame_data` is optional. If your dataset lacks ground-truth poses, simply omit the key or set it to `None`. The benchmark pipeline will skip pose-dependent evaluations unless specifically configured otherwise in the experiment config.

### Can I implement custom sampling strategies for my dataset?

Yes. While `BaseDataset` provides a default `apply_sampling` implementation, you can override this method in your subclass to implement custom frame skipping, random sampling, or temporal windowing specific to your dataset's characteristics. The framework calls this method before processing frames if sampling is enabled in the configuration.