How to Add New Dataset Support to LingBot-Map’s Benchmark Evaluation Framework

To add new dataset support to the LingBot-Map benchmark evaluation framework, subclass BaseDataset from benchmark/benchmark/dataset/base.py, implement the required data loading interface, register the class in benchmark/datasets/__init__.py, and reference it in your benchmark configuration file.

LingBot-Map provides a flexible evaluation pipeline for mapping and localization algorithms. By implementing a standardized interface, you can integrate custom data sources—from indoor scans to drone footage—without modifying any core evaluation logic.

Understanding the BaseDataset Interface

The benchmark framework discovers datasets through Python classes that inherit from the abstract BaseDataset defined in benchmark/benchmark/dataset/base.py. This contract standardizes how the evaluation pipeline accesses scene lists, frame sequences, and sensor readings.

Required Methods

Your subclass must implement three mandatory methods and one optional method:

• get_scenes(self) → List[str] – Returns a list of scene identifiers, usually folder names. See the reference implementation in KittiDataset.get_scenes at benchmark/datasets/kitti.py lines 78-86.
• get_frame_list(self, scene: str) → List[int] – Returns the frame indices to be processed for a given scene. Reference implementation: benchmark/datasets/kitti.py lines 88-92.
• load_frame_data(self, scene: str, frame_id: int) → Dict[str, Any] – Loads per-frame data and returns a dictionary containing at least the key rgb (a uint8 image). Optional keys include depth, mask, pose, intrinsics, and custom data. Reference: benchmark/datasets/kitti.py lines 93-106.
• load_global_data(self, scene: str) → Dict[str, Any] (optional) – Returns scene-level data such as a ground-truth point cloud. Reference: benchmark/datasets/kitti.py lines 53-57.

The base class also provides a default apply_sampling method that you can reuse or override if your dataset requires a custom sampling strategy.

Optional Save Hooks

If your dataset outputs extra modalities (e.g., semantic masks or surface normals), implement methods named __save_<key>_file__ with the signature (self, output_dir: Path, base_name: str, data: np.ndarray). The framework automatically discovers these hooks during BSS-Saver initialization. See the documentation in benchmark/benchmark/dataset/base.py lines 21-30.

Step-by-Step Implementation Guide

Step 1: Create the Dataset Module

Create a new Python file under benchmark/datasets/, for example my_dataset.py. This file will contain your loader implementation.

Step 2: Implement the Interface

Import BaseDataset and implement the required methods. Ensure load_frame_data returns a dictionary with at least the rgb key to satisfy the pipeline requirements.

Step 3: Register the Dataset

Add an import statement to benchmark/datasets/__init__.py to expose your class to the configuration loader:

from .my_dataset import MyDataset  # noqa: F401

If __init__.py does not exist, create it with this import statement.

Step 4: Configure the Benchmark

Reference your dataset in a YAML or JSON configuration file (consumed by benchmark/prepare.py). Set dataset.name to the fully-qualified class name and provide constructor arguments under dataset.args:

dataset:
  name: benchmark.datasets.my_dataset.MyDataset
  args:
    raw_data_root: /path/to/my_dataset_root
    sequences: [train, test]

The loader in benchmark/benchmark/core/loader.py instantiates your class via importlib and passes the supplied kwargs to your constructor.

Complete Working Example

Below is a minimal implementation for a folder-structured image sequence dataset.

# benchmark/datasets/my_dataset.py

import numpy as np
from pathlib import Path
from PIL import Image
from benchmark.benchmark.dataset.base import BaseDataset

class MyDataset(BaseDataset):
    """Simple loader for a folder-structured image sequence.

    Expected layout:
    └─ <raw_root>/
       ├─ scene_A/
       │  ├─ images/
       │  │  ├─ 000000.png
       │  │  ├─ 000001.png
       │  │  └─ …
       │  └─ poses.txt            # optional, one 4x4 matrix per line

       └─ scene_B/
          …
    """

    def __init__(self, raw_data_root: str, sequences: list | None = None):
        super().__init__(raw_data_root)
        # optional whitelist of scenes

        self._whitelist = sequences

    def get_scenes(self) -> list[str]:
        if self._whitelist is not None:
            return self._whitelist
        return sorted(p.name for p in self.raw_data_root.iterdir() if p.is_dir())

    def get_frame_list(self, scene: str) -> list[int]:
        img_dir = self.raw_data_root / scene / "images"
        return sorted(int(p.stem) for p in img_dir.glob("*.png"))

    def load_frame_data(self, scene: str, frame_id: int) -> dict:
        img_path = self.raw_data_root / scene / "images" / f"{frame_id:06d}.png"
        rgb = np.array(Image.open(img_path).convert("RGB"))
        # optional pose loading

        pose = None
        poses_path = self.raw_data_root / scene / "poses.txt"
        if poses_path.is_file():
            pose = np.loadtxt(poses_path)[frame_id]
        # intrinsics are hard-coded here but could be read from a calibration file

        intrinsics = np.array([500.0, 500.0, 320.0, 240.0], dtype=np.float32)
        return {"rgb": rgb, "pose": pose, "intrinsics": intrinsics}

Register the class in benchmark/datasets/__init__.py:

from .my_dataset import MyDataset  # noqa: F401

Example benchmark configuration:

dataset:
  name: benchmark.datasets.my_dataset.MyDataset
  args:
    raw_data_root: /data/my_dataset
    sequences: [scene_A, scene_B]

Running python benchmark/prepare.py or python benchmark/run.py will now process scene_A and scene_B using your custom logic.

Summary

• Subclass BaseDataset from benchmark/benchmark/dataset/base.py to ensure compatibility with the evaluation pipeline.
• Implement four core methods: Provide get_scenes, get_frame_list, load_frame_data (must include rgb key), and optionally load_global_data for scene-level ground truth.
• Register in __init__.py: Import your class in benchmark/datasets/__init__.py to make it discoverable by name in configuration files.
• Configure via YAML: Reference the fully-qualified class name (e.g., benchmark.datasets.my_dataset.MyDataset) in the benchmark config under dataset.name, passing constructor arguments via dataset.args.
• Leverage save hooks: Implement __save_<key>_file__ methods to output custom modalities beyond standard RGB and depth data.

Frequently Asked Questions

What file format should the RGB images be in?

The load_frame_data method must return a dictionary with an rgb key containing a NumPy array of shape (H, W, 3) with dtype uint8. Your loader can read any image format (PNG, JPEG, etc.) using libraries like PIL or OpenCV, provided you convert the result to this NumPy representation before returning.

Do I need to modify the core benchmark code to add my dataset?

No. The framework uses dynamic loading via importlib in benchmark/benchmark/core/loader.py to instantiate your dataset class from the configuration file. As long as you subclass BaseDataset and register the import in benchmark/datasets/__init__.py, no changes to prepare.py, run.py, or other core modules are required.

How do I handle datasets without pose information?

The pose key in the dictionary returned by load_frame_data is optional. If your dataset lacks ground-truth poses, simply omit the key or set it to None. The benchmark pipeline will skip pose-dependent evaluations unless specifically configured otherwise in the experiment config.

Can I implement custom sampling strategies for my dataset?

Yes. While BaseDataset provides a default apply_sampling implementation, you can override this method in your subclass to implement custom frame skipping, random sampling, or temporal windowing specific to your dataset's characteristics. The framework calls this method before processing frames if sampling is enabled in the configuration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}