How to Use the LingBot-Map Evaluation Benchmark with KITTI and Oxford Spires Datasets

The LingBot-Map evaluation benchmark converts raw KITTI and Oxford Spires data into a unified Benchmark Storage Structure (BSS) using benchmark/prepare.py, then computes quantitative metrics via benchmark/evaluate.py with per-scene JSON reports and HTML summaries.

The LingBot-Map repository provides a complete evaluation framework for visual localization and mapping systems. This guide walks you through the standard workflow for running the LingBot-Map evaluation benchmark on both the KITTI Odometry and Oxford Spires datasets, covering dataset preparation, BSS conversion, and metric computation.

Understanding the Benchmark Storage Structure (BSS)

The LingBot-Map evaluation pipeline relies on a Benchmark Storage Structure (BSS) that standardizes how raw dataset frames are stored and accessed. According to the source code in benchmark/prepare.py (lines 22-30), the pipeline loads frames in parallel, resizes images when necessary, and writes them alongside pose and intrinsics data into a unified directory layout called BSSArtifact. This abstraction allows the evaluation framework to treat KITTI and Oxford Spires identically once they enter the BSS format.

Preparing the KITTI Dataset

Dataset Structure Requirements

The KittiDataset loader in benchmark/datasets/kitti.py expects the original KITTI Odometry directory layout. You need the poses/ folder containing ground-truth trajectories and the sequences/ folder with calibrated camera data. The benchmark supports both full-resolution data and a down-sampled variant labeled kitti_504x280 for faster prototyping.

Running the BSS Preparation

First, download the KITTI Odometry dataset and extract it to your data directory, such as /data/kitti_odometry. Then run the preparation script:

benchmark/prepare.py --config benchmark/configs/kitti.yaml

This command reads frames through the KITTI loader, converts them to the BSS format, and stores the resulting artifacts in the workspace directory defined in the configuration file.

Configuration File Details

The file benchmark/configs/kitti.yaml defines the workspace path, dataset list (including kitti_504x280), and the method to evaluate (typically lingbot_map). The workspace path determines where the BSS tree is created, usually under <workspace>/kitti/....

Preparing the Oxford Spires Dataset

Preprocessing Raw Data

Unlike KITTI, Oxford Spires requires an initial preprocessing step to generate the required folder structure. The script preprocess/oxford.py creates images/, depth/, intrinsics.txt, and poses_c2w.txt directories from the raw sequence data:

python preprocess/oxford.py --dataset_dir /data/oxford_spires --output_dir /data/oxford_spires_prepped

The OxfordSpiresDataset loader in benchmark/datasets/oxford_spires.py then reads these pre-processed files directly during BSS preparation.

BSS Conversion and Evaluation

After preprocessing, convert the data to BSS format and run evaluation:

benchmark/prepare.py --config benchmark/configs/oxford.yaml
benchmark/evaluate.py --config benchmark/configs/oxford.yaml

The configuration file benchmark/configs/oxford.yaml points to the pre-processed output directory and selects the same lingbot_map method used for KITTI.

Running the Evaluation Pipeline

The prepare.py Script

The benchmark/prepare.py script serves as the entry point for dataset ingestion. As implemented in the LingBot-Map source code, it handles parallel loading, optional sampling, and writes RGB images, poses, and intrinsics to the BSS artifact storage. This ensures consistent data formatting regardless of the original dataset source.

The evaluate.py Script

The benchmark/evaluate.py script creates an Evaluator instance (lines 24-28) that reads ground-truth and prediction artifacts. It runs the selected evaluators—such as TrajectoryEvaluator, AUCEvaluator, and depth or point-cloud evaluators—then writes per-scene results via pred_artifact.save_eval. The final output includes JSON files in <workspace>/dataset/eval/ and an aggregated HTML report in <workspace>/dataset/report/.

By default, only trajectory metrics are computed. To enable point-cloud accuracy for Oxford Spires, modify benchmark/configs/oxford.yaml to set evaluation.points.enable: true.

Advanced Configuration Options

The benchmark framework automatically handles sampling strategies, windowed inference, and key-frame caching. You can control these via command-line arguments when running the model:

# Example: Windowed inference with specific keyframe intervals

python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --image_folder /data/kitti_odometry/sequences/00/image_2 \
    --mask_sky \
    --mode windowed --window_size 128 --keyframe_interval 2

For point-cloud evaluation on Oxford Spires, enable the metric in the configuration file:

sed -i 's/enable: false/enable: true/' benchmark/configs/oxford.yaml
benchmark/evaluate.py --config benchmark/configs/oxford.yaml

Summary

• The LingBot-Map evaluation benchmark uses a Benchmark Storage Structure (BSS) to unify dataset handling across KITTI and Oxford Spires.
• KITTI data requires only the standard Odometry format, while Oxford Spires needs preprocessing via preprocess/oxford.py.
• Use benchmark/prepare.py to convert raw data to BSS, and benchmark/evaluate.py to compute trajectory, AUC, depth, and point-cloud metrics.
• Configuration files in benchmark/configs/ control the workspace, dataset selection, and which evaluators are active.
• Results are saved as per-scene JSON files and aggregated HTML reports in the workspace directory.

Frequently Asked Questions

What is the Benchmark Storage Structure (BSS) in LingBot-Map?

The BSS is a unified directory layout defined in the LingBot-Map source code that stores RGB images, camera poses, intrinsics, and optional depth maps in a standardized format. The benchmark/prepare.py script creates BSSArtifact objects that allow the evaluation pipeline to treat different datasets identically, regardless of their original format.

Can I evaluate on full-resolution KITTI data instead of the 504x280 variant?

Yes. The benchmark/configs/kitti.yaml file lists available datasets, and you can modify the configuration to point to full-resolution KITTI sequences instead of kitti_504x280. Ensure your system has sufficient memory and storage, as the BSS conversion will process the original image sizes.

How do I enable point-cloud metrics for Oxford Spires evaluation?

By default, the Oxford Spires configuration only evaluates trajectory accuracy. To compute point-cloud metrics, edit benchmark/configs/oxford.yaml and set evaluation.points.enable: true before running benchmark/evaluate.py. This activates the point-cloud evaluator in addition to the standard trajectory analysis.

Where are the evaluation results stored after running the benchmark?

After running benchmark/evaluate.py, per-scene JSON results are written to <workspace>/<dataset>/eval/, where <workspace> is defined in your configuration file. An aggregated HTML report is generated in <workspace>/<dataset>/report/, providing a comprehensive overview of the quantitative metrics across all scenes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}