How to Benchmark LingBot-Map Performance on KITTI and Oxford Spires Datasets

Benchmarking LingBot-Map on KITTI and Oxford Spires involves converting raw data to the Benchmark Storage Structure (BSS) using benchmark/prepare.py and running quantitative metrics via benchmark/evaluate.py with YAML configuration files.

The LingBot-Map repository provides a complete evaluation pipeline that standardizes datasets into a unified format and computes trajectory, AUC, depth, and point-cloud metrics. This guide walks through the exact steps to benchmark LingBot-Map performance on KITTI and Oxford Spires datasets using the official scripts and configuration files.

Overview of the Benchmark Pipeline

The benchmark system operates through three distinct stages:

1. Dataset Loading – Raw sequences are parsed by dataset-specific loaders (KittiDataset or OxfordSpiresDataset) that extract RGB frames, camera poses, and intrinsics
2. BSS Conversion – The benchmark/prepare.py script transforms loaded frames into the Benchmark Storage Structure, a standardized directory layout containing images, poses, and optional depth maps
3. Quantitative Evaluation – benchmark/evaluate.py runs per-scene evaluators (TrajectoryEvaluator, AUCEvaluator, etc.) against ground truth, outputting JSON metrics and HTML reports

Both KITTI Odometry and Oxford Spires are natively supported with pre-built configuration files in benchmark/configs/.

Prerequisites and Installation

Set up the environment and install dependencies before running benchmarks:

conda create -n lingbot-map python=3.10 -y
conda activate lingbot-map
pip install -e .

For accelerated inference during evaluation, optionally install flash-infer:

pip install flashinfer-python

Benchmarking on KITTI Dataset

The KITTI loader in benchmark/datasets/kitti.py expects the standard KITTI Odometry layout with poses/ and sequences/ directories.

Dataset Preparation

Download the KITTI Odometry dataset (poses and sequences) and extract it to your data directory. The benchmark supports both full-resolution data and the down-sampled kitti_504x280 variant.

Convert the raw data to BSS format:

benchmark/prepare.py --config benchmark/configs/kitti.yaml

This script (lines 22-30 in benchmark/prepare.py) loads frames in parallel, resizes images if specified, and writes BSSArtifact objects containing RGB, pose, and intrinsics to the workspace directory.

Running the Evaluation

Execute the benchmark with the default configuration:

benchmark/evaluate.py --config benchmark/configs/kitti.yaml

The configuration file benchmark/configs/kitti.yaml defines the workspace path, selects the kitti_504x280 dataset, and specifies the lingbot_map method. Results appear in <workspace>/kitti/eval/ as per-scene JSON files, with an aggregated HTML report in <workspace>/kitti/report/.

Benchmarking on Oxford Spires Dataset

Oxford Spires requires preprocessing to generate the required folder structure before BSS conversion.

Preprocessing Raw Data

Run the preprocessing script to generate standardized inputs:

python preprocess/oxford.py --dataset_dir /data/oxford_spires --output_dir /data/oxford_spires_prepped

This creates images/, depth/, intrinsics.txt, and poses_c2w.txt in the output directory. The OxfordSpiresDataset class in benchmark/datasets/oxford_spires.py reads these pre-processed files directly.

BSS Conversion and Evaluation

Convert the pre-processed data to BSS:

benchmark/prepare.py --config benchmark/configs/oxford.yaml

Run the evaluation (trajectory metrics enabled by default):

benchmark/evaluate.py --config benchmark/configs/oxford.yaml

To enable point-cloud accuracy metrics, modify benchmark/configs/oxford.yaml to set evaluation.points.enable: true before running the evaluation script.

Understanding the Evaluation Scripts

The benchmark framework handles sampling and inference automatically through two core scripts:

• benchmark/prepare.py – As implemented in lines 22-30, this script initializes the dataset loader, processes frames with optional resizing, and serializes artifacts to the BSS tree structure
• benchmark/evaluate.py – Lines 24-28 instantiate an Evaluator object that loads ground-truth and prediction artifacts, runs the selected evaluators, and calls pred_artifact.save_eval to write JSON results

The system automatically manages key-frame caching and windowed inference based on method-specific command-line arguments passed through the configuration.

Advanced Configuration and Windowed Inference

The YAML files in benchmark/configs/ control evaluation behavior. Key parameters include:

• evaluation.points.enable – Toggle point-cloud metrics (default: false)
• keyframe_interval – Control temporal sampling density
• mode – Set to windowed for streaming evaluation with --window_size

Example: Run full-resolution KITTI with windowed inference:

python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --image_folder /data/kitti_odometry/sequences/00/image_2 \
    --mask_sky \
    --mode windowed --window_size 128 --keyframe_interval 2

Example: Enable point-cloud metrics for Oxford Spires:

sed -i 's/enable: false/enable: true/' benchmark/configs/oxford.yaml
benchmark/prepare.py --config benchmark/configs/oxford.yaml
benchmark/evaluate.py --config benchmark/configs/oxford.yaml

Summary

• Standardized Pipeline – LingBot-Map uses the Benchmark Storage Structure (BSS) to unify KITTI and Oxford Spires data formats through benchmark/prepare.py
• Dataset Loaders – benchmark/datasets/kitti.py parses KITTI Odometry layout while benchmark/datasets/oxford_spires.py reads pre-processed Oxford Spires data
• Configuration-Driven – YAML files in benchmark/configs/ define workspaces, datasets, and enabled metrics (trajectory, AUC, depth, point-cloud)
• Preprocessing Required – Oxford Spires requires running preprocess/oxford.py to generate poses_c2w.txt and intrinsics.txt before benchmark conversion
• Quantitative Output – benchmark/evaluate.py generates per-scene JSON metrics and aggregated HTML reports in the workspace directory

Frequently Asked Questions

What file structure does the KITTI benchmark expect?

The KittiDataset loader expects the standard KITTI Odometry directory layout with poses/ containing ground-truth pose files and sequences/ containing image subdirectories. The benchmark supports both original resolution and the kitti_504x280 down-sampled variant specified in benchmark/configs/kitti.yaml.

How do I enable point-cloud evaluation metrics for Oxford Spires?

Edit benchmark/configs/oxford.yaml and set evaluation.points.enable to true before running benchmark/evaluate.py. By default, only trajectory metrics are computed to reduce computation time.

Can I run the benchmark on custom datasets?

Yes, by implementing a new dataset loader class similar to benchmark/datasets/kitti.py or benchmark/datasets/oxford_spires.py that returns RGB images, camera poses, and intrinsics. The loader must integrate with benchmark/prepare.py to export data into the BSS format required by the evaluation pipeline.

What is the difference between prepare.py and evaluate.py?

benchmark/prepare.py converts raw dataset formats into the standardized Benchmark Storage Structure (BSS), handling parallel loading and image resizing. benchmark/evaluate.py performs the actual quantitative comparison between predictions and ground truth, running evaluators like TrajectoryEvaluator and AUCEvaluator to generate JSON reports and HTML summaries.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}