# How to Benchmark LingBot-Map Performance on KITTI and Oxford Spires Datasets

> Benchmark LingBot-Map performance on KITTI and Oxford Spires datasets. Learn to convert data to BSS and run quantitative metrics using prepare.py and evaluate.py.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: performance
- Published: 2026-07-21

---

**Benchmarking LingBot-Map on KITTI and Oxford Spires involves converting raw data to the Benchmark Storage Structure (BSS) using [`benchmark/prepare.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/prepare.py) and running quantitative metrics via [`benchmark/evaluate.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/evaluate.py) with YAML configuration files.**

The LingBot-Map repository provides a complete evaluation pipeline that standardizes datasets into a unified format and computes trajectory, AUC, depth, and point-cloud metrics. This guide walks through the exact steps to benchmark LingBot-Map performance on KITTI and Oxford Spires datasets using the official scripts and configuration files.

## Overview of the Benchmark Pipeline

The benchmark system operates through three distinct stages:

1. **Dataset Loading** – Raw sequences are parsed by dataset-specific loaders (`KittiDataset` or `OxfordSpiresDataset`) that extract RGB frames, camera poses, and intrinsics
2. **BSS Conversion** – The [`benchmark/prepare.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/prepare.py) script transforms loaded frames into the **Benchmark Storage Structure**, a standardized directory layout containing images, poses, and optional depth maps
3. **Quantitative Evaluation** – [`benchmark/evaluate.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/evaluate.py) runs per-scene evaluators (`TrajectoryEvaluator`, `AUCEvaluator`, etc.) against ground truth, outputting JSON metrics and HTML reports

Both KITTI Odometry and Oxford Spires are natively supported with pre-built configuration files in `benchmark/configs/`.

## Prerequisites and Installation

Set up the environment and install dependencies before running benchmarks:

```bash
conda create -n lingbot-map python=3.10 -y
conda activate lingbot-map
pip install -e .

```

For accelerated inference during evaluation, optionally install flash-infer:

```bash
pip install flashinfer-python

```

## Benchmarking on KITTI Dataset

The KITTI loader in [`benchmark/datasets/kitti.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/datasets/kitti.py) expects the standard KITTI Odometry layout with `poses/` and `sequences/` directories.

### Dataset Preparation

Download the KITTI Odometry dataset (poses and sequences) and extract it to your data directory. The benchmark supports both full-resolution data and the down-sampled `kitti_504x280` variant.

Convert the raw data to BSS format:

```bash
benchmark/prepare.py --config benchmark/configs/kitti.yaml

```

This script (lines 22-30 in [`benchmark/prepare.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/prepare.py)) loads frames in parallel, resizes images if specified, and writes `BSSArtifact` objects containing RGB, pose, and intrinsics to the workspace directory.

### Running the Evaluation

Execute the benchmark with the default configuration:

```bash
benchmark/evaluate.py --config benchmark/configs/kitti.yaml

```

The configuration file [`benchmark/configs/kitti.yaml`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/configs/kitti.yaml) defines the workspace path, selects the `kitti_504x280` dataset, and specifies the `lingbot_map` method. Results appear in `<workspace>/kitti/eval/` as per-scene JSON files, with an aggregated HTML report in `<workspace>/kitti/report/`.

## Benchmarking on Oxford Spires Dataset

Oxford Spires requires preprocessing to generate the required folder structure before BSS conversion.

### Preprocessing Raw Data

Run the preprocessing script to generate standardized inputs:

```bash
python preprocess/oxford.py --dataset_dir /data/oxford_spires --output_dir /data/oxford_spires_prepped

```

This creates `images/`, `depth/`, [`intrinsics.txt`](https://github.com/Robbyant/lingbot-map/blob/main/intrinsics.txt), and [`poses_c2w.txt`](https://github.com/Robbyant/lingbot-map/blob/main/poses_c2w.txt) in the output directory. The `OxfordSpiresDataset` class in [`benchmark/datasets/oxford_spires.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/datasets/oxford_spires.py) reads these pre-processed files directly.

### BSS Conversion and Evaluation

Convert the pre-processed data to BSS:

```bash
benchmark/prepare.py --config benchmark/configs/oxford.yaml

```

Run the evaluation (trajectory metrics enabled by default):

```bash
benchmark/evaluate.py --config benchmark/configs/oxford.yaml

```

To enable point-cloud accuracy metrics, modify [`benchmark/configs/oxford.yaml`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/configs/oxford.yaml) to set `evaluation.points.enable: true` before running the evaluation script.

## Understanding the Evaluation Scripts

The benchmark framework handles sampling and inference automatically through two core scripts:

- **[`benchmark/prepare.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/prepare.py)** – As implemented in lines 22-30, this script initializes the dataset loader, processes frames with optional resizing, and serializes artifacts to the BSS tree structure
- **[`benchmark/evaluate.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/evaluate.py)** – Lines 24-28 instantiate an `Evaluator` object that loads ground-truth and prediction artifacts, runs the selected evaluators, and calls `pred_artifact.save_eval` to write JSON results

The system automatically manages key-frame caching and windowed inference based on method-specific command-line arguments passed through the configuration.

## Advanced Configuration and Windowed Inference

The YAML files in `benchmark/configs/` control evaluation behavior. Key parameters include:

- **`evaluation.points.enable`** – Toggle point-cloud metrics (default: false)
- **`keyframe_interval`** – Control temporal sampling density
- **`mode`** – Set to `windowed` for streaming evaluation with `--window_size`

Example: Run full-resolution KITTI with windowed inference:

```bash
python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --image_folder /data/kitti_odometry/sequences/00/image_2 \
    --mask_sky \
    --mode windowed --window_size 128 --keyframe_interval 2

```

Example: Enable point-cloud metrics for Oxford Spires:

```bash
sed -i 's/enable: false/enable: true/' benchmark/configs/oxford.yaml
benchmark/prepare.py --config benchmark/configs/oxford.yaml
benchmark/evaluate.py --config benchmark/configs/oxford.yaml

```

## Summary

- **Standardized Pipeline** – LingBot-Map uses the Benchmark Storage Structure (BSS) to unify KITTI and Oxford Spires data formats through [`benchmark/prepare.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/prepare.py)
- **Dataset Loaders** – [`benchmark/datasets/kitti.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/datasets/kitti.py) parses KITTI Odometry layout while [`benchmark/datasets/oxford_spires.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/datasets/oxford_spires.py) reads pre-processed Oxford Spires data
- **Configuration-Driven** – YAML files in `benchmark/configs/` define workspaces, datasets, and enabled metrics (trajectory, AUC, depth, point-cloud)
- **Preprocessing Required** – Oxford Spires requires running [`preprocess/oxford.py`](https://github.com/Robbyant/lingbot-map/blob/main/preprocess/oxford.py) to generate [`poses_c2w.txt`](https://github.com/Robbyant/lingbot-map/blob/main/poses_c2w.txt) and [`intrinsics.txt`](https://github.com/Robbyant/lingbot-map/blob/main/intrinsics.txt) before benchmark conversion
- **Quantitative Output** – [`benchmark/evaluate.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/evaluate.py) generates per-scene JSON metrics and aggregated HTML reports in the workspace directory

## Frequently Asked Questions

### What file structure does the KITTI benchmark expect?

The `KittiDataset` loader expects the standard KITTI Odometry directory layout with `poses/` containing ground-truth pose files and `sequences/` containing image subdirectories. The benchmark supports both original resolution and the `kitti_504x280` down-sampled variant specified in [`benchmark/configs/kitti.yaml`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/configs/kitti.yaml).

### How do I enable point-cloud evaluation metrics for Oxford Spires?

Edit [`benchmark/configs/oxford.yaml`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/configs/oxford.yaml) and set `evaluation.points.enable` to `true` before running [`benchmark/evaluate.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/evaluate.py). By default, only trajectory metrics are computed to reduce computation time.

### Can I run the benchmark on custom datasets?

Yes, by implementing a new dataset loader class similar to [`benchmark/datasets/kitti.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/datasets/kitti.py) or [`benchmark/datasets/oxford_spires.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/datasets/oxford_spires.py) that returns RGB images, camera poses, and intrinsics. The loader must integrate with [`benchmark/prepare.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/prepare.py) to export data into the BSS format required by the evaluation pipeline.

### What is the difference between [`prepare.py`](https://github.com/Robbyant/lingbot-map/blob/main/prepare.py) and [`evaluate.py`](https://github.com/Robbyant/lingbot-map/blob/main/evaluate.py)?

[`benchmark/prepare.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/prepare.py) converts raw dataset formats into the standardized Benchmark Storage Structure (BSS), handling parallel loading and image resizing. [`benchmark/evaluate.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/evaluate.py) performs the actual quantitative comparison between predictions and ground truth, running evaluators like `TrajectoryEvaluator` and `AUCEvaluator` to generate JSON reports and HTML summaries.