# How to Use the LingBot-Map Evaluation Benchmark with KITTI and Oxford Spires Datasets > Learn to use the LingBot-Map evaluation benchmark with KITTI and Oxford Spires datasets. Convert raw data to BSS and compute metrics with detailed reports. - Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map) - Tags: how-to-guide - Published: 2026-06-29 --- **The LingBot-Map evaluation benchmark converts raw KITTI and Oxford Spires data into a unified Benchmark Storage Structure (BSS) using [`benchmark/prepare.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/prepare.py), then computes quantitative metrics via [`benchmark/evaluate.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/evaluate.py) with per-scene JSON reports and HTML summaries.** The LingBot-Map repository provides a complete evaluation framework for visual localization and mapping systems. This guide walks you through the standard workflow for running the LingBot-Map evaluation benchmark on both the KITTI Odometry and Oxford Spires datasets, covering dataset preparation, BSS conversion, and metric computation. ## Understanding the Benchmark Storage Structure (BSS) The LingBot-Map evaluation pipeline relies on a **Benchmark Storage Structure (BSS)** that standardizes how raw dataset frames are stored and accessed. According to the source code in [`benchmark/prepare.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/prepare.py) (lines 22-30), the pipeline loads frames in parallel, resizes images when necessary, and writes them alongside pose and intrinsics data into a unified directory layout called `BSSArtifact`. This abstraction allows the evaluation framework to treat KITTI and Oxford Spires identically once they enter the BSS format. ## Preparing the KITTI Dataset ### Dataset Structure Requirements The **KittiDataset** loader in [`benchmark/datasets/kitti.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/datasets/kitti.py) expects the original KITTI Odometry directory layout. You need the `poses/` folder containing ground-truth trajectories and the `sequences/` folder with calibrated camera data. The benchmark supports both full-resolution data and a down-sampled variant labeled `kitti_504x280` for faster prototyping. ### Running the BSS Preparation First, download the KITTI Odometry dataset and extract it to your data directory, such as `/data/kitti_odometry`. Then run the preparation script: ```bash benchmark/prepare.py --config benchmark/configs/kitti.yaml ``` This command reads frames through the KITTI loader, converts them to the BSS format, and stores the resulting artifacts in the workspace directory defined in the configuration file. ### Configuration File Details The file [`benchmark/configs/kitti.yaml`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/configs/kitti.yaml) defines the workspace path, dataset list (including `kitti_504x280`), and the method to evaluate (typically `lingbot_map`). The workspace path determines where the BSS tree is created, usually under `/kitti/...`. ## Preparing the Oxford Spires Dataset ### Preprocessing Raw Data Unlike KITTI, Oxford Spires requires an initial preprocessing step to generate the required folder structure. The script [`preprocess/oxford.py`](https://github.com/Robbyant/lingbot-map/blob/main/preprocess/oxford.py) creates `images/`, `depth/`, [`intrinsics.txt`](https://github.com/Robbyant/lingbot-map/blob/main/intrinsics.txt), and [`poses_c2w.txt`](https://github.com/Robbyant/lingbot-map/blob/main/poses_c2w.txt) directories from the raw sequence data: ```bash python preprocess/oxford.py --dataset_dir /data/oxford_spires --output_dir /data/oxford_spires_prepped ``` The **OxfordSpiresDataset** loader in [`benchmark/datasets/oxford_spires.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/datasets/oxford_spires.py) then reads these pre-processed files directly during BSS preparation. ### BSS Conversion and Evaluation After preprocessing, convert the data to BSS format and run evaluation: ```bash benchmark/prepare.py --config benchmark/configs/oxford.yaml benchmark/evaluate.py --config benchmark/configs/oxford.yaml ``` The configuration file [`benchmark/configs/oxford.yaml`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/configs/oxford.yaml) points to the pre-processed output directory and selects the same `lingbot_map` method used for KITTI. ## Running the Evaluation Pipeline ### The prepare.py Script The [`benchmark/prepare.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/prepare.py) script serves as the entry point for dataset ingestion. As implemented in the LingBot-Map source code, it handles parallel loading, optional sampling, and writes RGB images, poses, and intrinsics to the BSS artifact storage. This ensures consistent data formatting regardless of the original dataset source. ### The evaluate.py Script The [`benchmark/evaluate.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/evaluate.py) script creates an `Evaluator` instance (lines 24-28) that reads ground-truth and prediction artifacts. It runs the selected evaluators—such as `TrajectoryEvaluator`, `AUCEvaluator`, and depth or point-cloud evaluators—then writes per-scene results via `pred_artifact.save_eval`. The final output includes JSON files in `/dataset/eval/` and an aggregated HTML report in `/dataset/report/`. By default, only trajectory metrics are computed. To enable point-cloud accuracy for Oxford Spires, modify [`benchmark/configs/oxford.yaml`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/configs/oxford.yaml) to set `evaluation.points.enable: true`. ## Advanced Configuration Options The benchmark framework automatically handles sampling strategies, windowed inference, and key-frame caching. You can control these via command-line arguments when running the model: ```bash # Example: Windowed inference with specific keyframe intervals python demo.py \ --model_path /path/to/lingbot-map-long.pt \ --image_folder /data/kitti_odometry/sequences/00/image_2 \ --mask_sky \ --mode windowed --window_size 128 --keyframe_interval 2 ``` For point-cloud evaluation on Oxford Spires, enable the metric in the configuration file: ```bash sed -i 's/enable: false/enable: true/' benchmark/configs/oxford.yaml benchmark/evaluate.py --config benchmark/configs/oxford.yaml ``` ## Summary - The LingBot-Map evaluation benchmark uses a **Benchmark Storage Structure (BSS)** to unify dataset handling across KITTI and Oxford Spires. - **KITTI** data requires only the standard Odometry format, while **Oxford Spires** needs preprocessing via [`preprocess/oxford.py`](https://github.com/Robbyant/lingbot-map/blob/main/preprocess/oxford.py). - Use [`benchmark/prepare.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/prepare.py) to convert raw data to BSS, and [`benchmark/evaluate.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/evaluate.py) to compute trajectory, AUC, depth, and point-cloud metrics. - Configuration files in `benchmark/configs/` control the workspace, dataset selection, and which evaluators are active. - Results are saved as per-scene JSON files and aggregated HTML reports in the workspace directory. ## Frequently Asked Questions ### What is the Benchmark Storage Structure (BSS) in LingBot-Map? The BSS is a unified directory layout defined in the LingBot-Map source code that stores RGB images, camera poses, intrinsics, and optional depth maps in a standardized format. The [`benchmark/prepare.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/prepare.py) script creates `BSSArtifact` objects that allow the evaluation pipeline to treat different datasets identically, regardless of their original format. ### Can I evaluate on full-resolution KITTI data instead of the 504x280 variant? Yes. The [`benchmark/configs/kitti.yaml`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/configs/kitti.yaml) file lists available datasets, and you can modify the configuration to point to full-resolution KITTI sequences instead of `kitti_504x280`. Ensure your system has sufficient memory and storage, as the BSS conversion will process the original image sizes. ### How do I enable point-cloud metrics for Oxford Spires evaluation? By default, the Oxford Spires configuration only evaluates trajectory accuracy. To compute point-cloud metrics, edit [`benchmark/configs/oxford.yaml`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/configs/oxford.yaml) and set `evaluation.points.enable: true` before running [`benchmark/evaluate.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/evaluate.py). This activates the point-cloud evaluator in addition to the standard trajectory analysis. ### Where are the evaluation results stored after running the benchmark? After running [`benchmark/evaluate.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/evaluate.py), per-scene JSON results are written to `//eval/`, where `` is defined in your configuration file. An aggregated HTML report is generated in `//report/`, providing a comprehensive overview of the quantitative metrics across all scenes.