# How to Add a New Method to the LingBot-Map Benchmark Evaluation Framework > Learn how to add a new method to the LingBot-Map benchmark evaluation framework. Subclass BaseMethod, implement the process_scene method, and register your class. - Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map) - Tags: how-to-guide - Published: 2026-07-08 --- **To add a new method to the LingBot-Map benchmark, subclass `BaseMethod` from [`benchmark/benchmark/method/base.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/method/base.py), implement the `process_scene` method, and register your class in the `METHOD_REGISTRY` inside [`benchmark/benchmark/method/__init__.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/method/__init__.py).** The LingBot-Map repository provides a modular evaluation framework for SLAM and 3D reconstruction algorithms. Adding a new method requires understanding the three-layer architecture—base interface, concrete implementations, and method registry—that governs how the benchmark driver instantiates and executes evaluation pipelines. This guide walks through the exact steps to integrate custom models while maintaining compatibility with the existing `BSSLoader` and result serialization systems. ## Understanding the Method Architecture The framework separates concerns across three distinct layers. This design allows you to plug in arbitrary reconstruction models without modifying the core evaluation logic. ### Base Interface The abstract contract every method must satisfy is defined in [`benchmark/benchmark/method/base.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/method/base.py). The `BaseMethod` class establishes the API surface: subclasses receive a `resize_context`, alignment parameters, and a logger during initialization, and must implement `process_scene(self, gt_artifact) → Dict[str, Any]`. The returned dictionary must follow a strict schema with `frame` and `global` keys to ensure downstream evaluators can parse the results consistently. ### Method Implementations Concrete adapters reside in the `benchmark/methods/` directory. The reference implementation, `LingbotMapMethod` in [`benchmark/methods/lingbot_map.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/methods/lingbot_map.py), demonstrates how to wrap an external SLAM package. These classes handle model-specific logic such as checkpoint loading, tensor conversion, and inference device management, while translating proprietary outputs into the standardized format required by `BaseMethod`. ### Registry and Factory The benchmark driver discovers methods through a centralized registry in [`benchmark/benchmark/method/__init__.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/method/__init__.py). This file maintains a `METHOD_REGISTRY` dictionary that maps CLI-friendly strings (e.g., `"lingbot_map"`) to class constructors. When you invoke `python benchmark/run.py --method `, the driver looks up the name in this registry and instantiates the corresponding class with remaining command-line arguments. ## Implementing Your Custom Method Adding a new evaluation method involves three concrete steps: creating the subclass, implementing the processing logic, and registering the component. ### Step 1: Subclass BaseMethod Create a new file in `benchmark/methods/` (e.g., [`my_new_method.py`](https://github.com/Robbyant/lingbot-map/blob/main/my_new_method.py)). Inherit from `BaseMethod` and call `super().__init__(align=align, area_budget=area_budget, logger=logger)` to initialize the resize context and logging infrastructure. Define your constructor to accept model-specific parameters such as `checkpoint` paths and `device` strings. ```python # benchmark/methods/my_new_method.py import logging import torch from typing import Any, Dict from benchmark.method.base import BaseMethod class MyNewMethod(BaseMethod): """Adapter for custom reconstruction pipeline.""" def __init__(self, checkpoint: str | None = None, device: str = "cuda", align: int = 14, area_budget: int | None = None, logger: logging.Logger | None = None, **kwargs): super().__init__(align=align, area_budget=area_budget, logger=logger) self.checkpoint = checkpoint self.device = device self._load_model() ``` ### Step 2: Implement process_scene The `process_scene` method is the core entry point. It receives a `gt_artifact` (ground truth metadata) and must return a dictionary containing per-frame predictions. Use `BSSLoader` from [`benchmark/core/loader.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/core/loader.py) to load RGB sequences while respecting the resize context. ```python def process_scene(self, gt_artifact) -> Dict[str, Any]: from benchmark.core.loader import BSSLoader loader = BSSLoader(gt_artifact, resize_context=self.resize_context) rgb_list = loader.load_rgb_list() # Convert to tensors and run inference images = self._prepare_images(rgb_list) with torch.no_grad(): pred = self.model(images) # Format outputs to match BaseMethod contract return { "frame": { "rgb": rgb_list, "depth": [p["depth"].cpu().numpy() for p in pred], "pose": [p["pose"].cpu().numpy() for p in pred], "intrinsics": [p["intrinsics"].cpu().numpy() for p in pred], }, "global": {} } ``` If your model produces additional outputs (e.g., semantic maps), add them to the `frame` dictionary and implement optional helper methods named `__save__file__` to enable automatic serialization by the benchmark's saver. ### Step 3: Register in METHOD_REGISTRY Import your class in [`benchmark/benchmark/method/__init__.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/method/__init__.py) and add it to the dictionary: ```python # benchmark/benchmark/method/__init__.py from .lingbot_map import LingbotMapMethod from .my_new_method import MyNewMethod METHOD_REGISTRY = { "lingbot_map": LingbotMapMethod, "my_new_method": MyNewMethod, } ``` ## Complete Implementation Example Below is a complete, runnable template for a hypothetical `GCTModel` integration. This mirrors the structure used by `LingbotMapMethod` while demonstrating proper tensor handling and device management. ```python # benchmark/methods/my_new_method.py import logging import numpy as np import torch from typing import Any, Dict from benchmark.method.base import BaseMethod from benchmark.core.loader import BSSLoader class MyNewMethod(BaseMethod): """Adapter for the MyModel GCT reconstruction pipeline.""" def __init__( self, checkpoint: str | None = None, device: str = "cuda", align: int = 14, area_budget: int | None = None, logger: logging.Logger | None = None, **kwargs, ): super().__init__(align=align, area_budget=area_budget, logger=logger) self.checkpoint = checkpoint self.device = device self._load_model() def _load_model(self): from my_model.models.gct import GCTModel self.model = GCTModel() if self.checkpoint: ckpt = torch.load(self.checkpoint, map_location=self.device) self.model.load_state_dict(ckpt["model"], strict=False) self.model = self.model.to(self.device).eval() def _prepare_images(self, rgb_list): from torchvision import transforms as TF to_tensor = TF.ToTensor() return torch.stack([to_tensor(rgb) for rgb in rgb_list]).to(self.device) def process_scene(self, gt_artifact) -> Dict[str, Any]: loader = BSSLoader(gt_artifact, resize_context=self.resize_context) rgb_list = loader.load_rgb_list() self.logger.info( f"Processing {len(rgb_list)} frames (HxW={loader.get_processing_dimensions()})" ) images = self._prepare_images(rgb_list) with torch.no_grad(): pred = self.model(images) rgb_out = rgb_list depth_out = [p["depth"].cpu().numpy() for p in pred] pose_out = [p["pose"].cpu().numpy() for p in pred] intrinsics_out = [p["intrinsics"].cpu().numpy() for p in pred] return { "frame": { "rgb": rgb_out, "depth": depth_out, "pose": pose_out, "intrinsics": intrinsics_out, }, "global": {}, } ``` ## Running Your New Method Once registered, invoke your method through the benchmark driver: ```bash python benchmark/run.py --method my_new_method \ --checkpoint /path/to/checkpoint.pt \ --area_budget 8000000 \ --align 14 \ ``` The driver instantiates `MyNewMethod` via the `METHOD_REGISTRY` lookup and passes the `--checkpoint`, `--area_budget`, and `--align` arguments directly to your constructor. ## Summary - **Subclass `BaseMethod`** from [`benchmark/benchmark/method/base.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/method/base.py) to create a compatible adapter for your reconstruction model. - **Implement `process_scene`** to load data via `BSSLoader`, run inference, and return standardized dictionaries with `frame` and `global` keys. - **Register the class** in [`benchmark/benchmark/method/__init__.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/benchmark/method/__init__.py) by adding it to `METHOD_REGISTRY` to enable CLI discovery. - **Optional serialization helpers** named `__save__file__` enable automatic export of custom output types beyond the standard RGB, depth, and pose. ## Frequently Asked Questions ### What is the minimum required method to implement in a new subclass? You must implement `process_scene(self, gt_artifact) → Dict[str, Any]`. This method receives the ground truth artifact, processes the scene through your model, and returns a dictionary containing at minimum a `frame` key with per-frame predictions (RGB, depth, pose, intrinsics) and optionally a `global` key for scene-wide outputs. ### Where does the benchmark load ground truth RGB sequences? The `BSSLoader` class in [`benchmark/core/loader.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/core/loader.py) handles ground truth loading. Instantiate it with `BSSLoader(gt_artifact, resize_context=self.resize_context)` inside `process_scene` to respect the benchmark's resizing and alignment constraints, then call `loader.load_rgb_list()` to retrieve the frame sequence. ### Can I add custom output types beyond depth and pose? Yes. Add any additional keys to the `frame` dictionary returned by `process_scene`. To enable automatic file serialization, implement a helper method named `__save__file__` (e.g., `__save_semantic_map_file__`) following the pattern documented in the `BaseMethod` docstring. The benchmark's saver will detect and invoke these methods automatically. ### How do I specify which GPU device my method uses? Accept a `device` parameter in your `__init__` method (as shown in the `LingbotMapMethod` reference) and pass it when calling `super().__init__()`. Move your model to this device with `.to(self.device)` and ensure tensor operations in `process_scene` explicitly transfer data to `self.device` to avoid CPU/GPU mismatches.