How to Add a New Method to the LingBot-Map Benchmark Evaluation Framework

To add a new method to the LingBot-Map benchmark, subclass BaseMethod from benchmark/benchmark/method/base.py, implement the process_scene method, and register your class in the METHOD_REGISTRY inside benchmark/benchmark/method/__init__.py.

The LingBot-Map repository provides a modular evaluation framework for SLAM and 3D reconstruction algorithms. Adding a new method requires understanding the three-layer architecture—base interface, concrete implementations, and method registry—that governs how the benchmark driver instantiates and executes evaluation pipelines. This guide walks through the exact steps to integrate custom models while maintaining compatibility with the existing BSSLoader and result serialization systems.

Understanding the Method Architecture

The framework separates concerns across three distinct layers. This design allows you to plug in arbitrary reconstruction models without modifying the core evaluation logic.

Base Interface

The abstract contract every method must satisfy is defined in benchmark/benchmark/method/base.py. The BaseMethod class establishes the API surface: subclasses receive a resize_context, alignment parameters, and a logger during initialization, and must implement process_scene(self, gt_artifact) → Dict[str, Any]. The returned dictionary must follow a strict schema with frame and global keys to ensure downstream evaluators can parse the results consistently.

Method Implementations

Concrete adapters reside in the benchmark/methods/ directory. The reference implementation, LingbotMapMethod in benchmark/methods/lingbot_map.py, demonstrates how to wrap an external SLAM package. These classes handle model-specific logic such as checkpoint loading, tensor conversion, and inference device management, while translating proprietary outputs into the standardized format required by BaseMethod.

Registry and Factory

The benchmark driver discovers methods through a centralized registry in benchmark/benchmark/method/__init__.py. This file maintains a METHOD_REGISTRY dictionary that maps CLI-friendly strings (e.g., "lingbot_map") to class constructors. When you invoke python benchmark/run.py --method <name>, the driver looks up the name in this registry and instantiates the corresponding class with remaining command-line arguments.

Implementing Your Custom Method

Adding a new evaluation method involves three concrete steps: creating the subclass, implementing the processing logic, and registering the component.

Step 1: Subclass BaseMethod

Create a new file in benchmark/methods/ (e.g., my_new_method.py). Inherit from BaseMethod and call super().__init__(align=align, area_budget=area_budget, logger=logger) to initialize the resize context and logging infrastructure. Define your constructor to accept model-specific parameters such as checkpoint paths and device strings.

# benchmark/methods/my_new_method.py

import logging
import torch
from typing import Any, Dict
from benchmark.method.base import BaseMethod

class MyNewMethod(BaseMethod):
    """Adapter for custom reconstruction pipeline."""
    
    def __init__(self, checkpoint: str | None = None, device: str = "cuda", 
                 align: int = 14, area_budget: int | None = None,
                 logger: logging.Logger | None = None, **kwargs):
        super().__init__(align=align, area_budget=area_budget, logger=logger)
        self.checkpoint = checkpoint
        self.device = device
        self._load_model()

Step 2: Implement process_scene

The process_scene method is the core entry point. It receives a gt_artifact (ground truth metadata) and must return a dictionary containing per-frame predictions. Use BSSLoader from benchmark/core/loader.py to load RGB sequences while respecting the resize context.

    def process_scene(self, gt_artifact) -> Dict[str, Any]:
        from benchmark.core.loader import BSSLoader
        
        loader = BSSLoader(gt_artifact, resize_context=self.resize_context)
        rgb_list = loader.load_rgb_list()
        
        # Convert to tensors and run inference

        images = self._prepare_images(rgb_list)
        with torch.no_grad():
            pred = self.model(images)
        
        # Format outputs to match BaseMethod contract

        return {
            "frame": {
                "rgb": rgb_list,
                "depth": [p["depth"].cpu().numpy() for p in pred],
                "pose": [p["pose"].cpu().numpy() for p in pred],
                "intrinsics": [p["intrinsics"].cpu().numpy() for p in pred],
            },
            "global": {}
        }

If your model produces additional outputs (e.g., semantic maps), add them to the frame dictionary and implement optional helper methods named __save_<key>_file__ to enable automatic serialization by the benchmark's saver.

Step 3: Register in METHOD_REGISTRY

Import your class in benchmark/benchmark/method/__init__.py and add it to the dictionary:

# benchmark/benchmark/method/__init__.py

from .lingbot_map import LingbotMapMethod
from .my_new_method import MyNewMethod

METHOD_REGISTRY = {
    "lingbot_map": LingbotMapMethod,
    "my_new_method": MyNewMethod,
}

Complete Implementation Example

Below is a complete, runnable template for a hypothetical GCTModel integration. This mirrors the structure used by LingbotMapMethod while demonstrating proper tensor handling and device management.

# benchmark/methods/my_new_method.py

import logging
import numpy as np
import torch
from typing import Any, Dict

from benchmark.method.base import BaseMethod
from benchmark.core.loader import BSSLoader


class MyNewMethod(BaseMethod):
    """Adapter for the MyModel GCT reconstruction pipeline."""

    def __init__(
        self,
        checkpoint: str | None = None,
        device: str = "cuda",
        align: int = 14,
        area_budget: int | None = None,
        logger: logging.Logger | None = None,
        **kwargs,
    ):
        super().__init__(align=align, area_budget=area_budget, logger=logger)
        self.checkpoint = checkpoint
        self.device = device
        self._load_model()

    def _load_model(self):
        from my_model.models.gct import GCTModel
        
        self.model = GCTModel()
        if self.checkpoint:
            ckpt = torch.load(self.checkpoint, map_location=self.device)
            self.model.load_state_dict(ckpt["model"], strict=False)
        self.model = self.model.to(self.device).eval()

    def _prepare_images(self, rgb_list):
        from torchvision import transforms as TF
        
        to_tensor = TF.ToTensor()
        return torch.stack([to_tensor(rgb) for rgb in rgb_list]).to(self.device)

    def process_scene(self, gt_artifact) -> Dict[str, Any]:
        loader = BSSLoader(gt_artifact, resize_context=self.resize_context)
        rgb_list = loader.load_rgb_list()
        self.logger.info(
            f"Processing {len(rgb_list)} frames (HxW={loader.get_processing_dimensions()})"
        )

        images = self._prepare_images(rgb_list)
        with torch.no_grad():
            pred = self.model(images)

        rgb_out = rgb_list
        depth_out = [p["depth"].cpu().numpy() for p in pred]
        pose_out = [p["pose"].cpu().numpy() for p in pred]
        intrinsics_out = [p["intrinsics"].cpu().numpy() for p in pred]

        return {
            "frame": {
                "rgb": rgb_out,
                "depth": depth_out,
                "pose": pose_out,
                "intrinsics": intrinsics_out,
            },
            "global": {},
        }

Running Your New Method

Once registered, invoke your method through the benchmark driver:

python benchmark/run.py --method my_new_method \
    --checkpoint /path/to/checkpoint.pt \
    --area_budget 8000000 \
    --align 14 \
    <other benchmark flags>

The driver instantiates MyNewMethod via the METHOD_REGISTRY lookup and passes the --checkpoint, --area_budget, and --align arguments directly to your constructor.

Summary

• Subclass BaseMethod from benchmark/benchmark/method/base.py to create a compatible adapter for your reconstruction model.
• Implement process_scene to load data via BSSLoader, run inference, and return standardized dictionaries with frame and global keys.
• Register the class in benchmark/benchmark/method/__init__.py by adding it to METHOD_REGISTRY to enable CLI discovery.
• Optional serialization helpers named __save_<key>_file__ enable automatic export of custom output types beyond the standard RGB, depth, and pose.

Frequently Asked Questions

What is the minimum required method to implement in a new subclass?

You must implement process_scene(self, gt_artifact) → Dict[str, Any]. This method receives the ground truth artifact, processes the scene through your model, and returns a dictionary containing at minimum a frame key with per-frame predictions (RGB, depth, pose, intrinsics) and optionally a global key for scene-wide outputs.

Where does the benchmark load ground truth RGB sequences?

The BSSLoader class in benchmark/core/loader.py handles ground truth loading. Instantiate it with BSSLoader(gt_artifact, resize_context=self.resize_context) inside process_scene to respect the benchmark's resizing and alignment constraints, then call loader.load_rgb_list() to retrieve the frame sequence.

Can I add custom output types beyond depth and pose?

Yes. Add any additional keys to the frame dictionary returned by process_scene. To enable automatic file serialization, implement a helper method named __save_<key>_file__ (e.g., __save_semantic_map_file__) following the pattern documented in the BaseMethod docstring. The benchmark's saver will detect and invoke these methods automatically.

How do I specify which GPU device my method uses?

Accept a device parameter in your __init__ method (as shown in the LingbotMapMethod reference) and pass it when calling super().__init__(). Move your model to this device with .to(self.device) and ensure tensor operations in process_scene explicitly transfer data to self.device to avoid CPU/GPU mismatches.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}