How to Manage LingBot-Map Configurations Using YAML Files: A Complete Guide

LingBot-Map configurations are managed through a hierarchical YAML file system coordinated by the ConfigManager class, which auto-discovers dataset and method configs from flat files and deep-merges them with base settings at runtime.

The Robbyant/lingbot-map repository implements a lightweight, file-based configuration system built on YAML that enables flexible benchmark management without code changes. This architecture separates global defaults from dataset-specific and method-specific settings, allowing researchers to add new experiments by simply dropping new YAML files into designated directories. Understanding how LingBot-Map configurations are structured and parsed is essential for customizing evaluation pipelines and extending the framework.

Configuration File Hierarchy in LingBot-Map

LingBot-Map uses a three-tier configuration system where settings cascade from global defaults to specific implementation details. All files are flat dictionaries parsed with yaml.safe_load, requiring no top-level keys.

Base Configuration (configs/base.yaml)

The foundation resides in benchmark/configs/base.yaml. This file defines the workspace path, global evaluation defaults, and optional selection lists for datasets and methods. The ConfigManager loads this file first to establish the execution environment and determine which dataset and method configurations to activate.

Dataset Configurations (configs/datasets/*.yaml)

Dataset configs are flat files stored in configs/datasets/ where the filename stem serves as the config name. Each YAML specifies the dataset class, optional sampling parameters, and an evaluation block that can override global defaults. The system auto-discovers every *.yaml in this directory, parsing only those named in the base configuration's selection list.

Method Configurations (configs/methods/*.yaml)

Method configurations follow the same flat-file pattern in configs/methods/. These files define the model class, optional conda environment via the env key, and method-specific evaluation overrides. Like datasets, these are auto-discovered and merged based on the base configuration selections.

The ConfigManager Implementation

The ConfigManager class in benchmark/benchmark/core/config.py orchestrates the loading and merging process. It provides a centralized interface for accessing configuration data while abstracting the file system operations. This class is typically instantiated in benchmark/run_worker.py, which serves as the entry point for driving the benchmark pipeline.

Auto-Discovery Mechanism

When initialized with a path to base.yaml, the ConfigManager automatically scans configs/datasets/ and configs/methods/ for YAML files. It matches discovered filenames against the datasets and methods lists defined in the base configuration, loading only the selected configurations into memory.

Deep Merge Strategy

The private _deep_merge function implements a recursive dictionary merge that preserves nested structures while allowing later values to override earlier ones. The merge order follows: base.yaml → dataset.evaluation → method.evaluation. This guarantees that method-specific settings take precedence over dataset defaults, which in turn override global settings.

Runtime Access to Merged Configurations

After construction, the ConfigManager exposes several methods for retrieving configuration data:

from pathlib import Path
from lingbot_map.benchmark.core.config import ConfigManager

# Initialize with the central base config

cfg = ConfigManager(Path("benchmark/configs/base.yaml"))

# Retrieve workspace path

workspace = cfg.get_workspace()

# List activated datasets and methods

datasets = cfg.get_selected_dataset_names()
methods = cfg.get_selected_method_names()

# Access specific configurations

oxford_cfg = cfg.get_dataset_config("oxford")
lm_cfg = cfg.get_method_config("lingbot_map")

# Get merged evaluation config for a specific pair

eval_cfg = cfg.get_merged_evaluation_config(
    dataset_config_name="oxford",
    method_config_name="lingbot_map"
)

The get_merged_evaluation_config() method returns the final, deep-merged configuration containing all applicable settings for a specific dataset-method combination.

Summary

• LingBot-Map configurations reside in three hierarchical tiers: base, dataset, and method YAML files located in benchmark/configs/.
• The ConfigManager class in benchmark/benchmark/core/config.py auto-discovers configurations by scanning configs/datasets/ and configs/methods/ directories.
• Configuration inheritance follows a deep-merge strategy where base.yaml provides defaults, overridden by dataset-specific evaluation blocks, then method-specific settings.
• The system uses flat YAML files without top-level keys, enabling new experiments through simple file additions rather than code modifications.
• Runtime access is provided through methods like get_workspace(), get_dataset_config(), and get_merged_evaluation_config().

Frequently Asked Questions

Where is the ConfigManager class implemented in the LingBot-Map repository?

The ConfigManager class is implemented in benchmark/benchmark/core/config.py. This file contains the core logic for YAML parsing, auto-discovery of dataset and method configurations, and the _deep_merge helper function that handles recursive dictionary merging.

How does LingBot-Map handle configuration overrides between base, dataset, and method settings?

The system uses a deep-merge strategy implemented in the _deep_merge function. When get_merged_evaluation_config() is called, it recursively merges dictionaries in the order: base defaults → dataset evaluation settings → method evaluation settings. Later values replace earlier ones while preserving nested structure, ensuring method-specific parameters take final precedence.

Can I add new datasets or methods without modifying the source code?

Yes. The configuration system is designed for zero-code modifications. Simply create a new YAML file in configs/datasets/ or configs/methods/ using the flat dictionary structure (no top-level key), then add the filename stem to the datasets or methods list in configs/base.yaml. The ConfigManager will auto-discover and load the new configuration on the next run.

What is the difference between get_dataset_config() and get_merged_evaluation_config()?

The get_dataset_config() method returns the raw configuration dictionary for a specific dataset, including its class definition, parameters, and evaluation overrides. In contrast, get_merged_evaluation_config() returns the final, deep-merged evaluation configuration for a specific dataset-method pair, combining global defaults with both dataset-specific and method-specific evaluation settings.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}