How the Three-Layer YAML Configuration System Works in Lingbot-Map

The three-layer YAML configuration system in Robbyant/lingbot-map merges a base configuration with dataset-specific and method-specific overrides using a deep dictionary merge, allowing global defaults to cascade through modular, auto-discovered configuration files.

The lingbot-map repository implements a hierarchical configuration architecture to manage complex benchmarking experiments without duplication. This three-layer YAML configuration system separates global defaults from specific dataset parameters and method-specific settings, enabling researchers to compose experiments by selecting combinations of pre-defined components. The system centers on the ConfigManager class in benchmark/core/config.py, which handles auto-discovery, loading, and merging of configuration layers.

The Three Configuration Layers

The architecture divides settings across three distinct levels, each serving a specific purpose in the experiment lifecycle.

Base Configuration Layer

The foundation resides in benchmark/configs/base.yaml. This file establishes global defaults for workspace paths, logging levels, and evaluation criteria. It also contains the master selection lists under datasets and methods keys, determining which configurations the system discovers and runs during the benchmark phase.

Dataset Configuration Layer

Individual dataset definitions live in benchmark/configs/datasets/*.yaml. Each file is flat (no top-level key) and named after the dataset it describes—for example, oxford.yaml or tum.yaml. These files contain dataset-specific parameters such as data paths, sampling strategies, and optional evaluation blocks that override base defaults. The filename stem serves as the configuration identifier throughout the system.

Method Configuration Layer

Method-specific settings are stored in benchmark/configs/methods/*.yaml, following the same flat-file structure as datasets. Files like lingbot_map.yaml define algorithm parameters, optional conda environment specifications via the env field, and further evaluation overrides. This layer receives the highest precedence in the merge hierarchy, allowing methods to fine-tune evaluation parameters without modifying global or dataset configurations.

How ConfigManager Merges the Layers

The ConfigManager class orchestrates the loading and merging process in benchmark/benchmark/core/config.py. During initialization (lines 84-99), it loads the base configuration and auto-discovers all YAML files in the datasets and methods directories.

Loading occurs via _load_yaml (lines 75-78), which uses yaml.safe_load to parse individual files:

from pathlib import Path
from benchmark.core.config import ConfigManager

# Initialize manager with base config

cfg = ConfigManager(Path("benchmark/configs/base.yaml"))

# Access selected components

datasets = cfg.get_selected_dataset_names()   # ["oxford", "tum", ...]

methods = cfg.get_selected_method_names()     # ["lingbot_map", ...]

When computing effective configurations, the system applies _deep_merge (lines 19-35) through the public method get_merged_evaluation_config (lines 30-39). This implements a deep dictionary merge: nested dictionaries are combined recursively, with later values overwriting earlier ones in the sequence base → dataset → method.

# Retrieve merged evaluation configuration

eval_cfg = cfg.get_merged_evaluation_config(
    dataset_config_name="oxford",
    method_config_name="lingbot_map"
)

The resulting eval_cfg contains base evaluation parameters, overridden by any evaluation block in oxford.yaml, then further overridden by any evaluation block in lingbot_map.yaml.

Configuration Hierarchy and Precedence

The precedence rules follow a simple but strict hierarchy:

1. Base values provide the fallback defaults
2. Dataset configurations override base values for specific data handling
3. Method configurations override both previous layers for algorithm-specific needs

This deep merge behavior ensures that partial overrides are possible. A dataset can change only the evaluation metrics it affects, while a method can adjust logging or output directories without redeclaring unrelated base parameters. The env field in method configs remains separate from the merge and is accessed directly via get_method_config().

Code Examples in Practice

The benchmark pipeline in benchmark/run.py demonstrates practical consumption of the three-layer system. The run phase retrieves method configurations and merged evaluation settings for each dataset-method combination:

# Inside benchmark/run.py workflow

method_cfg = config_manager.get_method_config(method_name)
merged_eval = config_manager.get_method_config(method_name)

# Direct access to specific layer configurations

dataset_params = cfg.get_dataset_config("oxford")["params"]
method_env = cfg.get_method_config("lingbot_map").get("env")  # conda environment name

Each configuration retrieval method returns a dictionary containing:

• Dataset configs: dataset_class, params, sampling, optional evaluation
• Method configs: method_class, params, env, optional evaluation

Summary

• The three-layer YAML configuration system separates concerns across global defaults, dataset specifics, and method specifics
• ConfigManager in benchmark/core/config.py auto-discovers configurations and performs deep merges in base → dataset → method order
• Dataset files in benchmark/configs/datasets/*.yaml override base settings without duplication
• Method files in benchmark/configs/methods/*.yaml receive final precedence and can specify conda environments via the env field
• The merge algorithm is recursive, ensuring nested dictionaries combine properly rather than replacing each other wholesale

Frequently Asked Questions

What happens if the same key exists in multiple configuration layers?

The deep merge algorithm resolves conflicts by giving precedence to later layers. Values from base.yaml are overwritten by dataset configurations, which are then overwritten by method configurations. For nested dictionaries, the merge is recursive, meaning only conflicting leaf values are replaced, preserving the structure of non-conflicting siblings.

How does the system discover available dataset and method configurations?

ConfigManager auto-discovers configurations by scanning the directories specified in the base config. During initialization (lines 86-99 in benchmark/core/config.py), it iterates through benchmark/configs/datasets/ and benchmark/configs/methods/, loading every YAML file found and using the filename stem as the configuration identifier.

Can I specify different conda environments for different methods?

Yes. Each method configuration file can include an optional env field specifying the conda environment name. This field is accessed directly through get_method_config() and is not part of the merged evaluation configuration, allowing different algorithms to run in isolated environments without affecting the evaluation logic.

Where is the merged configuration actually used in the benchmark pipeline?

The merged evaluation configuration is consumed in benchmark/run.py, where the pipeline iterates through selected dataset-method combinations. For each pair, it calls get_merged_evaluation_config() to produce the final parameter set passed to the evaluation routines, ensuring each experiment runs with the correct combination of dataset-specific and method-specific parameters.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}