# How Configuration Is Managed for LLM Training Jobs in train-llm-from-scratch

> Discover how LLM training jobs use a layered JSON config system with dataclass defaults, file merging, and CLI overrides. Learn configuration management in train-llm-from-scratch.

- Repository: [Fareed Khan/train-llm-from-scratch](https://github.com/FareedKhan-dev/train-llm-from-scratch)
- Tags: internals
- Published: 2026-06-11

---

**The repository implements a layered JSON-based configuration system that merges dataclass defaults, base JSON files, stage-specific JSON files, and CLI overrides using the custom [`config/loader.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/config/loader.py) module.**

Training large language models requires juggling hundreds of hyperparameters across multiple stages. The `FareedKhan-dev/train-llm-from-scratch` repository solves this with a robust, hierarchical configuration management system that resolves settings at runtime through a four-layer precedence stack.

## The Four-Layer Configuration Hierarchy

### Layer 1: Dataclass Defaults (Lowest Precedence)

The foundation of the system rests on Python dataclasses defined in [`config/post_training_config.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/config/post_training_config.py) (for post-training stages) and [`config/config.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/config/config.py) (for pre-training). These classes declare fields like vocabulary size, context length, and device settings, providing type-safe fallback values when no JSON or CLI values are specified.

### Layer 2: Base JSON Configuration

The [`configs/base.json`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/configs/base.json) file contains model-wide hyperparameters shared across all training stages, including SFT, PPO, and DPO. When a stage-specific JSON file resides in a subdirectory (e.g., [`configs/smoke/sft.json`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/configs/smoke/sft.json)), the loader automatically discovers and merges a sibling [`base.json`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/base.json) in that same directory ([`configs/smoke/base.json`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/configs/smoke/base.json)).

### Layer 3: Stage-Specific JSON Files

Individual training stages define their own configurations in files like [`configs/sft.json`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/configs/sft.json), [`configs/ppo.json`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/configs/ppo.json), and [`configs/reward.json`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/configs/reward.json). These files override any overlapping keys from the base configuration, allowing stage-specific tuning of learning rates, batch sizes, and optimization settings.

### Layer 4: CLI Overrides (Highest Precedence)

When invoking training scripts, users can pass `--field key=value` arguments. These command-line overrides are merged last in the `load_config()` function, giving the highest precedence and enabling ad-hoc experimentation without modifying JSON files.

## How the Configuration Loader Works

The resolution logic lives in [`config/loader.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/config/loader.py), which exposes the `load_config` function and a `_deep_merge` helper for nested dictionary merging. The loader sequentially applies the four layers, with later layers overwriting earlier ones.

The system gracefully handles unknown keys by printing warnings rather than raising exceptions, preventing crashes from typos or experimental parameters while alerting users to potential issues.

```python
from config.loader import load_config
from config.post_training_config import SFTConfig

# Resolve the full SFT configuration with all layers applied

cfg = load_config(
    SFTConfig,
    json_path="configs/sft.json",          # Stage-specific JSON

    overrides={"t_lr": 3e-4},              # CLI overrides (highest precedence)

)

```

## Running Training Jobs with Configuration Files

Training scripts in the `scripts/` directory consume these resolved configurations through the loader module.

Run a supervised fine-tuning job with default configuration:

```bash
python scripts/train_sft.py \
    --config configs/sft.json

```

Override a hyperparameter from the command line:

```bash
python scripts/train_sft.py \
    --config configs/sft.json \
    --field t_lr=3e-4

```

Use a custom configuration directory for smoke testing:

```bash
python scripts/train_sft.py \
    --config configs/smoke/sft.json

```

## Key Configuration Files

- **[`config/config.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/config/config.py)** – Global defaults for pre-training transformers (vocabulary size, device, etc.)
- **[`config/loader.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/config/loader.py)** – Core JSON loading and deep-merge logic (`_deep_merge`, `load_config`)
- **[`config/post_training_config.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/config/post_training_config.py)** – Dataclasses defining SFT, PPO, and DPO stage-specific fields
- **[`configs/base.json`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/configs/base.json)** – Shared hyperparameters across all post-training stages
- **[`configs/sft.json`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/configs/sft.json)** – Supervised fine-tuning specific hyperparameters
- **[`configs/ppo.json`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/configs/ppo.json)** – PPO-specific training settings
- **[`scripts/train_sft.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/scripts/train_sft.py)** – Entry point demonstrating configuration loading for SFT
- **[`scripts/train_ppo.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/scripts/train_ppo.py)** – PPO training entry point using the same loader pattern

## Summary

- The configuration system uses a four-layer precedence stack: dataclass defaults, base JSON, stage-specific JSON, and CLI overrides.
- The [`config/loader.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/config/loader.py) module handles resolution via the `_deep_merge` helper and `load_config` function.
- Automatic discovery of sibling [`base.json`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/base.json) files enables organized configuration sets in subdirectories.
- Unknown keys trigger warnings rather than crashes, providing a forgiving user experience for iterative development.

## Frequently Asked Questions

### What is the precedence order for configuration values?

Values cascade from lowest to highest: dataclass defaults < [`configs/base.json`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/configs/base.json) < stage-specific JSON < CLI overrides. The `load_config` function in [`config/loader.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/config/loader.py) merges these layers sequentially, with later layers overwriting earlier ones.

### How does the loader handle unknown configuration keys?

The [`config/loader.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/config/loader.py) module prints warnings for unrecognized keys but continues execution. This prevents crashes due to typos or experimental parameters while alerting users to potential configuration issues.

### Can I organize configurations into custom subdirectories?

Yes. When you place a stage JSON file in a subdirectory (e.g., [`configs/smoke/sft.json`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/configs/smoke/sft.json)), the loader automatically discovers and merges a sibling [`base.json`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/base.json) in that same directory ([`configs/smoke/base.json`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/configs/smoke/base.json)), allowing isolated configuration sets for different environments.

### Where are the training stage dataclasses defined?

Post-training stage dataclasses like `SFTConfig`, `PPOConfig`, and `DPOConfig` are defined in [`config/post_training_config.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/config/post_training_config.py). Pre-training defaults reside in [`config/config.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/config/config.py). These dataclasses provide the schema and default values for the configuration system.