How to Configure AFAB vs 1F1B Pipeline Schedules in Nanotron

Set pipeline_parallel.schedule to "afab" for All‑Forward‑All‑Backward execution or "1f1b" for Interleaved‑Forward‑Interleaved‑Backward via YAML configs, Python APIs, or CLI overrides in Hugging Face Nanotron.

Nanotron is Hugging Face's lightweight framework for large‑scale transformer pretraining, offering configurable pipeline parallelism to optimize distributed training. Selecting the appropriate pipeline schedule directly impacts GPU memory consumption and computational throughput when scaling across multiple devices. This guide explains how to configure AFAB vs 1F1B pipeline schedules in Nanotron using the PipelineParallelConfig dataclass and runtime interfaces.

Understanding AFAB and 1F1B Pipeline Schedules

AFAB (All‑Forward‑All‑Backward)

The AFAB schedule executes the forward pass for every micro‑batch across all pipeline stages before initiating any backward passes. In nanotron/config/config.py, this strategy is activated by setting schedule="afab" within the PipelineParallelConfig dataclass. While this approach simplifies reasoning about execution order and aids debugging, it requires storing the full set of activations for the entire pipeline depth simultaneously, increasing peak memory usage.

1F1B (Interleaved‑Forward‑Interleaved‑Backward)

The 1F1B schedule (also documented as IFIB) interleaves forward and backward operations, immediately processing the backward pass for a micro‑batch once its forward pass completes. Configure this by setting schedule="1f1b" in the same configuration object. This pipelining strategy minimizes the activation memory footprint by releasing tensors as soon as they are consumed by the gradient computation, making it ideal for training large models on memory‑constrained hardware.

Configuration Methods

YAML Configuration

Define the schedule within the pipeline_parallel block of your training configuration file:

pipeline_parallel:
  schedule: "1f1b"  # Use "afab" for All‑Forward‑All‑Backward

  # Additional pipeline parallel settings...

Switching strategies requires only changing the string value between "afab" and "1f1b".

Programmatic Configuration

When constructing training configurations dynamically, modify the NanotronConfig object before instantiating the Trainer:

from nanotron.config import NanotronConfig

cfg = NanotronConfig.load_from_yaml("examples/config_tiny_llama.yaml")
cfg.pipeline_parallel.schedule = "afab"  # Options: "afab" or "1f1b"

trainer = Trainer(cfg, model, optimizer)
trainer.fit()

Command‑Line Override

Override the YAML configuration at runtime using the --pipeline_parallel_schedule flag when executing run_train.py:

python run_train.py \
  --config examples/config_tiny_llama.yaml \
  --pipeline_parallel_schedule 1f1b

Performance Considerations

Memory Requirements: AFAB stores all activations for the complete pipeline, typically requiring ≥2× the per‑stage model size in GPU memory. 1F1B maintains a constant memory footprint regardless of pipeline depth by immediately freeing activation buffers after backward computation.

Throughput Optimization: 1F1B generally achieves higher throughput on memory‑constrained systems because it reduces memory pressure, allowing increased micro‑batch sizes or deeper pipelines without out‑of‑memory errors.

Debugging Workflow: AFAB provides deterministic execution order (all forwards followed by all backwards), making it preferable for initial prototyping and numerical validation. Transition to 1F1B for production‑scale training runs where memory efficiency is critical.

Summary

• AFAB (schedule: "afab"): Executes all forward passes before any backward passes; higher memory usage but deterministic and easier to debug.
• 1F1B (schedule: "1f1b"): Interleaves forward and backward passes per micro‑batch; minimizes activation memory and scales to larger models.
• Configuration is controlled via pipeline_parallel.schedule in YAML files, cfg.pipeline_parallel.schedule programmatically, or the --pipeline_parallel_schedule CLI argument.

Frequently Asked Questions

What is the difference between AFAB and 1F1B in Nanotron?

AFAB runs all forward micro‑batches through the pipeline first, then executes all backward passes, keeping every activation in memory simultaneously. 1F1B immediately follows each forward pass with its corresponding backward pass, reducing peak memory usage by freeing activations as soon as gradients are computed.

Where is the pipeline schedule defined in the Nanotron source code?

The schedule parameter is defined in nanotron/config/config.py within the PipelineParallelConfig dataclass as the schedule field, which accepts the string literals "afab" or "1f1b" according to the repository's pipeline parallel implementation.

Can I switch between AFAB and 1F1B without modifying my model code?

Yes. The pipeline schedule is purely a runtime configuration parameter. You can switch strategies by changing the schedule value in your YAML configuration, passing the --pipeline_parallel_schedule CLI flag, or modifying the config object programmatically without altering model definitions or training loop logic.

Which schedule should I use for large‑scale training?

Use 1F1B for large models or when GPU memory is limited, as it maintains constant activation memory regardless of pipeline depth. Use AFAB when debugging distributed training issues or when you have abundant GPU memory and prefer the simplicity of deterministic execution ordering.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}