# How to Configure Multi‑GPU Training with DeepSpeed or FSDP in Soup

> Learn how to configure multi-GPU training with DeepSpeed or FSDP in Soup. Effortlessly scale your models using simple flags for automatic config and GPU count adaptation.

- Repository: [Alpamys Makazhan/Soup](https://github.com/MakazhanAlpamys/Soup)
- Tags: how-to-guide
- Published: 2026-08-16

---

**Soup provides first‑class DeepSpeed and FSDP utilities that let you scale training across multiple GPUs using simple preset flags, with automatic config resolution and GPU‑count adaptation.**

The [Soup](https://github.com/MakazhanAlpamys/Soup) CLI framework offers two complementary strategies for distributed training: **Microsoft DeepSpeed** (with ZeRO‑2/3 sharding and off‑loading) and **PyTorch FSDP** (Fully‑Sharded Data Parallel). Both are implemented as helper modules that translate human‑readable presets into complete configuration objects, handling temporary files, GPU scaling, and validation automatically.

## DeepSpeed: ZeRO Sharding with Preset Configs

DeepSpeed integration lives in [[`src/soup_cli/utils/deepspeed.py`](https://github.com/MakazhanAlpamys/Soup/blob/main/src/soup_cli/utils/deepspeed.py)](https://github.com/MakazhanAlpamys/Soup/blob/main/src/soup_cli/utils/deepspeed.py). This module exposes three core functions:

- **`get_deepspeed_config(preset)`** — Returns a mutable dictionary containing the full DeepSpeed JSON schema for presets like `zero2`, `zero3`, or `zero3_offload`.
- **`resolve_deepspeed_config(cfg, gpu_count=n)`** — Adapts batch sizes and gradient accumulation steps to match your target GPU count.
- **`write_deepspeed_config(cfg)`** — Persists the config to a temporary `.json` file that the trainer consumes.

### Enabling DeepSpeed via CLI

Pass `--deepspeed <preset>` to the `soup train` command:

```bash
soup train \
  --model meta-llama/Llama-2-7b-chat-hf \
  --dataset my_dataset \
  --deepspeed zero3 \
  --gpus 4 \
  --output_dir ./checkpoints

```

Available presets include:

| Preset | Description |
|--------|-------------|
| `zero2` | ZeRO‑2 (optimizer state sharding) |
| `zero3` | ZeRO‑3 (parameter + gradient + optimizer sharding) |
| `zero3_offload` | ZeRO‑3 with optimizer state off‑loaded to CPU/NVMe |

### Programmatic DeepSpeed Configuration

Customize configs in Python before writing to disk:

```python
from soup_cli.utils.deepspeed import (
    get_deepspeed_config,
    resolve_deepspeed_config,
    write_deepspeed_config,
)

# Load preset configuration

cfg = get_deepspeed_config("zero3_offload")

# Adapt for 2 GPUs (adjusts micro_batch_size and grad_accum)

cfg, notes = resolve_deepspeed_config(cfg, gpu_count=2)

# Write temporary JSON for trainer consumption

config_path = write_deepspeed_config(cfg)
print(f"DeepSpeed config: {config_path}")

```

## FSDP: PyTorch Native Sharding

For cases where you prefer PyTorch's built‑in distributed strategy, Soup provides [[`src/soup_cli/utils/fsdp.py`](https://github.com/MakazhanAlpamys/Soup/blob/main/src/soup_cli/utils/fsdp.py)](https://github.com/MakazhanAlpamys/Soup/blob/main/src/soup_cli/utils/fsdp.py). The primary entry point is:

- **`get_fsdp_config(preset)`** — Returns a configuration dict with the `"fsdp"` key containing `sharding_strategy`, `auto_wrap_policy`, and `offload_params` settings.

### Enabling FSDP via CLI

Use the `--fsdp` flag with a preset name:

```bash
soup train \
  --model meta-llama/Llama-2-7b-chat-hf \
  --dataset my_dataset \
  --fsdp full_offload \
  --gpus 4 \
  --output_dir ./checkpoints

```

FSDP presets map directly to PyTorch sharding strategies:

| Preset | PyTorch Equivalent | Use Case |
|--------|-------------------|----------|
| `full_shard` | `FULL_SHARD` | Maximum memory efficiency |
| `shard_grad` | `SHARD_GRAD_OP` | Lower communication overhead |
| `full_offload` | `FULL_SHARD` + CPU off‑load | Extreme memory constraints |

### Programmatic FSDP Configuration

```python
from soup_cli.utils.fsdp import get_fsdp_config

fsdp_cfg = get_fsdp_config("shard_grad")
print(fsdp_cfg["fsdp"]["sharding_strategy"])

# Output: "SHARD_GRAD_OP"

```

## Choosing Between DeepSpeed and FSDP

Both strategies are **mutually exclusive**. Soup validates CLI arguments and raises a clear error if you attempt to enable both simultaneously. This validation is enforced in [[`tests/test_multi_gpu.py`](https://github.com/MakazhanAlpamys/Soup/blob/main/tests/test_multi_gpu.py)](https://github.com/MakazhanAlpamys/Soup/blob/main/tests/test_multi_gpu.py) and [[`tests/test_performance.py`](https://github.com/MakazhanAlpamys/Soup/blob/main/tests/test_performance.py)](https://github.com/MakazhanAlpamys/Soup/blob/main/tests/test_performance.py).

**DeepSpeed** excels when you need:
- CPU/NVMe off‑loading for massive models
- ZeRO‑Infinity for training models larger than aggregate GPU memory
- Built‑in DeepSpeed‑specific optimizers (AdamW, OneBitAdam)

**FSDP** is preferable when you want:
- Native PyTorch integration without additional dependencies
- Simpler debugging with standard `torch.distributed` tooling
- Faster setup for medium‑scale models that fit with standard sharding

## Source Files and Architecture

| File | Purpose |
|------|---------|
| [[`src/soup_cli/utils/deepspeed.py`](https://github.com/MakazhanAlpamys/Soup/blob/main/src/soup_cli/utils/deepspeed.py)](https://github.com/MakazhanAlpamys/Soup/blob/main/src/soup_cli/utils/deepspeed.py) | DeepSpeed config loading, GPU adaptation, and JSON serialization |
| [[`src/soup_cli/utils/fsdp.py`](https://github.com/MakazhanAlpamys/Soup/blob/main/src/soup_cli/utils/fsdp.py)](https://github.com/MakazhanAlpamys/Soup/blob/main/src/soup_cli/utils/fsdp.py) | FSDP preset definitions and config generation |
| [[`src/soup_cli/commands/train.py`](https://github.com/MakazhanAlpamys/Soup/blob/main/src/soup_cli/commands/train.py)](https://github.com/MakazhanAlpamys/Soup/blob/main/src/soup_cli/commands/train.py) | CLI argument parsing and trainer wiring |
| [[`tests/test_multi_gpu.py`](https://github.com/MakazhanAlpamys/Soup/blob/main/tests/test_multi_gpu.py)](https://github.com/MakazhanAlpamys/Soup/blob/main/tests/test_multi_gpu.py) | DeepSpeed config resolution and conflict detection tests |
| [[`tests/test_performance.py`](https://github.com/MakazhanAlpamys/Soup/blob/main/tests/test_performance.py)](https://github.com/MakazhanAlpamys/Soup/blob/main/tests/test_performance.py) | FSDP preset validation and structure verification |
| [[`tests/test_issue336_deepspeed_lora.py`](https://github.com/MakazhanAlpamys/Soup/blob/main/tests/test_issue336_deepspeed_lora.py)](https://github.com/MakazhanAlpamys/Soup/blob/main/tests/test_issue336_deepspeed_lora.py) | GPU‑count adaptation logic for DeepSpeed + LoRA combinations |

## Summary

- **Soup enables multi‑GPU training** through two integrated strategies: DeepSpeed (ZeRO‑2/3 with off‑loading) and PyTorch FSDP (native sharding).
- **Preset flags** (`--deepspeed zero3`, `--fsdp full_shard`) automatically generate complete configurations without manual JSON editing.
- **Programmatic APIs** in [[`deepspeed.py`](https://github.com/MakazhanAlpamys/Soup/blob/main/deepspeed.py)](https://github.com/MakazhanAlpamys/Soup/blob/main/src/soup_cli/utils/deepspeed.py) and [[`fsdp.py`](https://github.com/MakazhanAlpamys/Soup/blob/main/fsdp.py)](https://github.com/MakazhanAlpamys/Soup/blob/main/src/soup_cli/utils/fsdp.py) allow custom modifications while preserving validation and temporary file handling.
- **GPU scaling** is handled automatically via `resolve_deepspeed_config()` for DeepSpeed, ensuring batch settings match your hardware.

## Frequently Asked Questions

### What happens if I specify both `--deepspeed` and `--fsdp`?

Soup raises a validation error before training starts. The CLI explicitly checks for conflicting flags in the argument parser, as verified by [[`tests/test_multi_gpu.py`](https://github.com/MakazhanAlpamys/Soup/blob/main/tests/test_multi_gpu.py)](https://github.com/MakazhanAlpamys/Soup/blob/main/tests/test_multi_gpu.py). You must choose one distributed strategy per run.

### Does Soup support DeepSpeed ZeRO‑Infinity or NVMe off‑loading?

Yes. The `zero3_offload` preset in [[`deepspeed.py`](https://github.com/MakazhanAlpamys/Soup/blob/main/deepspeed.py)](https://github.com/MakazhanAlpamys/Soup/blob/main/src/soup_cli/utils/deepspeed.py) includes configuration keys for CPU and NVMe off‑loading. Extend the preset or use `get_deepspeed_config()` as a base for custom ZeRO‑Infinity configurations.

### Can I use a custom DeepSpeed JSON file instead of presets?

Absolutely. Pass `--deepspeed-config /path/to/config.json` directly to `soup train`. The CLI skips preset resolution and uses your file verbatim. The [[`deepspeed.py`](https://github.com/MakazhanAlpamys/Soup/blob/main/deepspeed.py)](https://github.com/MakazhanAlpamys/Soup/blob/main/src/soup_cli/utils/deepspeed.py) helpers remain available for inspection or modification if needed.

### How does Soup determine batch sizes for different GPU counts?

The `resolve_deepspeed_config(cfg, gpu_count=n)` function in [[`deepspeed.py`](https://github.com/MakazhanAlpamys/Soup/blob/main/deepspeed.py)](https://github.com/MakazhanAlpamys/Soup/blob/main/src/soup_cli/utils/deepspeed.py) recalculates `train_micro_batch_size_per_gpu` and `gradient_accumulation_steps`—`

 `` ` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` `` bash ` ` ` ` ` ` ` ` ` ` ` ` ` ` ``