Stable Diffusion Fine-Tuning: Complete Guide to Training Configuration Options in main.py

Stable Diffusion's main.py exposes all fine-tuning hyperparameters through a hybrid command-line and YAML configuration system, combining custom arguments in get_parser() with PyTorch Lightning's full trainer interface to control learning rate scaling, checkpoint resumption, GPU allocation, and experiment logging.

The main.py script serves as the central orchestration point for fine-tuning Stable Diffusion models in the CompVis/stable-diffusion repository. It stitches together the model architecture, data module, logger, callbacks, and the PyTorch Lightning trainer through a sophisticated configuration pipeline that merges command-line arguments with YAML base configs.

Command-Line Arguments in get_parser()

The get_parser() function in main.py defines the custom argument parser that handles experiment-specific controls before Lightning trainer arguments are injected.

Run Control and Experiment Naming

The script provides granular control over experiment identification and directory structure:

• -n / --name (str, default: ""): Appends a post-fix to the automatic log directory name at main.py:L35-L44, enabling distinction between multiple experimental runs.
• -p / --project (str): Specifies the project directory name passed directly to the logger configuration at main.py:L80-L84.
• -l / --logdir (str, default: "logs"): Sets the base directory where all experiment folders are created, as implemented at main.py:L108-L114.
• -f / --postfix (str, default: ""): Adds an additional postfix to the automatically generated log-dir name beyond the --name parameter at main.py:L100-L106.

Resume and Checkpoint Management

Fine-tuning workflows depend heavily on checkpoint resumption capabilities:

• -r / --resume (str, default: ""): Accepts a path to a previous log folder or specific checkpoint file to resume training from, parsed at main.py:L46-L53. The script automatically detects whether the path points to a directory or a .ckpt file.
• -t / --train (bool, default: False): Acts as the training gatekeeper; only when set to true does the script invoke trainer.fit() at main.py:L64-L71.
• --no-test (bool, default: False): When enabled, skips the final test run after training completion, defined at main.py:L73-L78.

Debugging and Reproducibility

For development and deterministic experimentation:

• -d / --debug (bool, default: False): Enables post-mortem debugging using pudb on uncaught exceptions at main.py:L86-L92.
• -s / --seed (int, default: 23): Sets the global random seed via pl.seed_everything() at main.py:L94-L99, ensuring reproducible fine-tuning runs.

Learning Rate Scaling

• --scale_lr (bool, default: True): Controls automatic learning rate scaling based on hardware configuration at main.py:L115-L122. When enabled, the base learning rate is multiplied by accumulate_grad_batches × num_gpus × batch_size.

PyTorch Lightning Trainer Integration

After the custom parser is constructed, pytorch_lightning.Trainer.add_argparse_args(parser) is called at main.py:L66-L68 to inject the complete suite of Lightning trainer flags. This exposes critical distributed training and optimization parameters including:

• --gpus: GPU device indices or count for training
• --max_epochs: Maximum number of training epochs
• --accelerator: Backend selection (e.g., "gpu", "cpu", "tpu")
• --precision: Numerical precision (16, 32, or 64-bit)
• --accumulate_grad_batches: Gradient accumulation steps
• --gradient_clip_val: Gradient clipping threshold

These values are subsequently merged into trainer_config at lines ~1190-1225 and filtered through nondefault_trainer_args() at lines 1222-1230 to distinguish user-specified overrides from defaults.

Configuration Merging and YAML Base Files

The configuration system employs OmegaConf to merge hierarchical YAML files with command-line overrides:

1. Base Configuration Loading: The -b / --base argument (type list[str], default: []) accepts one or more YAML config files at main.py:L55-L62 that are merged left-to-right using OmegaConf.load().
2. Dotlist Overrides: Additional --key value pairs are parsed via OmegaConf.from_dotlist() and merged on top of the base configs at lines 1145-1154.
3. Final Config Object: The merged configuration produces a unified object containing model and data subtrees that instantiate the LightningModule and DataModule respectively.

This architecture separates hyperparameters (learning rates, model architecture) from runtime parameters (GPUs, logging directories).

Learning Rate Scaling Implementation

When --scale_lr is enabled, the script computes the effective learning rate at lines 862-889:

model.learning_rate = accumulate_grad_batches * ngpu * batch_size * base_lr

This calculation directly modifies the learning_rate attribute of the model's LightningModule (typically defined in ldm/models/diffusion/ddpm.py) before optimizer initialization, ensuring linear scaling rules are applied consistently across different hardware configurations.

Callbacks and Logging Infrastructure

The parsed configuration options drive the instantiation of several critical callbacks at lines 1245-1270:

• SetupCallback: Handles directory creation for logdir, ckptdir, and cfgdir based on parsed paths.
• ImageLogger: Configures logging intervals and image visualization parameters.
• LearningRateMonitor: Tracks learning rate changes throughout training.
• CUDACallback: Manages GPU-specific optimizations and emptying caches.

The Trainer receives these callbacks alongside the logger configuration at lines 1257-1260, creating a fully instrumented training environment.

Data Module Instantiation

The data configuration subtree is instantiated via instantiate_from_config(config.data) at line 1262, which dynamically imports and constructs the DataModule specified in the YAML configuration. Batch size, number of workers, and dataset-specific parameters flow from the merged config into the ldm/data classes.

Practical Fine-Tuning Examples

The following command-line patterns demonstrate typical fine-tuning configurations using the training options available in main.py:

Basic fine-tuning with custom config and disabled LR scaling:

python main.py \
  -b configs/stable-diffusion/v1-finetune.yaml \
  --scale_lr false \
  -t true \
  --seed 42 \
  --logdir ./my_runs \
  --name "my_finetune"

Resume from checkpoint with multi-GPU training:

python main.py \
  -b configs/base.yaml configs/extra.yaml \
  -r ./logs/2023-08-01T12-00-00_my_finetune/checkpoints/last.ckpt \
  -t true \
  --gpus 0,1,2,3 \
  --max_epochs 30

Evaluation-only run (no training):

python main.py \
  -b configs/eval.yaml \
  -r ./logs/2023-08-01T12-00-00_my_finetune/checkpoints/epoch=09.ckpt \
  --no-test false \
  -t false

Summary

• main.py serves as the unified entry point for Stable Diffusion fine-tuning, exposing controls through get_parser() and PyTorch Lightning's argument interface.
• Configuration merging combines multiple YAML base files (-b) with command-line dotlist overrides using OmegaConf at lines 1145-1154.
• Learning rate scaling automatically adjusts base rates by accumulate_grad_batches × GPUs × batch_size when --scale_lr is enabled (lines 862-889).
• Checkpoint resumption supports both directory paths and specific .ckpt files via the -r argument at lines 46-53.
• Trainer customization leverages Lightning's full flag suite for distributed training, precision, and gradient management.

Frequently Asked Questions

How do I resume training from a specific checkpoint in Stable Diffusion?

Use the -r / --resume argument followed by either a path to a checkpoint file (.ckpt) or the log directory containing a checkpoints folder. According to the source code at main.py:L46-L53, the script automatically detects the format and restores both model weights and optimizer states. You must also set -t true to continue training rather than just loading the model for inference.

What is the difference between --name and --postfix in main.py?

The --name argument (lines 35-44) sets the primary experiment identifier used in the log directory structure, while --postfix (lines 100-106) appends an additional string after the automatic timestamp and name components. Use --name for meaningful experiment labels (e.g., "lr_1e-4_b32") and --postfix for minor variations or version notes that should appear in the directory name but remain distinct from the core experiment identity.

How does the learning rate scaling work when fine-tuning?

When --scale_lr is set to true (the default at lines 115-122), the script calculates the effective learning rate as base_lr × accumulate_grad_batches × num_gpus × batch_size at lines 862-889. This linear scaling rule ensures that the optimization dynamics remain consistent regardless of whether you train on a single GPU with batch size 4 or eight GPUs with batch size 4 and gradient accumulation of 2.

Can I override specific values in the YAML config from the command line?

Yes. After specifying base configs with -b, provide additional arguments as --key value pairs (e.g., --model.params.lr_scheduler.target 5e-5). The parser at lines 1145-1154 converts these to dotlist notation and merges them into the OmegaConf configuration object, overriding any values defined in the YAML files. This allows quick hyperparameter sweeps without creating multiple config files.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}