What Is save_interval in Faceswap Training? How Model Checkpointing Works

The save_interval parameter controls how frequently the Faceswap training loop writes model weights to disk, defaulting to every 250 iterations.

The save_interval setting is a critical command-line option in the deepfakes/faceswap repository that balances training performance against data safety. By configuring this value, you determine exactly how many iterations pass between each automatic model checkpoint, directly impacting your ability to recover from unexpected crashes and manage disk I/O during long training sessions.

Where save_interval Is Defined

The save_interval argument is declared in lib/cli/args_train.py as part of the training argument parser. According to the source code, this parameter is implemented as a slider-type argument with the following characteristics:

• Valid range: 10 to 1000 iterations
• Default value: 250 iterations
• Purpose: "Sets the number of iterations between each model save"

The argument definition appears at lines 208–215 in lib/cli/args_train.py, where it is registered alongside other training hyperparameters. When you launch a training session via scripts/train.py, this value is parsed into self._args.save_interval and consumed by the main training controller.

How save_interval Works in the Training Loop

Inside scripts/train.py, the main training loop evaluates the save_interval condition on every iteration to determine whether to trigger a checkpoint. The logic resides in the core training iteration handler (lines 37–41) and uses a modulo operation to test for interval alignment.

The Checkpointing Condition

At each iteration, the training script computes a boolean flag using the following logic:

save_iteration = iteration % self._args.save_interval == 0 or iteration == 1

This condition evaluates to True in two specific cases:

1. When the current iteration number is an exact multiple of save_interval (e.g., 250, 500, 750 when using the default)
2. On the very first iteration (iteration 1), ensuring an immediate initial checkpoint

What Happens When a Save Is Triggered

When save_iteration evaluates to True, the training loop executes several persistence operations:

• Calls the trainer's save() method (implemented in plugins/train/trainer/_base.py) to write model weights to disk
• Optionally generates a timelapse snapshot if timelapse functionality is enabled
• Forces the GUI preview to refresh, ensuring the latest state is visible

This mechanism operates independently of manual saves (triggered by pressing the S key, which sets self._save_now) and the separate --snapshot-interval option that creates periodic backup snapshots.

Adjusting save_interval for Your Workflow

Changing the save_interval value creates distinct trade-offs between safety and performance:

• Smaller values (e.g., 50–100): Generate frequent checkpoints that minimize data loss if training crashes, but increase disk I/O overhead and may slow training on systems with slow storage.
• Larger values (e.g., 500–1000): Reduce disk write frequency to maximize training throughput, but risk losing more progress if the process terminates unexpectedly.

The optimal setting depends on your hardware stability and the total length of your training session. For experimental short runs, a larger interval reduces overhead; for production training lasting days, a conservative interval provides insurance against power failures or system crashes.

Using save_interval in Practice

To launch training with a custom checkpoint frequency, pass the -s or --save-interval flag followed by your desired iteration count:

python scripts/train.py -i 5000 -s 100

This command trains for 5,000 iterations while saving model weights every 100 iterations instead of the default 250.

When developing custom trainer plugins, you can inspect the save condition programmatically through the trainer state:

def train_one_step(self, viewer, timelapse):
    # ... perform forward/backward pass ...

    if self._trainer.save_iteration:
        self.save(is_exit=False)  # Called automatically by the main loop

Summary

• save_interval is defined in lib/cli/args_train.py (lines 208–215) and defaults to 250 iterations.
• The training loop in scripts/train.py uses modulo arithmetic (iteration % save_interval == 0) to trigger saves at fixed intervals.
• Each checkpoint invokes the trainer's save() method and refreshes the GUI preview.
• Lower values increase save frequency and safety at the cost of I/O performance; higher values reduce overhead but increase risk of data loss.
• The parameter is set via the -s flag when running scripts/train.py.

Frequently Asked Questions

What is the default save_interval in Faceswap training?

The default save_interval is 250 iterations, as defined in lib/cli/args_train.py. This means the model automatically writes weights to disk every 250 training steps unless you specify a different value using the -s command-line option.

How does save_interval differ from snapshot-interval?

save_interval controls the primary model checkpoint frequency (the weights you load to resume training), while snapshot-interval creates separate backup snapshots for version history. Additionally, pressing S during training triggers an immediate manual save independent of both interval timers.

Can I change the save_interval during an active training session?

No, you cannot modify the save_interval value without restarting the training process. The parameter is parsed once at startup from self._args in scripts/train.py. To change checkpoint frequency, stop training and relaunch with a new -s value.

What happens if I set save_interval to a value higher than my total iterations?

If you set save_interval higher than your total iteration count (specified by -i), the model will only save once—on the first iteration—and again only if you manually trigger a save with the S key. The automatic checkpoint condition will never evaluate to True because the modulo check requires the iteration number to equal or exceed the interval value.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}