# How Hypernetworks Work in AUTOMATIC1111: Architecture, Injection, and Training Guide

> Learn how Hypernetworks in AUTOMATIC1111 inject style and subject adaptations into Stable Diffusion's cross-attention layers without altering core weights. Explore architecture, injection, and training methods for custom AI art...

- Repository: [AUTOMATIC1111/stable-diffusion-webui](https://github.com/AUTOMATIC1111/stable-diffusion-webui)
- Tags: deep-dive
- Published: 2026-02-24

---

**Hypernetworks in AUTOMATIC1111 are small auxiliary MLPs that inject learned offsets into the cross-attention layers of Stable Diffusion, allowing style and subject adaptation without modifying the base checkpoint weights.**

The `AUTOMATIC1111/stable-diffusion-webui` repository implements hypernetworks as lightweight neural network modules that intercept key and value tensors during the diffusion process. Unlike full model fine-tuning, this approach trains only a tiny auxiliary network while keeping the base UNet frozen, significantly reducing storage and compute requirements.

## What Are Hypernetworks in AUTOMATIC1111?

Hypernetworks are **tiny auxiliary neural networks** that modify the behavior of the base diffusion model at inference time. According to the source code in [`modules/hypernetworks/hypernetwork.py`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/modules/hypernetworks/hypernetwork.py), a hypernetwork consists of small multi-layer perceptrons (MLPs) that generate additive offsets for the key (**K**) and value (**V**) tensors within the model's cross-attention blocks.

The system works by:

- **Freezing the base model**: The large Stable Diffusion checkpoint remains unchanged during training
- **Training small networks**: Only the hypernetwork parameters (typically a few megabytes) are updated
- **Runtime injection**: During inference, the hypernetwork outputs are added to attention tensors, altering the model's output distribution

This architecture allows users to create specialized models for specific styles, characters, or concepts without distributing multi-gigabyte checkpoint files.

## Architecture of Hypernetworks

### HypernetworkModule: The Core MLP

The fundamental building block is the `HypernetworkModule` class defined in [`modules/hypernetworks/hypernetwork.py`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/modules/hypernetworks/hypernetwork.py) (lines 25-73). Each module is configured with a **dimension** (`dim`) matching the base model's latent size (commonly 768 or 1024).

Key architectural features include:

- **Layer structure**: Defined by a list of multipliers where the first and last values must be `1`. For example, `[1, 2, 1]` creates a hidden layer twice as wide as the input dimension.
- **Activation functions**: Selected from `activation_dict` (lines 26-35), supporting `relu`, `swish`, and other nonlinearities.
- **Regularization**: Optional `LayerNorm` and `dropout_structure` for training stability.
- **Weight initialization**: Configurable schemes including `Normal` and `KaimingUniform` (lines 78-96).

The forward pass implements a residual connection with scaling:

```python

# From HypernetworkModule.forward (lines 17-19)

return x + self.linear(x) * self.multiplier

```

Here, `multiplier` is a user-controllable scalar (default `1.0`) that scales the hypernetwork's influence at runtime.

### The Hypernetwork Container

The `Hypernetwork` class (lines 44-77) acts as a container managing multiple `HypernetworkModule` instances. For each enabled size (e.g., 768, 1024), it maintains **two modules**: one for transforming the **K** (key) tensor and one for the **V** (value) tensor.

Critical methods include:

- **`weights()`**: Returns an iterator over all trainable parameters for the optimizer
- **`set_multiplier()`**: Globally scales all module outputs to control generation strength
- **`train()/eval()`**: Toggles `requires_grad` and training mode for the modules

Metadata including layer structure, activation functions, and optimizer state is preserved in the saved `.pt` files.

## Injection Mechanism: How Hypernetworks Modify Inference

Hypernetworks influence the diffusion process through **monkey-patching** of the UNet's attention mechanism. The WebUI replaces the standard `CrossAttention.forward` method with `attention_CrossAttention_forward` located in [`modules/sd_hijack_optimizations.py`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/modules/sd_hijack_optimizations.py) (lines 44-102).

Standard cross-attention computes:

```python
k = self.to_k(context)
v = self.to_v(context)

```

With hypernetworks enabled, the injection hook intercepts the context tensor:

```python

# Modified forward pass with hypernetwork injection

context_k, context_v = apply_hypernetworks(shared.loaded_hypernetworks, context, self)
k = self.to_k(context_k)
v = self.to_v(context_v)

```

The `apply_hypernetworks` function (defined in [`hypernetwork.py`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/hypernetwork.py)) iterates through all loaded hypernetworks, selects the appropriate module pair matching the current tensor dimension (`context_k.shape[2]`), and applies the learned offsets. The modified tensors are then cast back to the UNet's dtype using `devices.cond_cast_unet` before projection.

## Training Hypernetworks in AUTOMATIC1111

### Creating a Hypernetwork

Training begins with file creation via the `create_hypernetwork` function (lines 39-67). This initializes the architecture and saves an empty hypernetwork file:

```python
from modules.hypernetworks.hypernetwork import create_hypernetwork

create_hypernetwork(
    name="art_style_v1",
    enable_sizes=[768, 1024],
    overwrite_old=False,
    layer_structure="1,2,1",      # Input → 2x hidden → Output

    activation_func="swish",
    weight_init="KaimingUniform",
    add_layer_norm=False,
    use_dropout=False,
)

```

### The Training Loop

The `train_hypernetwork` function (lines 72-154) implements the training protocol. The process follows these steps:

1. **Dataset preparation**: Loads images paired with a placeholder token (the hypernetwork name) using `modules.textual_inversion.dataset.PersonalizedBase`
2. **Forward pass**: Runs latents through the frozen UNet with hypernetwork injection active
3. **Loss computation**: Calculates prediction error using the standard diffusion loss
4. **Backpropagation**: Updates **only** hypernetwork parameters via a dedicated optimizer:

```python

# Optimizer construction from train_hypernetwork (lines 49-55)

weights = hypernetwork.weights()
optimizer = optimizer_dict.get(hypernetwork.optimizer_name, torch.optim.AdamW)(
    params=weights,
    lr=scheduler.learn_rate,
)

```

5. **Checkpointing**: Periodically saves the hypernetwork state to `.pt` files and optionally generates preview images

Key training parameters include `learn_rate` (typically 1e-5 to 5e-6), `batch_size`, and `gradient_step` for accumulation. The `clip_grad_mode` option supports gradient clipping to prevent instability.

### Saving and Checkpointing

During training, `save_hypernetwork` (lines 70-84) writes both the model weights and optimizer state to disk. The primary file uses the `.pt` extension, while optimizer state (if enabled) saves as `.pt.optim`. This allows resuming training from exact checkpoints.

## Using Hypernetworks: Practical Examples

### Activating in Prompts

After training, activate the hypernetwork by including its name as a placeholder token in your prompt:

```text
"a portrait of a warrior, <art_style_v1>, highly detailed"

```

The WebUI automatically replaces `<art_style_v1>` with the hypernetwork's K/V transformations when the file is loaded in the **Extra networks** dropdown (controlled by the `sd_hypernetwork` option in [`modules/shared_options.py`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/modules/shared_options.py)).

### Runtime Multiplier Adjustment

Control the effect strength without retraining by adjusting the multiplier:

```python
from modules import shared

for hn in shared.loaded_hypernetworks:
    hn.set_multiplier(0.7)  # Apply at 70% strength

```

Setting the multiplier to `0.0` effectively disables the hypernetwork, while values above `1.0` amplify the effect.

### Programmatic Loading

Load hypernetworks dynamically in scripts:

```python
from modules.hypernetworks.hypernetwork import load_hypernetwork
from modules import shared, reload_hypernetworks

reload_hypernetworks()  # Refresh available hypernetworks from disk

hypernet = load_hypernetwork(shared.hypernetworks.get('art_style_v1'))
shared.loaded_hypernetworks = [hypernet]  # Activate for generation

```

### Exporting Weights for Research

Extract specific layer weights for analysis or transfer learning:

```python
import torch
from modules.hypernetworks.hypernetwork import load_hypernetwork

hn = load_hypernetwork('art_style_v1')
k_module_768 = hn.layers[768][0]  # K-module for 768-dim attention

torch.save(k_module_768.state_dict(), "extracted_k_weights.pt")

```

## Summary

- **Hypernetworks in AUTOMATIC1111** are small MLPs stored in `*.pt` files that modify cross-attention K/V tensors without changing the base model.
- **Architecture** consists of `HypernetworkModule` instances (defined in [`modules/hypernetworks/hypernetwork.py`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/modules/hypernetworks/hypernetwork.py)) arranged in K/V pairs for each supported latent dimension.
- **Injection** occurs via monkey-patching `CrossAttention.forward` in [`modules/sd_hijack_optimizations.py`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/modules/sd_hijack_optimizations.py), where `apply_hypernetworks` adds learned offsets to the context tensors.
- **Training** uses the `train_hypernetwork` function with a dedicated optimizer that updates only hypernetwork weights while the base UNet remains frozen.
- **Usage** involves loading the hypernetwork into `shared.loaded_hypernetworks` and referencing it via placeholder tokens in prompts, with runtime control via the `multiplier` attribute.

## Frequently Asked Questions

### How do Hypernetworks differ from LoRA in AUTOMATIC1111?

**Hypernetworks** inject transformations at the cross-attention context level before the K/V projections, while **LoRA** (Low-Rank Adaptation) modifies the weight matrices of the attention layers themselves directly. Hypernetworks use small MLPs to generate additive offsets for context embeddings, whereas LoRA adds low-rank matrices to the frozen weights. According to the source code, hypernetworks are implemented in [`modules/hypernetworks/hypernetwork.py`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/modules/hypernetworks/hypernetwork.py) and intercept tensors in `attention_CrossAttention_forward`, while LoRA patches apply directly to `to_k`, `to_q`, `to_v`, and `to_out` linear layers.

### What layer structure should I use for training a new Hypernetwork?

The optimal **layer structure** depends on your dataset complexity and the base model dimension. For most use cases, `[1, 2, 1]` (one input layer, one hidden layer twice the dimension, one output layer) provides sufficient capacity without overfitting. The source code requires the first and last multipliers to be `1`, with intermediate values defining hidden layer widths. For 768-dimensional models, `[1, 2, 1]` creates a 768→1536→768 MLP. Complex styles may benefit from deeper structures like `[1, 2, 2, 1]`, though this increases training time and overfitting risk.

### Why is my Hypernetwork not appearing in the generated images?

First, verify the hypernetwork is loaded in `shared.loaded_hypernetworks` and the placeholder token (e.g., `<hypernetwork_name>`) is included in your prompt. Check that the `multiplier` is not set to `0.0` via `set_multiplier()`. If using custom training, ensure the `enable_sizes` list includes your model's latent dimension (768 for SD 1.x, 1024 for SD 2.x). The injection only occurs in `attention_CrossAttention_forward` when the context tensor shape matches an enabled size. Also confirm the hypernetwork file was saved correctly by checking [`modules/hypernetworks/hypernetwork.py`](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/main/modules/hypernetworks/hypernetwork.py) for load errors.

### Can I use multiple Hypernetworks simultaneously?

Yes, the WebUI supports loading multiple hypernetworks into `shared.loaded_hypernetworks`, which functions as a list. During inference, `apply_hypernetworks` iterates through all loaded hypernetworks sequentially, applying each transformation to the context tensors. The effects combine multiplicatively through the attention mechanism. You can adjust individual strengths by setting different `multiplier` values on each hypernetwork instance before generation.