# How the Dual-Autoregressive Architecture Works in Fish Speech S2: A Deep Dive

> Explore the Dual-Autoregressive architecture in Fish Speech S2. Learn how it separates semantic token prediction from VQ-codebook generation using slow and fast transformers for detailed audio.

- Repository: [Fish Audio/fish-speech](https://github.com/fishaudio/fish-speech)
- Tags: deep-dive
- Published: 2026-03-12

---

**Fish Speech S2 uses a Dual-Autoregressive (Dual-AR) architecture that separates semantic token prediction from VQ-codebook generation, employing a slow transformer for high-level semantics and a fast transformer for fine-grained audio details.**

The Dual-Autoregressive architecture is the core innovation powering Fish Speech S2's ability to generate high-quality, coherent speech. Implemented in the `fishaudio/fish-speech` repository, this design splits the generation process into two distinct autoregressive stages that operate in tandem. Understanding how these stages interact is essential for anyone working with or extending the Fish Speech codebase.

## What Is the Dual-Autoregressive Architecture?

The Dual-AR model decouples speech generation into **semantic reasoning** and **acoustic reconstruction**. This separation allows the model to maintain long-range coherence through a large "slow" transformer while efficiently generating high-frequency audio details through a compact "fast" transformer.

| Stage | Purpose | Key Components |
|-------|---------|----------------|
| **Slow Transformer** | Predicts the next **semantic token** (high-level text-like representation) | `BaseTransformer` with shared embeddings, self-attention layers, and RMSNorm |
| **Fast Transformer** | Conditioned on semantic hidden states, autoregressively predicts VQ-codebook indices | `DualARTransformer` fast sub-network with dedicated embeddings, transformer blocks, and output head |

Both stages are autoregressive—each feeds its own outputs back as inputs—but they operate at different temporal resolutions and computational scales.

## Core Components and Implementation

The Dual-Autoregressive architecture is implemented in [`fish_speech/models/text2semantic/llama.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py), which defines the `DualARTransformer` class extending `BaseTransformer`.

### Slow Transformer: BaseTransformer

The slow transformer serves as the foundation. It processes the input token sequence through standard transformer layers to produce hidden states and logits for the next semantic token.

```python
parent_result = super().forward(inp=inp, key_padding_mask=key_padding_mask)
token_logits = parent_result.logits
x = parent_result.hidden_states

```

*Source*: [[`llama.py`](https://github.com/fishaudio/fish-speech/blob/main/llama.py) lines 23-33](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py#L23-L33)

### Fast Transformer: The Codebook Decoder

The fast transformer is a dedicated sub-network within `DualARTransformer`. It includes:

- **Fast embeddings**: `nn.Embedding(config.codebook_size, config.fast_dim)` for VQ-codebook indices
- **Fast layers**: A stack of `TransformerBlock` instances (typically 4 layers) with `use_sdpa=False`
- **Fast normalization**: RMSNorm for training stability
- **Output projection**: Linear layer mapping to `codebook_size`

```python
self.fast_embeddings = nn.Embedding(config.codebook_size, config.fast_dim)
self.fast_layers = nn.ModuleList(
    TransformerBlock(override_config, use_sdpa=False) for _ in range(config.n_fast_layer)
)
self.fast_norm = RMSNorm(config.fast_dim, eps=config.norm_eps)
self.fast_output = nn.Linear(config.fast_dim, config.codebook_size, bias=False)

```

*Source*: [[`llama.py`](https://github.com/fishaudio/fish-speech/blob/main/llama.py) lines 69-95](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py#L69-L95)

### DualARModelArgs Configuration

The architecture is controlled by `DualARModelArgs`, a dataclass extending `BaseModelArgs`:

```python
@dataclass
class DualARModelArgs(BaseModelArgs):
    model_type: str = "dual_ar"
    n_fast_layer: int = 4
    fast_dim: int | None = None
    fast_n_head: int | None = None
    fast_n_local_heads: int | None = None
    fast_head_dim: int | None = None
    fast_intermediate_size: int | None = None
    fast_attention_qkv_bias: bool | None = None
    fast_attention_qk_norm: bool | None = None
    fast_attention_o_bias: bool | None = None
    norm_fastlayer_input: bool = False

```

During initialization, missing fast-specific values automatically fall back to their slow-transformer counterparts, ensuring backward compatibility.

*Source*: [[`llama.py`](https://github.com/fishaudio/fish-speech/blob/main/llama.py) lines 55-69](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py#L55-L69)

## Forward Pass: How the Two Stages Interact

The `DualARTransformer.forward` method orchestrates the interaction between slow and fast stages through a precise sequence of operations.

### Step 1: Input Processing

The input sequence first passes through the slow transformer to obtain hidden states and semantic logits:

```python
parent_result = super().forward(inp=inp, key_padding_mask=key_padding_mask)
token_logits = parent_result.logits
x = parent_result.hidden_states

```

### Step 2: Semantic Masking

The model identifies which positions correspond to semantic tokens (those within the range `semantic_begin_id` to `semantic_end_id`). Only these positions proceed to the fast stage:

```python
codebook_mask = (token_labels >= self.config.semantic_begin_id) & (
    token_labels <= self.config.semantic_end_id
)
x = x[codebook_mask]

```

*Source*: [[`llama.py`](https://github.com/fishaudio/fish-speech/blob/main/llama.py) lines 51-55](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py#L51-L55)

### Step 3: Fast Sub-Network Construction

If configured with a different dimensionality, the hidden states pass through `fast_project_in`. The fast transformer then processes these states alongside VQ-codebook embeddings:

```python
x = self.fast_project_in(x)
codebook_embeddings = self.fast_embeddings(codebooks)
x = torch.cat([x[:, None], codebook_embeddings], dim=1)
for layer in self.fast_layers:
    # attention + feed-forward processing

    ...
fast_out = self.fast_norm(x)
codebook_logits = self.fast_output(fast_out)

```

*Source*: [[`llama.py`](https://github.com/fishaudio/fish-speech/blob/main/llama.py) lines 77-94](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py#L77-L94)

### Step 4: Output Generation

The forward pass returns both semantic token logits and codebook logits, enabling single-pass generation of the complete token stream:

```python
return TransformerForwardResult(
    token_logits=token_logits,
    codebook_logits=codebook_logits,
)

```

*Source*: [[`llama.py`](https://github.com/fishaudio/fish-speech/blob/main/llama.py) lines 92-96](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py#L92-L96)

## Inference Pipeline

The inference utilities in [`fish_speech/models/text2semantic/inference.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/inference.py) orchestrate the two-stage decoding process during generation.

### decode_one_token_ar: Single Step Decoding

The `decode_one_token_ar` function implements the core dual-stage generation logic. It first invokes `model.forward_generate` (slow stage) to obtain the semantic token, then repeatedly calls `model.forward_generate_fast` to produce the VQ codebooks:

```python

# Slow stage: generate semantic token

# Fast stage: generate codebook tokens conditioned on semantic hidden state

```

*Source*: [[`inference.py`](https://github.com/fishaudio/fish-speech/blob/main/inference.py) lines 96-124](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/inference.py#L96-L124)

### generate: Full Sequence Generation

The `generate` function sets up KV caches for both transformers once, then iteratively calls `decode_n_tokens` to emit sequences of `(num_codebooks + 1)`-wide tokens (semantic + fast):

```python

# Setup KV caches for slow and fast transformers

# Iterative generation of complete token sequences

```

*Source*: [[`inference.py`](https://github.com/fishaudio/fish-speech/blob/main/inference.py) lines 44-89](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/inference.py#L44-L89)

## Summary

- **Fish Speech S2** implements a **Dual-Autoregressive architecture** that splits speech generation into semantic prediction and acoustic detail generation.
- The **slow transformer** (`BaseTransformer`) handles high-level semantic tokens, providing long-range coherence through standard autoregressive processing.
- The **fast transformer** operates as a dedicated sub-network within `DualARTransformer`, generating VQ-codebook indices conditioned on semantic hidden states.
- **Semantic masking** ensures only relevant positions trigger the fast stage, optimizing computational efficiency.
- The architecture is configured through `DualARModelArgs`, allowing independent control of fast-layer dimensions, attention heads, and normalization strategies.
- Inference utilities in [`inference.py`](https://github.com/fishaudio/fish-speech/blob/main/inference.py) orchestrate the two-stage decoding through `decode_one_token_ar` and `generate`, maintaining separate KV caches for each stage.

## Frequently Asked Questions

### What is the difference between the slow and fast transformers in Dual-AR?

The **slow transformer** is the main `BaseTransformer` that processes the full input sequence to predict semantic tokens—high-level representations that capture linguistic content and long-range structure. The **fast transformer** is a smaller, dedicated sub-network within `DualARTransformer` that operates only on positions marked as semantic tokens. It predicts the VQ-codebook indices needed to reconstruct the actual audio waveform, conditioned on the hidden states produced by the slow stage.

### How does the Dual-Autoregressive architecture improve speech quality?

By separating **semantic reasoning** from **acoustic generation**, the Dual-AR architecture allows each component to specialize. The slow transformer can be large and deep, capturing complex linguistic patterns and prosody over long contexts without being burdened by the high resolution of audio tokens. Meanwhile, the fast transformer efficiently generates the fine-grained VQ-codebook sequences needed for high-fidelity audio reconstruction. This division prevents the "one model fits all" compromise, yielding clearer, more coherent, and more natural-sounding speech.

### Can I adjust the fast transformer dimensions independently?

Yes. The `DualARModelArgs` configuration class allows you to specify independent dimensions for the fast transformer using parameters like `fast_dim`, `fast_n_head`, `fast_head_dim`, and `fast_intermediate_size`. If these values are not explicitly set, they automatically fall back to their counterparts in the slow transformer (`dim`, `n_head`, etc.), ensuring backward compatibility while providing flexibility for architectural experimentation.

### Where is the Dual-AR model defined in the Fish Speech codebase?

The core implementation resides in [`fish_speech/models/text2semantic/llama.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py), which defines the `DualARTransformer` class, `DualARModelArgs` configuration, and the fast sub-network components. The inference logic that orchestrates the two-stage generation is located in [`fish_speech/models/text2semantic/inference.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/inference.py), containing functions like `decode_one_token_ar` and `generate` that manage the KV caches and token generation pipeline.