How the Dual-Autoregressive Architecture Works in Fish Speech S2: A Deep Dive

Fish Speech S2 uses a Dual-Autoregressive (Dual-AR) architecture that separates semantic token prediction from VQ-codebook generation, employing a slow transformer for high-level semantics and a fast transformer for fine-grained audio details.

The Dual-Autoregressive architecture is the core innovation powering Fish Speech S2's ability to generate high-quality, coherent speech. Implemented in the fishaudio/fish-speech repository, this design splits the generation process into two distinct autoregressive stages that operate in tandem. Understanding how these stages interact is essential for anyone working with or extending the Fish Speech codebase.

What Is the Dual-Autoregressive Architecture?

The Dual-AR model decouples speech generation into semantic reasoning and acoustic reconstruction. This separation allows the model to maintain long-range coherence through a large "slow" transformer while efficiently generating high-frequency audio details through a compact "fast" transformer.

Stage	Purpose	Key Components
Slow Transformer	Predicts the next semantic token (high-level text-like representation)	BaseTransformer with shared embeddings, self-attention layers, and RMSNorm
Fast Transformer	Conditioned on semantic hidden states, autoregressively predicts VQ-codebook indices	DualARTransformer fast sub-network with dedicated embeddings, transformer blocks, and output head

Both stages are autoregressive—each feeds its own outputs back as inputs—but they operate at different temporal resolutions and computational scales.

Core Components and Implementation

The Dual-Autoregressive architecture is implemented in fish_speech/models/text2semantic/llama.py, which defines the DualARTransformer class extending BaseTransformer.

Slow Transformer: BaseTransformer

The slow transformer serves as the foundation. It processes the input token sequence through standard transformer layers to produce hidden states and logits for the next semantic token.

parent_result = super().forward(inp=inp, key_padding_mask=key_padding_mask)
token_logits = parent_result.logits
x = parent_result.hidden_states

Source: [llama.py lines 23-33](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py#L23-L33)

Fast Transformer: The Codebook Decoder

The fast transformer is a dedicated sub-network within DualARTransformer. It includes:

• Fast embeddings: nn.Embedding(config.codebook_size, config.fast_dim) for VQ-codebook indices
• Fast layers: A stack of TransformerBlock instances (typically 4 layers) with use_sdpa=False
• Fast normalization: RMSNorm for training stability
• Output projection: Linear layer mapping to codebook_size

self.fast_embeddings = nn.Embedding(config.codebook_size, config.fast_dim)
self.fast_layers = nn.ModuleList(
    TransformerBlock(override_config, use_sdpa=False) for _ in range(config.n_fast_layer)
)
self.fast_norm = RMSNorm(config.fast_dim, eps=config.norm_eps)
self.fast_output = nn.Linear(config.fast_dim, config.codebook_size, bias=False)

Source: [llama.py lines 69-95](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py#L69-L95)

DualARModelArgs Configuration

The architecture is controlled by DualARModelArgs, a dataclass extending BaseModelArgs:

@dataclass
class DualARModelArgs(BaseModelArgs):
    model_type: str = "dual_ar"
    n_fast_layer: int = 4
    fast_dim: int | None = None
    fast_n_head: int | None = None
    fast_n_local_heads: int | None = None
    fast_head_dim: int | None = None
    fast_intermediate_size: int | None = None
    fast_attention_qkv_bias: bool | None = None
    fast_attention_qk_norm: bool | None = None
    fast_attention_o_bias: bool | None = None
    norm_fastlayer_input: bool = False

During initialization, missing fast-specific values automatically fall back to their slow-transformer counterparts, ensuring backward compatibility.

Source: [llama.py lines 55-69](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py#L55-L69)

Forward Pass: How the Two Stages Interact

The DualARTransformer.forward method orchestrates the interaction between slow and fast stages through a precise sequence of operations.

Step 1: Input Processing

The input sequence first passes through the slow transformer to obtain hidden states and semantic logits:

parent_result = super().forward(inp=inp, key_padding_mask=key_padding_mask)
token_logits = parent_result.logits
x = parent_result.hidden_states

Step 2: Semantic Masking

The model identifies which positions correspond to semantic tokens (those within the range semantic_begin_id to semantic_end_id). Only these positions proceed to the fast stage:

codebook_mask = (token_labels >= self.config.semantic_begin_id) & (
    token_labels <= self.config.semantic_end_id
)
x = x[codebook_mask]

Source: [llama.py lines 51-55](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py#L51-L55)

Step 3: Fast Sub-Network Construction

If configured with a different dimensionality, the hidden states pass through fast_project_in. The fast transformer then processes these states alongside VQ-codebook embeddings:

x = self.fast_project_in(x)
codebook_embeddings = self.fast_embeddings(codebooks)
x = torch.cat([x[:, None], codebook_embeddings], dim=1)
for layer in self.fast_layers:
    # attention + feed-forward processing

    ...
fast_out = self.fast_norm(x)
codebook_logits = self.fast_output(fast_out)

Source: [llama.py lines 77-94](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py#L77-L94)

Step 4: Output Generation

The forward pass returns both semantic token logits and codebook logits, enabling single-pass generation of the complete token stream:

return TransformerForwardResult(
    token_logits=token_logits,
    codebook_logits=codebook_logits,
)

Source: [llama.py lines 92-96](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py#L92-L96)

Inference Pipeline

The inference utilities in fish_speech/models/text2semantic/inference.py orchestrate the two-stage decoding process during generation.

decode_one_token_ar: Single Step Decoding

The decode_one_token_ar function implements the core dual-stage generation logic. It first invokes model.forward_generate (slow stage) to obtain the semantic token, then repeatedly calls model.forward_generate_fast to produce the VQ codebooks:

# Slow stage: generate semantic token

# Fast stage: generate codebook tokens conditioned on semantic hidden state

Source: [inference.py lines 96-124](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/inference.py#L96-L124)

generate: Full Sequence Generation

The generate function sets up KV caches for both transformers once, then iteratively calls decode_n_tokens to emit sequences of (num_codebooks + 1)-wide tokens (semantic + fast):

# Setup KV caches for slow and fast transformers

# Iterative generation of complete token sequences

Source: [inference.py lines 44-89](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/inference.py#L44-L89)

Summary

• Fish Speech S2 implements a Dual-Autoregressive architecture that splits speech generation into semantic prediction and acoustic detail generation.
• The slow transformer (BaseTransformer) handles high-level semantic tokens, providing long-range coherence through standard autoregressive processing.
• The fast transformer operates as a dedicated sub-network within DualARTransformer, generating VQ-codebook indices conditioned on semantic hidden states.
• Semantic masking ensures only relevant positions trigger the fast stage, optimizing computational efficiency.
• The architecture is configured through DualARModelArgs, allowing independent control of fast-layer dimensions, attention heads, and normalization strategies.
• Inference utilities in inference.py orchestrate the two-stage decoding through decode_one_token_ar and generate, maintaining separate KV caches for each stage.

Frequently Asked Questions

What is the difference between the slow and fast transformers in Dual-AR?

The slow transformer is the main BaseTransformer that processes the full input sequence to predict semantic tokens—high-level representations that capture linguistic content and long-range structure. The fast transformer is a smaller, dedicated sub-network within DualARTransformer that operates only on positions marked as semantic tokens. It predicts the VQ-codebook indices needed to reconstruct the actual audio waveform, conditioned on the hidden states produced by the slow stage.

How does the Dual-Autoregressive architecture improve speech quality?

By separating semantic reasoning from acoustic generation, the Dual-AR architecture allows each component to specialize. The slow transformer can be large and deep, capturing complex linguistic patterns and prosody over long contexts without being burdened by the high resolution of audio tokens. Meanwhile, the fast transformer efficiently generates the fine-grained VQ-codebook sequences needed for high-fidelity audio reconstruction. This division prevents the "one model fits all" compromise, yielding clearer, more coherent, and more natural-sounding speech.

Can I adjust the fast transformer dimensions independently?

Yes. The DualARModelArgs configuration class allows you to specify independent dimensions for the fast transformer using parameters like fast_dim, fast_n_head, fast_head_dim, and fast_intermediate_size. If these values are not explicitly set, they automatically fall back to their counterparts in the slow transformer (dim, n_head, etc.), ensuring backward compatibility while providing flexibility for architectural experimentation.

Where is the Dual-AR model defined in the Fish Speech codebase?

The core implementation resides in fish_speech/models/text2semantic/llama.py, which defines the DualARTransformer class, DualARModelArgs configuration, and the fast sub-network components. The inference logic that orchestrates the two-stage generation is located in fish_speech/models/text2semantic/inference.py, containing functions like decode_one_token_ar and generate that manage the KV caches and token generation pipeline.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}