How GRPO Reinforcement Learning Alignment Works in Fish-Speech

Fish-Speech implements Group Relative Policy Optimization (GRPO) by repurposing its data-filtering models as reward models, eliminating distribution mismatch during post-training alignment of the text-to-semantic transformer.

Fish-Speech uses GRPO reinforcement learning alignment to fine-tune its text-to-semantic model after large-scale pre-training. This approach reuses the same transformer architecture that filtered and annotated the training data, ensuring the reward model shares identical weights and data distribution with the policy model.

What Is GRPO and Why Fish-Speech Uses It

Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm that updates the policy relative to groups of tokens rather than individual actions. Standard Proximal Policy Optimization (PPO) often suffers from distribution mismatch when a separate reward model—trained on different data—is used for post-training alignment.

Fish-Speech solves this by reusing the very same models that filtered and annotated the training data as reward models for RL. Because the reward model is identical to the pre-training filter, it sees the exact same data distribution as the policy, removing the "pre-training vs. post-training mismatch" that causes reward hacking and instability.

Architecture of GRPO Reinforcement Learning Alignment

The Reward Model Flag and Score Head

The architecture centers on a boolean flag in BaseModelArgs that switches the transformer between generation and reward modes:

is_reward_model: bool = False   # set to True for RL alignment

Source: [fish_speech/models/text2semantic/llama.py](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py#L61)

When is_reward_model is True, the model replaces the standard language-model head with a score head (self.score_output). This head produces a scalar reward for each generated token (or for the whole sequence) that is fed to the RL optimizer.

Switching Between Generation and Reward Modes

The forward pass that toggles between modes is implemented in BaseTransformer.forward_generate (lines 452-456 of the model file). When self.config.is_reward_model is enabled, hidden states are passed through self.score_output instead of the usual language-model head:

if self.config.is_reward_model:
    token_logits = self.score_output(slow_out)   # <-- reward head

elif self.config.tie_word_embeddings:
    token_logits = F.linear(slow_out, self.embeddings.weight)
else:
    token_logits = self.output(slow_out)

Source: [fish_speech/models/text2semantic/llama.py](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py#L452-L456)

The Four-Component Reward Signal

GRPO’s reward combines four complementary criteria that guide the text-to-semantic model toward high-quality speech synthesis:

1. Semantic accuracy – how well the generated semantic tokens match the intended meaning of the input text.
2. Instruction adherence – compliance with textual control tags (e.g., style, tone, speaking rate) provided in the prompt.
3. Acoustic preference scoring – a learned preference over acoustic qualities such as clarity, naturalness, and absence of artifacts.
4. Timbre similarity – consistency of the generated speaker’s voice identity with the reference audio or speaker embedding.

According to the Fish-Speech documentation, "The reward signal combines semantic accuracy, instruction adherence, acoustic preference scoring, and timbre similarity."

Source: [docs/en/index.md](https://github.com/fishaudio/fish-speech/blob/main/docs/en/index.md#reinforcement-learning-alignment)

Implementing GRPO Alignment in Practice

Loading a Model in Reward Mode

To enable reward computation, instantiate the BaseTransformer and toggle the reward flag:

from fish_speech.models.text2semantic.llama import BaseTransformer, BaseModelArgs

# Load the pretrained checkpoint (weights are optional here)

model = BaseTransformer.from_pretrained(
    path="checkpoints/openaudio-s1-mini",
    load_weights=True,
    max_length=4096,
)

# Enable reward‑model mode

model.config.is_reward_model = True

# (Optional) Replace the default head with a custom scoring head

# For illustration we attach a tiny linear layer; in practice a

# pretrained reward head would be loaded.

import torch.nn as nn
model.score_output = nn.Linear(model.config.dim, 1)   # scalar reward

Key files: [fish_speech/models/text2semantic/llama.py](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py)

Running a Forward Pass for Rewards

Once in reward mode, the forward pass returns scalar rewards instead of token logits:

import torch

# Dummy token sequence (batch=1, seq_len=5, 1+num_codebooks channels)

# The first channel holds semantic ids; the remaining channels hold VQ codebook ids.

dummy_input = torch.randint(
    low=0,
    high=model.config.vocab_size,
    size=(1, model.config.num_codebooks + 1, 5),
)

# Forward‑generate returns a BaseTransformerForwardResult; in reward mode,

# `logits` are the scalar rewards.

result = model.forward_generate(dummy_input)

rewards = result.logits.squeeze()   # shape: [batch, seq_len]

print("Reward per token:", rewards)

GRPO Training Loop Skeleton

While the repository does not ship a full GRPO trainer, the reward head can be plugged into any policy-gradient optimizer:

import torch
import torch.optim as optim

optimizer = optim.AdamW(model.parameters(), lr=1e-4)

for epoch in range(num_epochs):
    for batch in dataloader:
        # 1️⃣ Generate tokens & rewards

        result = model.forward_generate(batch["tokens"])

        # 2️⃣ Compute GRPO loss (pseudo‑code)

        #    The actual GRPO algorithm is defined in the official paper;

        #    here we illustrate the typical policy‑gradient term.

        rewards = result.logits  # [B, T, 1]

        log_probs = torch.log_softmax(result.hidden_states, dim=-1)  # policy logits

        loss = - (rewards * log_probs).mean()   # maximise expected reward

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

Note – The repository does not ship a full GRPO trainer; the snippet shows how the reward head can be plugged into any policy‑gradient style optimiser.

Key Files for GRPO Implementation

• fish_speech/models/text2semantic/llama.py – Defines BaseModelArgs (including is_reward_model), the main transformer (BaseTransformer), and the conditional switch to self.score_output for reward generation.
• docs/en/index.md – Human‑readable description of GRPO, its motivation, and the composition of the reward signal.
• fish_speech/models/text2semantic/lora.py – Shows how LoRA adapters can be attached to the model; useful if GRPO fine‑tuning is performed with parameter‑efficient updates.
• fish_speech/configs/text2semantic_finetune.yaml – Example configuration for finetuning the text‑semantic model; can be extended with is_reward_model: true to enable RL alignment.

Summary

• GRPO eliminates distribution mismatch by reusing the same transformer that filtered training data as the reward model for reinforcement learning.
• Architecture toggle – Setting is_reward_model=True in BaseModelArgs switches the forward pass from token generation to scalar reward computation via self.score_output.
• Multi-objective reward – The reward signal combines semantic accuracy, instruction adherence, acoustic preference scoring, and timbre similarity.
• Implementation path – The core logic resides in fish_speech/models/text2semantic/llama.py, with configuration support in the YAML configs and documentation in docs/en/index.md.

Frequently Asked Questions

What makes GRPO different from standard PPO in speech synthesis?

GRPO updates the policy relative to groups of tokens (such as entire utterances) rather than individual token rewards, which stabilizes training for hierarchical autoregressive models. Unlike standard PPO, which often employs a separately trained reward model, Fish-Speech’s GRPO implementation reuses the identical checkpoint that filtered the pre-training data, eliminating distribution shift between the reward model and the policy.

How does Fish-Speech avoid reward hacking with GRPO?

Fish-Speech mitigates reward hacking by ensuring the reward model is architecturally identical to the data-filtering model used during pre-training. Because the reward head (self.score_output) operates on the same hidden states and sees the same data distribution as the original filter, it cannot exploit arbitrary gaps between a separately trained reward model and the policy network.

Can I use GRPO alignment with LoRA adapters?

Yes. The repository includes fish_speech/models/text2semantic/lora.py, which demonstrates how to attach Low-Rank Adaptation (LoRA) layers to the base transformer. You can enable is_reward_model=True while using LoRA to perform parameter-efficient GRPO fine-tuning, updating only the adapter weights and the score head during reinforcement learning.

Where is the reward score calculated in the forward pass?

The reward score is calculated in BaseTransformer.forward_generate within fish_speech/models/text2semantic/llama.py (lines 452-456). When self.config.is_reward_model is True, the method routes hidden states through self.score_output instead of the standard language-model head, producing scalar rewards used for the GRPO loss calculation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}