# How GRPO Reinforcement Learning Alignment Works in Fish-Speech

> Explore how GRPO reinforcement learning alignment works in Fish-Speech. Discover how text-to-semantic transformers are aligned using data-filtering models as reward models.

- Repository: [Fish Audio/fish-speech](https://github.com/fishaudio/fish-speech)
- Tags: deep-dive
- Published: 2026-03-12

---

**Fish-Speech implements Group Relative Policy Optimization (GRPO) by repurposing its data-filtering models as reward models, eliminating distribution mismatch during post-training alignment of the text-to-semantic transformer.**

Fish-Speech uses **GRPO reinforcement learning alignment** to fine-tune its text-to-semantic model after large-scale pre-training. This approach reuses the same transformer architecture that filtered and annotated the training data, ensuring the reward model shares identical weights and data distribution with the policy model.

## What Is GRPO and Why Fish-Speech Uses It

**Group Relative Policy Optimization (GRPO)** is a reinforcement learning algorithm that updates the policy relative to groups of tokens rather than individual actions. Standard Proximal Policy Optimization (PPO) often suffers from distribution mismatch when a separate reward model—trained on different data—is used for post-training alignment.

Fish-Speech solves this by **reusing the very same models that filtered and annotated the training data as reward models for RL**. Because the reward model is identical to the pre-training filter, it sees the exact same data distribution as the policy, removing the "pre-training vs. post-training mismatch" that causes reward hacking and instability.

## Architecture of GRPO Reinforcement Learning Alignment

### The Reward Model Flag and Score Head

The architecture centers on a boolean flag in `BaseModelArgs` that switches the transformer between generation and reward modes:

```python
is_reward_model: bool = False   # set to True for RL alignment

```

*Source*: [[`fish_speech/models/text2semantic/llama.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py)](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py#L61)

When `is_reward_model` is `True`, the model replaces the standard language-model head with a **score head** (`self.score_output`). This head produces a scalar reward for each generated token (or for the whole sequence) that is fed to the RL optimizer.

### Switching Between Generation and Reward Modes

The forward pass that toggles between modes is implemented in `BaseTransformer.forward_generate` (lines 452-456 of the model file). When `self.config.is_reward_model` is enabled, hidden states are passed through `self.score_output` instead of the usual language-model head:

```python
if self.config.is_reward_model:
    token_logits = self.score_output(slow_out)   # <-- reward head

elif self.config.tie_word_embeddings:
    token_logits = F.linear(slow_out, self.embeddings.weight)
else:
    token_logits = self.output(slow_out)

```

*Source*: [[`fish_speech/models/text2semantic/llama.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py)](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py#L452-L456)

## The Four-Component Reward Signal

GRPO’s reward combines **four complementary criteria** that guide the text-to-semantic model toward high-quality speech synthesis:

1. **Semantic accuracy** – how well the generated semantic tokens match the intended meaning of the input text.
2. **Instruction adherence** – compliance with textual control tags (e.g., style, tone, speaking rate) provided in the prompt.
3. **Acoustic preference scoring** – a learned preference over acoustic qualities such as clarity, naturalness, and absence of artifacts.
4. **Timbre similarity** – consistency of the generated speaker’s voice identity with the reference audio or speaker embedding.

According to the Fish-Speech documentation, "The reward signal combines semantic accuracy, instruction adherence, acoustic preference scoring, and timbre similarity."

*Source*: [[`docs/en/index.md`](https://github.com/fishaudio/fish-speech/blob/main/docs/en/index.md)](https://github.com/fishaudio/fish-speech/blob/main/docs/en/index.md#reinforcement-learning-alignment)

## Implementing GRPO Alignment in Practice

### Loading a Model in Reward Mode

To enable reward computation, instantiate the `BaseTransformer` and toggle the reward flag:

```python
from fish_speech.models.text2semantic.llama import BaseTransformer, BaseModelArgs

# Load the pretrained checkpoint (weights are optional here)

model = BaseTransformer.from_pretrained(
    path="checkpoints/openaudio-s1-mini",
    load_weights=True,
    max_length=4096,
)

# Enable reward‑model mode

model.config.is_reward_model = True

# (Optional) Replace the default head with a custom scoring head

# For illustration we attach a tiny linear layer; in practice a

# pretrained reward head would be loaded.

import torch.nn as nn
model.score_output = nn.Linear(model.config.dim, 1)   # scalar reward

```

*Key files*: [[`fish_speech/models/text2semantic/llama.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py)](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py)

### Running a Forward Pass for Rewards

Once in reward mode, the forward pass returns scalar rewards instead of token logits:

```python
import torch

# Dummy token sequence (batch=1, seq_len=5, 1+num_codebooks channels)

# The first channel holds semantic ids; the remaining channels hold VQ codebook ids.

dummy_input = torch.randint(
    low=0,
    high=model.config.vocab_size,
    size=(1, model.config.num_codebooks + 1, 5),
)

# Forward‑generate returns a BaseTransformerForwardResult; in reward mode,

# `logits` are the scalar rewards.

result = model.forward_generate(dummy_input)

rewards = result.logits.squeeze()   # shape: [batch, seq_len]

print("Reward per token:", rewards)

```

### GRPO Training Loop Skeleton

While the repository does not ship a full GRPO trainer, the reward head can be plugged into any policy-gradient optimizer:

```python
import torch
import torch.optim as optim

optimizer = optim.AdamW(model.parameters(), lr=1e-4)

for epoch in range(num_epochs):
    for batch in dataloader:
        # 1️⃣ Generate tokens & rewards

        result = model.forward_generate(batch["tokens"])

        # 2️⃣ Compute GRPO loss (pseudo‑code)

        #    The actual GRPO algorithm is defined in the official paper;

        #    here we illustrate the typical policy‑gradient term.

        rewards = result.logits  # [B, T, 1]

        log_probs = torch.log_softmax(result.hidden_states, dim=-1)  # policy logits

        loss = - (rewards * log_probs).mean()   # maximise expected reward

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

```

> **Note** – The repository does not ship a full GRPO trainer; the snippet shows how the *reward head* can be plugged into any policy‑gradient style optimiser.

## Key Files for GRPO Implementation

- **[`fish_speech/models/text2semantic/llama.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py)** – Defines `BaseModelArgs` (including `is_reward_model`), the main transformer (`BaseTransformer`), and the conditional switch to `self.score_output` for reward generation.
- **[`docs/en/index.md`](https://github.com/fishaudio/fish-speech/blob/main/docs/en/index.md)** – Human‑readable description of GRPO, its motivation, and the composition of the reward signal.
- **[`fish_speech/models/text2semantic/lora.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/lora.py)** – Shows how LoRA adapters can be attached to the model; useful if GRPO fine‑tuning is performed with parameter‑efficient updates.
- **[`fish_speech/configs/text2semantic_finetune.yaml`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/configs/text2semantic_finetune.yaml)** – Example configuration for finetuning the text‑semantic model; can be extended with `is_reward_model: true` to enable RL alignment.

## Summary

- **GRPO eliminates distribution mismatch** by reusing the same transformer that filtered training data as the reward model for reinforcement learning.
- **Architecture toggle** – Setting `is_reward_model=True` in `BaseModelArgs` switches the forward pass from token generation to scalar reward computation via `self.score_output`.
- **Multi-objective reward** – The reward signal combines semantic accuracy, instruction adherence, acoustic preference scoring, and timbre similarity.
- **Implementation path** – The core logic resides in [`fish_speech/models/text2semantic/llama.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py), with configuration support in the YAML configs and documentation in [`docs/en/index.md`](https://github.com/fishaudio/fish-speech/blob/main/docs/en/index.md).

## Frequently Asked Questions

### What makes GRPO different from standard PPO in speech synthesis?

**GRPO updates the policy relative to groups of tokens** (such as entire utterances) rather than individual token rewards, which stabilizes training for hierarchical autoregressive models. Unlike standard PPO, which often employs a separately trained reward model, Fish-Speech’s GRPO implementation **reuses the identical checkpoint** that filtered the pre-training data, eliminating distribution shift between the reward model and the policy.

### How does Fish-Speech avoid reward hacking with GRPO?

Fish-Speech mitigates reward hacking by ensuring the reward model is **architecturally identical** to the data-filtering model used during pre-training. Because the reward head (`self.score_output`) operates on the same hidden states and sees the same data distribution as the original filter, it cannot exploit arbitrary gaps between a separately trained reward model and the policy network.

### Can I use GRPO alignment with LoRA adapters?

Yes. The repository includes [`fish_speech/models/text2semantic/lora.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/lora.py), which demonstrates how to attach **Low-Rank Adaptation (LoRA)** layers to the base transformer. You can enable `is_reward_model=True` while using LoRA to perform parameter-efficient GRPO fine-tuning, updating only the adapter weights and the score head during reinforcement learning.

### Where is the reward score calculated in the forward pass?

The reward score is calculated in `BaseTransformer.forward_generate` within [`fish_speech/models/text2semantic/llama.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py) (lines 452-456). When `self.config.is_reward_model` is `True`, the method routes hidden states through `self.score_output` instead of the standard language-model head, producing scalar rewards used for the GRPO loss calculation.