# How to Configure LoRA Fine-Tuning for the Text2Semantic Model in Fish-Speech

> Configure LoRA fine-tuning for Fish-Speech text2semantic models using Hydra. Easily add low-rank adapters to transformer embeddings and linear projections for parameter-efficient training.

- Repository: [Fish Audio/fish-speech](https://github.com/fishaudio/fish-speech)
- Tags: how-to-guide
- Published: 2026-03-12

---

**Enable parameter-efficient fine-tuning on Fish-Speech's text2semantic LLAMA backbone by supplying a LoRA configuration through Hydra, which automatically wraps transformer embeddings and linear projections with low-rank adapters during training.**

The fishaudio/fish-speech repository provides a lightweight LoRA implementation designed specifically for the text2semantic model. This approach allows you to fine-tune the LLAMA-based transformer with minimal GPU memory overhead by updating only low-rank adapter weights instead of the full parameter set.

## Understanding the LoRA Architecture

The implementation resides in [`fish_speech/models/text2semantic/lora.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/lora.py), where the `LoraConfig` dataclass and `setup_lora` utility manage adapter injection. When activated, the system walks the `BaseTransformer` architecture and replaces standard `nn.Embedding` and linear layers with LoRA-wrapped equivalents that learn task-specific deltas while freezing base weights.

## Configuring LoRA for Fine-Tuning

### Select a LoRA Configuration File

Begin by choosing or creating a YAML configuration that specifies the rank `r`, scaling factor `lora_alpha`, and dropout rate. The repository includes a starter template at [`fish_speech/configs/lora/r_8_alpha_16.yaml`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/configs/lora/r_8_alpha_16.yaml), which configures adapters with `r=8` and `lora_alpha=16` alongside a dropout probability of 0.01.

### Inject LoRA Layers into the LLAMA Model

During model instantiation in [`fish_speech/models/text2semantic/llama.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py) (around line 589), the `BaseTransformer` checks for a `lora_config` parameter. If present, it invokes `setup_lora` from [`lora.py`](https://github.com/fishaudio/fish-speech/blob/main/lora.py) to recursively replace embedding layers via `_replace_embedding` and all linear projections with `lora.Linear` modules. This injection occurs before training begins, ensuring only adapter parameters receive gradient updates.

### Activate LoRA via Hydra CLI Override

Pass the configuration to the training script using Hydra's override syntax. The model receives the `LoraConfig` object during `from_pretrained`, triggering the architectural modifications described above.

## Training Command Example

Execute the fine-tuning run with the following command, replacing `my_project` with your experiment name:

```bash
python fish_speech/train.py --config-name text2semantic_finetune \
    project=my_project \
    +lora@model.model.lora_config=r_8_alpha_16

```

The `+lora@model.model.lora_config=r_8_alpha_16` argument instructs Hydra to load [`fish_speech/configs/lora/r_8_alpha_16.yaml`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/configs/lora/r_8_alpha_16.yaml), instantiate a `LoraConfig`, and inject it into the model's configuration path. This single flag activates the entire LoRA pipeline without modifying training loop code.

## Merging LoRA Weights for Inference

After fine-tuning, checkpoints contain only the adapter weights. To generate a standard model file for inference, you must merge the LoRA parameters back into the base LLAMA weights.

### Using the Merge Script

Run the utility script located at [`tools/llama/merge_lora.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/llama/merge_lora.py) to combine the base model with your trained adapters:

```bash
python tools/llama/merge_lora.py \
    --lora-config r_8_alpha_16 \
    --base-weight checkpoints/openaudio-s1-mini \
    --lora-weight results/my_project/checkpoints/step_000000010.ckpt \
    --output checkpoints/openaudio-s1-mini-merged

```

This script loads both checkpoints, concatenates the state dictionaries using `get_merged_state_dict`, and calls `save_pretrained` with `drop_lora=True` to produce a consolidated checkpoint without adapter layers. The script validates that the merged parameters differ from the original base model to confirm successful integration.

## Summary

- Store LoRA hyperparameters in YAML files under `fish_speech/configs/lora/`, such as [`r_8_alpha_16.yaml`](https://github.com/fishaudio/fish-speech/blob/main/r_8_alpha_16.yaml) with `r=8` and `lora_alpha=16`.
- LoRA injection occurs automatically via `setup_lora` in [`fish_speech/models/text2semantic/lora.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/lora.py) when a config is supplied to `BaseTransformer`.
- Activate fine-tuning with the Hydra override `+lora@model.model.lora_config=r_8_alpha_16` in your training command.
- Post-training, use [`tools/llama/merge_lora.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/llama/merge_lora.py) to bake adapters into the base weights for inference-compatible checkpoints.

## Frequently Asked Questions

### What rank and alpha values should I use for LoRA fine-tuning?

Start with the repository defaults of `r=8` and `lora_alpha=16` as defined in [`r_8_alpha_16.yaml`](https://github.com/fishaudio/fish-speech/blob/main/r_8_alpha_16.yaml). Higher ranks increase expressiveness but require more memory and computation, while the alpha value controls the scaling of adapter outputs relative to base activations. Adjust these based on your dataset size and overfitting observations.

### How does the LoRA injection work in the text2semantic model?

The `setup_lora` function in [`fish_speech/models/text2semantic/lora.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/lora.py) traverses the transformer hierarchy, replacing `nn.Embedding` layers via `_replace_embedding` and all linear projections with `lora.Linear` wrappers. This modification happens during `BaseTransformer.from_pretrained` in [`llama.py`](https://github.com/fishaudio/fish-speech/blob/main/llama.py) only when a `lora_config` is provided, freezing base weights and enabling gradient flow exclusively through the low-rank matrices.

### Can I merge LoRA weights back into the base model?

Yes. After fine-tuning, run [`tools/llama/merge_lora.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/llama/merge_lora.py) with paths to your base weights and LoRA checkpoint. The script concatenates the adapter deltas into the original parameters using `get_merged_state_dict`, then saves a merged checkpoint with `drop_lora=True` to remove adapter infrastructure, yielding a standard checkpoint suitable for standard inference pipelines.

### Where are the LoRA configuration files located?

The repository stores example configurations in `fish_speech/configs/lora/`. The default [`r_8_alpha_16.yaml`](https://github.com/fishaudio/fish-speech/blob/main/r_8_alpha_16.yaml) serves as the reference template, defining the `LoraConfig` structure consumed by the model initialization code. You can create additional YAML files in this directory to experiment with different ranks, alpha values, or dropout rates.