How to Configure LoRA Fine-Tuning for the Text2Semantic Model in Fish-Speech

Enable parameter-efficient fine-tuning on Fish-Speech's text2semantic LLAMA backbone by supplying a LoRA configuration through Hydra, which automatically wraps transformer embeddings and linear projections with low-rank adapters during training.

The fishaudio/fish-speech repository provides a lightweight LoRA implementation designed specifically for the text2semantic model. This approach allows you to fine-tune the LLAMA-based transformer with minimal GPU memory overhead by updating only low-rank adapter weights instead of the full parameter set.

Understanding the LoRA Architecture

The implementation resides in fish_speech/models/text2semantic/lora.py, where the LoraConfig dataclass and setup_lora utility manage adapter injection. When activated, the system walks the BaseTransformer architecture and replaces standard nn.Embedding and linear layers with LoRA-wrapped equivalents that learn task-specific deltas while freezing base weights.

Configuring LoRA for Fine-Tuning

Select a LoRA Configuration File

Begin by choosing or creating a YAML configuration that specifies the rank r, scaling factor lora_alpha, and dropout rate. The repository includes a starter template at fish_speech/configs/lora/r_8_alpha_16.yaml, which configures adapters with r=8 and lora_alpha=16 alongside a dropout probability of 0.01.

Inject LoRA Layers into the LLAMA Model

During model instantiation in fish_speech/models/text2semantic/llama.py (around line 589), the BaseTransformer checks for a lora_config parameter. If present, it invokes setup_lora from lora.py to recursively replace embedding layers via _replace_embedding and all linear projections with lora.Linear modules. This injection occurs before training begins, ensuring only adapter parameters receive gradient updates.

Activate LoRA via Hydra CLI Override

Pass the configuration to the training script using Hydra's override syntax. The model receives the LoraConfig object during from_pretrained, triggering the architectural modifications described above.

Training Command Example

Execute the fine-tuning run with the following command, replacing my_project with your experiment name:

python fish_speech/train.py --config-name text2semantic_finetune \
    project=my_project \
    +lora@model.model.lora_config=r_8_alpha_16

The +lora@model.model.lora_config=r_8_alpha_16 argument instructs Hydra to load fish_speech/configs/lora/r_8_alpha_16.yaml, instantiate a LoraConfig, and inject it into the model's configuration path. This single flag activates the entire LoRA pipeline without modifying training loop code.

Merging LoRA Weights for Inference

After fine-tuning, checkpoints contain only the adapter weights. To generate a standard model file for inference, you must merge the LoRA parameters back into the base LLAMA weights.

Using the Merge Script

Run the utility script located at tools/llama/merge_lora.py to combine the base model with your trained adapters:

python tools/llama/merge_lora.py \
    --lora-config r_8_alpha_16 \
    --base-weight checkpoints/openaudio-s1-mini \
    --lora-weight results/my_project/checkpoints/step_000000010.ckpt \
    --output checkpoints/openaudio-s1-mini-merged

This script loads both checkpoints, concatenates the state dictionaries using get_merged_state_dict, and calls save_pretrained with drop_lora=True to produce a consolidated checkpoint without adapter layers. The script validates that the merged parameters differ from the original base model to confirm successful integration.

Summary

• Store LoRA hyperparameters in YAML files under fish_speech/configs/lora/, such as r_8_alpha_16.yaml with r=8 and lora_alpha=16.
• LoRA injection occurs automatically via setup_lora in fish_speech/models/text2semantic/lora.py when a config is supplied to BaseTransformer.
• Activate fine-tuning with the Hydra override +lora@model.model.lora_config=r_8_alpha_16 in your training command.
• Post-training, use tools/llama/merge_lora.py to bake adapters into the base weights for inference-compatible checkpoints.

Frequently Asked Questions

What rank and alpha values should I use for LoRA fine-tuning?

Start with the repository defaults of r=8 and lora_alpha=16 as defined in r_8_alpha_16.yaml. Higher ranks increase expressiveness but require more memory and computation, while the alpha value controls the scaling of adapter outputs relative to base activations. Adjust these based on your dataset size and overfitting observations.

How does the LoRA injection work in the text2semantic model?

The setup_lora function in fish_speech/models/text2semantic/lora.py traverses the transformer hierarchy, replacing nn.Embedding layers via _replace_embedding and all linear projections with lora.Linear wrappers. This modification happens during BaseTransformer.from_pretrained in llama.py only when a lora_config is provided, freezing base weights and enabling gradient flow exclusively through the low-rank matrices.

Can I merge LoRA weights back into the base model?

Yes. After fine-tuning, run tools/llama/merge_lora.py with paths to your base weights and LoRA checkpoint. The script concatenates the adapter deltas into the original parameters using get_merged_state_dict, then saves a merged checkpoint with drop_lora=True to remove adapter infrastructure, yielding a standard checkpoint suitable for standard inference pipelines.

Where are the LoRA configuration files located?

The repository stores example configurations in fish_speech/configs/lora/. The default r_8_alpha_16.yaml serves as the reference template, defining the LoraConfig structure consumed by the model initialization code. You can create additional YAML files in this directory to experiment with different ranks, alpha values, or dropout rates.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}