# How to Implement Voice Cloning with Reference Audio in Fish-Speech

> Implement voice cloning with reference audio using Fish-Speech. Learn how Fish-Speech encodes audio into tokens and injects them into a TTS model for realistic voice generation.

- Repository: [Fish Audio/fish-speech](https://github.com/fishaudio/fish-speech)
- Tags: how-to-guide
- Published: 2026-03-12

---

**Fish-Speech implements voice cloning by encoding reference audio into discrete acoustic tokens via the DAC encoder, caching them in `ReferenceLoader`, and injecting them as in-context examples into the LLAMA-based TTS model during inference.**

Fish-Speech is an open-source text-to-speech (TTS) system that supports zero-shot voice cloning through in-context learning. To implement voice cloning with reference audio, you supply one or more short audio clips (5–10 seconds) that the model uses to extract speaker characteristics before generating new speech.

## Architecture of Voice Cloning in Fish-Speech

The cloning pipeline consists of three core components that work sequentially during inference.

### Reference Loading and Caching

The `ReferenceLoader` class in [`fish_speech/inference_engine/reference_loader.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/inference_engine/reference_loader.py) manages the encoding and caching of reference audio.

```python

# fish_speech/inference_engine/reference_loader.py

class ReferenceLoader:
    def __init__(self):
        self.ref_by_id: dict = {}      # cache keyed by user-provided ID

        self.ref_by_hash: dict = {}    # cache keyed by SHA-256 of audio bytes

```

The loader provides two primary methods:

- **`load_by_id(id, use_cache)`** – Reads all audio files under `references/<id>/`, encodes them via the DAC encoder, and stores the resulting tokens in `self.ref_by_id`.
- **`load_by_hash(references, use_cache)`** – Processes inline reference data passed directly in the API request. It computes SHA-256 hashes of the raw audio bytes to enable caching without filesystem I/O.

Both methods return a tuple of `(prompt_tokens, prompt_texts)` that is compatible with the downstream LLAMA request.

### TTS Inference Loop

The `TTSInferenceEngine` class in [`fish_speech/inference_engine/__init__.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/inference_engine/__init__.py) orchestrates the cloning process.

```python

# fish_speech/inference_engine/__init__.py

class TTSInferenceEngine(ReferenceLoader, VQManager):
    @torch.inference_mode()
    def inference(self, req: ServeTTSRequest):
        # 1️⃣ Load reference tokens

        if req.reference_id:
            prompt_tokens, prompt_texts = self.load_by_id(
                req.reference_id, req.use_memory_cache
            )
        elif req.references:
            prompt_tokens, prompt_texts = self.load_by_hash(
                req.references, req.use_memory_cache
            )

        # 2️⃣ Send request to LLAMA with reference context

        response_queue = self.send_Llama_request(req, prompt_tokens, prompt_texts)

        # 3️⃣ Decode VQ tokens to audio

        while True:
            wrapped_result = response_queue.get()
            # ... handle response

            segment = self.decode_vq_tokens(wrapped_result.response.codes)

```

The reference tokens are merged with the prompt inside `send_Llama_request`, conditioning the model to reproduce the speaker characteristics from the reference audio.

## How to Add Reference Audio

### Method 1: REST API Endpoint

The server exposes `/v1/references/add` in [`tools/server/views.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/server/views.py) for persistent reference storage.

| Parameter | Description |
|-----------|-------------|
| `id` | Human-readable identifier (alphanumerics, hyphens, underscores; max 255 chars) |
| `audio` | Audio file (WAV, MP3, etc. supported by torchaudio) |
| `text` | Transcript of the audio for alignment |

```python
import requests

API_URL = "http://localhost:8000/v1/references/add"
files = {"audio": open("my_voice.wav", "rb")}
data = {
    "id": "my_favorite_voice",
    "text": "Hello, this is my reference voice."
}
resp = requests.post(API_URL, files=files, data=data)
print(resp.json())

```

The endpoint stores audio under `./references/<id>/sample.<ext>` and creates `sample.lab` containing the transcript.

### Method 2: Direct Filesystem Access

For scripting without HTTP overhead:

```python
from fish_speech.inference_engine.reference_loader import ReferenceLoader

loader = ReferenceLoader()
loader.add_reference(
    id="my_voice",
    wav_file_path="path/to/my_voice.wav",
    reference_text="This is the voice I want to clone."
)

```

## How to Use Reference Audio for Voice Cloning

### Option 1: Web UI (Gradio)

The Gradio interface in [`tools/webui/inference.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/webui/inference.py) provides three relevant fields:

- **Reference ID** – Select a pre-saved reference folder
- **Reference Audio** – Upload a temporary clip for one-time use
- **Reference Text** – Optional transcript for the uploaded clip

When audio is uploaded, `inference_wrapper` calls `get_reference_audio()` (lines 58-66 in [`tools/webui/inference.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/webui/inference.py)) to build a `ServeReferenceAudio` object:

```python

# tools/webui/inference.py

def inference_wrapper(..., reference_audio, reference_text, ...):
    if reference_audio:
        references = get_reference_audio(reference_audio, reference_text)
    else:
        references = []

    req = ServeTTSRequest(
        text=text,
        reference_id=reference_id if reference_id else None,
        references=references,
        ...
    )

```

### Option 2: API with Stored Reference ID

```bash
curl -X POST http://localhost:8000/v1/tts \
  -H "Content-Type: application/json" \
  -d '{
        "text": "Please read this sentence in my voice.",
        "reference_id": "my_favorite_voice",
        "format": "wav",
        "streaming": false,
        "max_new_tokens": 1024,
        "top_p": 0.8,
        "repetition_penalty": 1.1,
        "temperature": 0.8
      }' --output output.wav

```

The server retrieves the cached encoding from `ref_by_id`, avoiding redundant DAC encoding on subsequent requests.

### Option 3: Inline Reference Without Persistence

```python
import base64
import requests

API_TTS = "http://localhost:8000/v1/tts"

with open("ref.wav", "rb") as f:
    ref_b64 = base64.b64encode(f.read()).decode()

payload = {
    "text": "Clone my voice for this sentence.",
    "references": [
        {"audio": ref_b64, "text": "Short reference utterance."}
    ],
    "format": "wav",
    "use_memory_cache": "on"
}

resp = requests.post(API_TTS, json=payload)
with open("cloned.wav", "wb") as f:
    f.write(resp.content)

```

The `load_by_hash` method computes SHA-256 hashes of the audio bytes, enabling cache hits when identical reference audio is submitted again.

## Key Source Files for Voice Cloning

| File | Role | Direct Link |
|------|------|-------------|
| [`fish_speech/inference_engine/reference_loader.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/inference_engine/reference_loader.py) | Loads, caches, and manages reference audio encodings | [reference_loader.py](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/inference_engine/reference_loader.py) |
| [`fish_speech/inference_engine/__init__.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/inference_engine/__init__.py) | Core TTS inference engine that orchestrates reference injection | [__init__.py](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/inference_engine/__init__.py) |
| [`fish_speech/utils/schema.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/utils/schema.py) | Pydantic models including `ServeReferenceAudio` and `ServeTTSRequest` | [schema.py](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/utils/schema.py) |
| [`tools/webui/inference.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/webui/inference.py) | Gradio interface wrapper for reference audio handling | [inference.py](https://github.com/fishaudio/fish-speech/blob/main/tools/webui/inference.py) |
| [`tools/server/views.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/server/views.py) | HTTP endpoints for reference management and TTS generation | [views.py](https://github.com/fishaudio/fish-speech/blob/main/tools/server/views.py) |

## Summary

- **Voice cloning** in Fish-Speech uses in-context learning by feeding reference audio tokens into the LLAMA-based TTS model.
- **ReferenceLoader** ([`fish_speech/inference_engine/reference_loader.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/inference_engine/reference_loader.py)) handles encoding via the DAC encoder and maintains two cache layers: `ref_by_id` for filesystem references and `ref_by_hash` for inline audio data.
- **Two workflows** exist: persistent references (uploaded via `/v1/references/add` and referenced by ID) and one-off cloning (inline base64 audio in the TTS request).
- **Integration points** include the Gradio Web UI ([`tools/webui/inference.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/webui/inference.py)) and the REST API ([`tools/server/views.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/server/views.py)), both utilizing the core `TTSInferenceEngine` class.

## Frequently Asked Questions

### How long should the reference audio be for optimal voice cloning?

**Use 5–10 seconds of clear, single-speaker speech.** The `ReferenceLoader` encodes this audio into discrete tokens via the DAC encoder, and the LLAMA model uses these tokens as in-context examples. Audio shorter than 3 seconds may not capture sufficient speaker characteristics, while very long clips increase encoding time without proportional quality gains.

### What is the difference between `reference_id` and inline `references` in the API?

**`reference_id`** refers to a pre-saved reference folder under `./references/<id>/` that was created via the `/v1/references/add` endpoint. The server caches these encodings in `ref_by_id` for reuse across multiple requests. **Inline `references`** are base64-encoded audio bytes sent directly in the JSON payload using the `ServeReferenceAudio` schema; these are processed by `load_by_hash` and cached by SHA-256 hash, making them ideal for one-off cloning without filesystem persistence.

### Can I use multiple reference audio clips for a single voice cloning request?

**Yes, both `reference_id` and inline `references` support multiple clips.** When using `reference_id`, place multiple audio files in the `./references/<id>/` directory; `load_by_id` will encode all of them and concatenate their tokens. For inline mode, pass a JSON array with multiple `ServeReferenceAudio` objects. The model treats these as multiple in-context examples, potentially improving speaker similarity.

### How does Fish-Speech handle caching to improve performance?

**Fish-Speech implements a two-tier caching strategy in `ReferenceLoader`.** The `ref_by_id` dictionary caches encodings for filesystem-based references keyed by the user-provided ID, while `ref_by_hash` caches inline references keyed by SHA-256 hashes of the raw audio bytes. When `use_memory_cache` is enabled (the default), subsequent requests with identical audio bypass the expensive DAC encoding step and reuse the cached tokens directly, reducing latency significantly for repeated cloning of the same voice.