How to Implement Voice Cloning with Reference Audio in Fish-Speech

Fish-Speech implements voice cloning by encoding reference audio into discrete acoustic tokens via the DAC encoder, caching them in ReferenceLoader, and injecting them as in-context examples into the LLAMA-based TTS model during inference.

Fish-Speech is an open-source text-to-speech (TTS) system that supports zero-shot voice cloning through in-context learning. To implement voice cloning with reference audio, you supply one or more short audio clips (5–10 seconds) that the model uses to extract speaker characteristics before generating new speech.

Architecture of Voice Cloning in Fish-Speech

The cloning pipeline consists of three core components that work sequentially during inference.

Reference Loading and Caching

The ReferenceLoader class in fish_speech/inference_engine/reference_loader.py manages the encoding and caching of reference audio.

# fish_speech/inference_engine/reference_loader.py

class ReferenceLoader:
    def __init__(self):
        self.ref_by_id: dict = {}      # cache keyed by user-provided ID

        self.ref_by_hash: dict = {}    # cache keyed by SHA-256 of audio bytes

The loader provides two primary methods:

• load_by_id(id, use_cache) – Reads all audio files under references/<id>/, encodes them via the DAC encoder, and stores the resulting tokens in self.ref_by_id.
• load_by_hash(references, use_cache) – Processes inline reference data passed directly in the API request. It computes SHA-256 hashes of the raw audio bytes to enable caching without filesystem I/O.

Both methods return a tuple of (prompt_tokens, prompt_texts) that is compatible with the downstream LLAMA request.

TTS Inference Loop

The TTSInferenceEngine class in fish_speech/inference_engine/__init__.py orchestrates the cloning process.

# fish_speech/inference_engine/__init__.py

class TTSInferenceEngine(ReferenceLoader, VQManager):
    @torch.inference_mode()
    def inference(self, req: ServeTTSRequest):
        # 1️⃣ Load reference tokens

        if req.reference_id:
            prompt_tokens, prompt_texts = self.load_by_id(
                req.reference_id, req.use_memory_cache
            )
        elif req.references:
            prompt_tokens, prompt_texts = self.load_by_hash(
                req.references, req.use_memory_cache
            )

        # 2️⃣ Send request to LLAMA with reference context

        response_queue = self.send_Llama_request(req, prompt_tokens, prompt_texts)

        # 3️⃣ Decode VQ tokens to audio

        while True:
            wrapped_result = response_queue.get()
            # ... handle response

            segment = self.decode_vq_tokens(wrapped_result.response.codes)

The reference tokens are merged with the prompt inside send_Llama_request, conditioning the model to reproduce the speaker characteristics from the reference audio.

How to Add Reference Audio

Method 1: REST API Endpoint

The server exposes /v1/references/add in tools/server/views.py for persistent reference storage.

Parameter	Description
id	Human-readable identifier (alphanumerics, hyphens, underscores; max 255 chars)
audio	Audio file (WAV, MP3, etc. supported by torchaudio)
text	Transcript of the audio for alignment

import requests

API_URL = "http://localhost:8000/v1/references/add"
files = {"audio": open("my_voice.wav", "rb")}
data = {
    "id": "my_favorite_voice",
    "text": "Hello, this is my reference voice."
}
resp = requests.post(API_URL, files=files, data=data)
print(resp.json())

The endpoint stores audio under ./references/<id>/sample.<ext> and creates sample.lab containing the transcript.

Method 2: Direct Filesystem Access

For scripting without HTTP overhead:

from fish_speech.inference_engine.reference_loader import ReferenceLoader

loader = ReferenceLoader()
loader.add_reference(
    id="my_voice",
    wav_file_path="path/to/my_voice.wav",
    reference_text="This is the voice I want to clone."
)

How to Use Reference Audio for Voice Cloning

Option 1: Web UI (Gradio)

The Gradio interface in tools/webui/inference.py provides three relevant fields:

• Reference ID – Select a pre-saved reference folder
• Reference Audio – Upload a temporary clip for one-time use
• Reference Text – Optional transcript for the uploaded clip

When audio is uploaded, inference_wrapper calls get_reference_audio() (lines 58-66 in tools/webui/inference.py) to build a ServeReferenceAudio object:

# tools/webui/inference.py

def inference_wrapper(..., reference_audio, reference_text, ...):
    if reference_audio:
        references = get_reference_audio(reference_audio, reference_text)
    else:
        references = []

    req = ServeTTSRequest(
        text=text,
        reference_id=reference_id if reference_id else None,
        references=references,
        ...
    )

Option 2: API with Stored Reference ID

curl -X POST http://localhost:8000/v1/tts \
  -H "Content-Type: application/json" \
  -d '{
        "text": "Please read this sentence in my voice.",
        "reference_id": "my_favorite_voice",
        "format": "wav",
        "streaming": false,
        "max_new_tokens": 1024,
        "top_p": 0.8,
        "repetition_penalty": 1.1,
        "temperature": 0.8
      }' --output output.wav

The server retrieves the cached encoding from ref_by_id, avoiding redundant DAC encoding on subsequent requests.

Option 3: Inline Reference Without Persistence

import base64
import requests

API_TTS = "http://localhost:8000/v1/tts"

with open("ref.wav", "rb") as f:
    ref_b64 = base64.b64encode(f.read()).decode()

payload = {
    "text": "Clone my voice for this sentence.",
    "references": [
        {"audio": ref_b64, "text": "Short reference utterance."}
    ],
    "format": "wav",
    "use_memory_cache": "on"
}

resp = requests.post(API_TTS, json=payload)
with open("cloned.wav", "wb") as f:
    f.write(resp.content)

The load_by_hash method computes SHA-256 hashes of the audio bytes, enabling cache hits when identical reference audio is submitted again.

Key Source Files for Voice Cloning

File	Role	Direct Link
fish_speech/inference_engine/reference_loader.py	Loads, caches, and manages reference audio encodings	reference_loader.py
fish_speech/inference_engine/__init__.py	Core TTS inference engine that orchestrates reference injection	init.py
fish_speech/utils/schema.py	Pydantic models including ServeReferenceAudio and ServeTTSRequest	schema.py
tools/webui/inference.py	Gradio interface wrapper for reference audio handling	inference.py
tools/server/views.py	HTTP endpoints for reference management and TTS generation	views.py

Summary

• Voice cloning in Fish-Speech uses in-context learning by feeding reference audio tokens into the LLAMA-based TTS model.
• ReferenceLoader (fish_speech/inference_engine/reference_loader.py) handles encoding via the DAC encoder and maintains two cache layers: ref_by_id for filesystem references and ref_by_hash for inline audio data.
• Two workflows exist: persistent references (uploaded via /v1/references/add and referenced by ID) and one-off cloning (inline base64 audio in the TTS request).
• Integration points include the Gradio Web UI (tools/webui/inference.py) and the REST API (tools/server/views.py), both utilizing the core TTSInferenceEngine class.

Frequently Asked Questions

How long should the reference audio be for optimal voice cloning?

Use 5–10 seconds of clear, single-speaker speech. The ReferenceLoader encodes this audio into discrete tokens via the DAC encoder, and the LLAMA model uses these tokens as in-context examples. Audio shorter than 3 seconds may not capture sufficient speaker characteristics, while very long clips increase encoding time without proportional quality gains.

What is the difference between reference_id and inline references in the API?

reference_id refers to a pre-saved reference folder under ./references/<id>/ that was created via the /v1/references/add endpoint. The server caches these encodings in ref_by_id for reuse across multiple requests. Inline references are base64-encoded audio bytes sent directly in the JSON payload using the ServeReferenceAudio schema; these are processed by load_by_hash and cached by SHA-256 hash, making them ideal for one-off cloning without filesystem persistence.

Can I use multiple reference audio clips for a single voice cloning request?

Yes, both reference_id and inline references support multiple clips. When using reference_id, place multiple audio files in the ./references/<id>/ directory; load_by_id will encode all of them and concatenate their tokens. For inline mode, pass a JSON array with multiple ServeReferenceAudio objects. The model treats these as multiple in-context examples, potentially improving speaker similarity.

How does Fish-Speech handle caching to improve performance?

Fish-Speech implements a two-tier caching strategy in ReferenceLoader. The ref_by_id dictionary caches encodings for filesystem-based references keyed by the user-provided ID, while ref_by_hash caches inline references keyed by SHA-256 hashes of the raw audio bytes. When use_memory_cache is enabled (the default), subsequent requests with identical audio bypass the expensive DAC encoding step and reuse the cached tokens directly, reducing latency significantly for repeated cloning of the same voice.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}