# How to Implement Custom STT or TTS Backends in the Speech-to-Speech Pipeline

> Learn to implement custom STT or TTS backends in the Hugging Face speech-to-speech pipeline. Discover how to create handler classes, define configurations, and register your custom backend for seamless integration.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-09

---

**You implement custom STT or TTS backends by creating a handler class that inherits from `BaseSTTHandler` or `BaseHandler`, defining a configuration dataclass, and registering a `BackendSpec` in [`speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/speech_to_speech/backend_registry.py) that maps CLI arguments to your factory function.**

The huggingface/speech-to-speech library builds its real-time voice pipeline from modular backend specifications that describe how concrete STT and TTS components are instantiated. To implement custom STT or TTS backends, you register new backend specifications that the CLI automatically discovers, making your components selectable via `--stt` or `--tts` flags without modifying core pipeline logic.

## Understanding the Backend Architecture

The backend registry system lives in [`speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/speech_to_speech/backend_registry.py). A backend is represented by a `BackendSpec` dataclass that encapsulates all metadata required to instantiate a handler:

```python
BackendSpec(
    name: str,                # identifier used on the CLI (e.g. "--stt my-stt")

    kind: BackendKind,        # "stt", "tts", or "llm"

    config_type: type,        # a dataclass that holds the backend's arguments

    create_handler: Callable, # factory receiving HandlerContext and normalized config

    config_prefix: str | None = None,
    normalize_config: Callable | None = None,
    required_extra: str | None = None,
    capabilities: BackendCapabilities = BackendCapabilities(),
)

```

When the CLI parses `--stt <name>` or `--tts <name>`, it looks up the requested identifier in the appropriate global registry (`STT_BACKENDS` or `TTS_BACKENDS`). The selected `BackendSpec` is normalized—converting the argument dataclass into a plain `dict`—and passed to the factory function, which constructs the concrete handler.

## Step-by-Step Implementation Guide

To add a custom backend to the speech-to-speech pipeline, follow these four steps:

### Create the Arguments Dataclass

Define a dataclass that inherits from `dataclasses.dataclass` to expose configuration options via CLI flags. This class can be empty if your backend requires no special options.

```python
from dataclasses import dataclass

@dataclass
class MySTTHandlerArguments:
    model_path: str = "default-model"
    language: str = "en"

```

### Implement the Handler Class

Create a handler that inherits from `BaseSTTHandler` (for speech-to-text) or `BaseHandler` (for text-to-speech). The handler must provide a `setup(...)` method for model initialization and a processing method that yields appropriate output messages.

For **STT handlers**, implement `process(self, vad_audio) -> Iterator[Transcription]`. For **TTS handlers**, implement `process(self, tts_in) -> TTSOut` or `__call__`.

### Register the Backend

Add a `BackendSpec` entry to the global registry in [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py). Use the helper `_simple_handler_factory` if your handler only needs the config dict; otherwise provide a custom factory function.

### Handle Optional Dependencies

Declare external dependencies via the `required_extra` field. This enables the library to raise a helpful `ImportError` if a user selects your custom backend without installing the extra package (e.g., `pip install "speech-to-speech[my-extra]"`).

## Practical Examples

### Example 1: Custom Echo STT Backend

This minimal STT backend returns a fixed transcription regardless of audio input.

Create the handler in [`src/speech_to_speech/STT/echo_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/echo_stt_handler.py):

```python
from speech_to_speech.STT.base_stt_handler import BaseSTTHandler
from speech_to_speech.pipeline.messages import Transcription
from typing import Iterator

class EchoSTTHandler(BaseSTTHandler):
    """Returns a fixed transcription regardless of the audio input."""
    def setup(self, *, echo_text: str = "hello world") -> None:
        self.echo_text = echo_text

    def process(self, vad_audio) -> Iterator[Transcription]:
        yield Transcription(
            text=self.echo_text,
            language_code="en",
            turn_id=vad_audio.turn_id,
            turn_revision=vad_audio.turn_revision,
            speech_stopped_at_s=vad_audio.created_at_s,
        )

```

Create the arguments dataclass in [`src/speech_to_speech/arguments_classes/echo_stt_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/echo_stt_arguments.py):

```python
from dataclasses import dataclass

@dataclass
class EchoSTTHandlerArguments:
    echo_text: str = "hello world"

```

Register the backend in [`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py):

```python
BackendSpec(
    name="echo",
    kind="stt",
    config_type=EchoSTTHandlerArguments,
    create_handler=_simple_handler_factory(
        "speech_to_speech.STT.echo_stt_handler",
        "EchoSTTHandler",
        attach_speculative_turns=True,
    ),
    config_prefix="stt",
)

```

Usage:

```bash
python -m speech_to_speech.cli serve \
    --stt echo \
    --stt.echo_text "custom greeting"

```

### Example 2: Custom Silence TTS Backend

This TTS backend generates a short silent audio buffer for any input text.

Create the handler in [`src/speech_to_speech/TTS/silence_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/silence_tts_handler.py):

```python
import numpy as np
from speech_to_speech.baseHandler import BaseHandler
from speech_to_speech.pipeline.handler_types import TTSIn, TTSOut

class SilenceTTSHandler(BaseHandler[TTSIn, TTSOut]):
    """Generates a short silent audio buffer for any input text."""
    def setup(self, *, sample_rate: int = 16000, duration_s: float = 0.5) -> None:
        self.sample_rate = sample_rate
        self.num_samples = int(sample_rate * duration_s)

    def process(self, tts_in) -> TTSOut:
        silent_audio = np.zeros(self.num_samples, dtype=np.float32)
        yield silent_audio

```

Create the arguments dataclass in [`src/speech_to_speech/arguments_classes/silence_tts_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/silence_tts_arguments.py):

```python
from dataclasses import dataclass

@dataclass
class SilenceTTSHandlerArguments:
    sample_rate: int = 16000
    duration_s: float = 0.5

```

Register the backend in [`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py):

```python
BackendSpec(
    name="silence",
    kind="tts",
    config_type=SilenceTTSHandlerArguments,
    create_handler=_simple_handler_factory(
        "speech_to_speech.TTS.silence_tts_handler",
        "SilenceTTSHandler",
        setup_should_listen=True,
        context_kwargs=True,
    ),
    config_prefix="tts",
)

```

Usage:

```bash
python -m speech_to_speech.cli serve \
    --tts silence \
    --tts.duration_s 1.0

```

### Declaring Optional Dependencies

If your backend requires third-party libraries, specify them in the `required_extra` field:

```python
BackendSpec(
    name="gtts",
    kind="tts",
    config_type=GTTSTTSHandlerArguments,
    create_handler=_simple_handler_factory(...),
    required_extra="gtts",
)

```

Users must install with `pip install "speech-to-speech[gtts]"` to use this backend.

## Key Source Files and Architecture

Understanding the following files is essential when implementing custom STT or TTS backends:

- **[`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py)**: Contains the `BackendSpec` definitions and builds the `STT_BACKENDS`, `TTS_BACKENDS`, and `LLM_BACKENDS` registries.
- **`src/speech_to_speech/arguments_classes/*.py`**: Dataclasses that define CLI-exposed configuration options for each backend.
- **`src/speech_to_speech/STT/*.py`**: Base class `BaseSTTHandler` and concrete STT implementations (e.g., [`whisper_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/whisper_stt_handler.py)).
- **`src/speech_to_speech/TTS/*.py`**: Base class `BaseHandler` and concrete TTS implementations (e.g., [`pocket_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/pocket_tts_handler.py)).
- **[`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py)**: Parses CLI arguments, selects backends via `select_backend`, and wires handlers together.
- **[`src/speech_to_speech/cli.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/cli.py)**: Entry point that forwards CLI flags such as `--stt` and `--tts` to the pipeline.

## Summary

- **Backend specifications** drive the speech-to-speech pipeline through the `BackendSpec` dataclass in [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py).
- **Handler classes** must inherit from `BaseSTTHandler` (for STT) or `BaseHandler` (for TTS) and implement `setup()` plus the appropriate processing method.
- **Configuration dataclasses** expose parameters to the CLI via the `config_prefix` mechanism.
- **Registration** occurs by appending a `BackendSpec` to `STT_BACKENDS` or `TTS_BACKENDS`, optionally using `_simple_handler_factory` for standard instantiation patterns.
- **Optional dependencies** are declared via `required_extra` to ensure users install necessary packages before using custom backends.

## Frequently Asked Questions

### What base class should I use for a custom STT backend?

You must inherit from `BaseSTTHandler` located in the `speech_to_speech.STT` module. Your subclass must implement the `setup(**kwargs)` method for initialization and the `process(vad_audio)` method that yields `Transcription` objects. The `process` method receives voice activity detection (VAD) audio segments and must return an iterator of transcription messages containing the recognized text and metadata.

### How do I expose configuration options for my custom backend to the CLI?

Define a dataclass decorated with `@dataclass` that holds your configuration fields. Pass this class as the `config_type` parameter when creating your `BackendSpec`. Set the `config_prefix` parameter to `"stt"` or `"tts"` to namespace the CLI flags (e.g., `--stt.model_path` or `--tts.sample_rate`). The pipeline automatically converts these flags into a normalized dictionary passed to your handler's `setup` method.

### Can I use external libraries like Google TTS or Azure Speech in my custom backend?

Yes. Implement your handler to import and use any third-party libraries. Declare the dependency in the `required_extra` field of your `BackendSpec` (e.g., `required_extra="azure"`). When users attempt to use your backend without installing the corresponding extra, the framework raises an informative `ImportError` suggesting the correct `pip install` command, such as `pip install "speech-to-speech[azure]"`.

### Where does the pipeline instantiate my custom backend?

The pipeline instantiates your backend in [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) through the `select_backend` function. This function looks up your backend's name in the global registries defined in [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py), normalizes the configuration using your dataclass, and calls your `create_handler` factory function with a `HandlerContext` and the configuration dictionary.