How to Implement Custom STT or TTS Backends in the Speech-to-Speech Pipeline

You implement custom STT or TTS backends by creating a handler class that inherits from BaseSTTHandler or BaseHandler, defining a configuration dataclass, and registering a BackendSpec in speech_to_speech/backend_registry.py that maps CLI arguments to your factory function.

The huggingface/speech-to-speech library builds its real-time voice pipeline from modular backend specifications that describe how concrete STT and TTS components are instantiated. To implement custom STT or TTS backends, you register new backend specifications that the CLI automatically discovers, making your components selectable via --stt or --tts flags without modifying core pipeline logic.

Understanding the Backend Architecture

The backend registry system lives in speech_to_speech/backend_registry.py. A backend is represented by a BackendSpec dataclass that encapsulates all metadata required to instantiate a handler:

BackendSpec(
    name: str,                # identifier used on the CLI (e.g. "--stt my-stt")

    kind: BackendKind,        # "stt", "tts", or "llm"

    config_type: type,        # a dataclass that holds the backend's arguments

    create_handler: Callable, # factory receiving HandlerContext and normalized config

    config_prefix: str | None = None,
    normalize_config: Callable | None = None,
    required_extra: str | None = None,
    capabilities: BackendCapabilities = BackendCapabilities(),
)

When the CLI parses --stt <name> or --tts <name>, it looks up the requested identifier in the appropriate global registry (STT_BACKENDS or TTS_BACKENDS). The selected BackendSpec is normalized—converting the argument dataclass into a plain dict—and passed to the factory function, which constructs the concrete handler.

Step-by-Step Implementation Guide

To add a custom backend to the speech-to-speech pipeline, follow these four steps:

Create the Arguments Dataclass

Define a dataclass that inherits from dataclasses.dataclass to expose configuration options via CLI flags. This class can be empty if your backend requires no special options.

from dataclasses import dataclass

@dataclass
class MySTTHandlerArguments:
    model_path: str = "default-model"
    language: str = "en"

Implement the Handler Class

Create a handler that inherits from BaseSTTHandler (for speech-to-text) or BaseHandler (for text-to-speech). The handler must provide a setup(...) method for model initialization and a processing method that yields appropriate output messages.

For STT handlers, implement process(self, vad_audio) -> Iterator[Transcription]. For TTS handlers, implement process(self, tts_in) -> TTSOut or __call__.

Register the Backend

Add a BackendSpec entry to the global registry in backend_registry.py. Use the helper _simple_handler_factory if your handler only needs the config dict; otherwise provide a custom factory function.

Handle Optional Dependencies

Declare external dependencies via the required_extra field. This enables the library to raise a helpful ImportError if a user selects your custom backend without installing the extra package (e.g., pip install "speech-to-speech[my-extra]").

Practical Examples

Example 1: Custom Echo STT Backend

This minimal STT backend returns a fixed transcription regardless of audio input.

Create the handler in src/speech_to_speech/STT/echo_stt_handler.py:

from speech_to_speech.STT.base_stt_handler import BaseSTTHandler
from speech_to_speech.pipeline.messages import Transcription
from typing import Iterator

class EchoSTTHandler(BaseSTTHandler):
    """Returns a fixed transcription regardless of the audio input."""
    def setup(self, *, echo_text: str = "hello world") -> None:
        self.echo_text = echo_text

    def process(self, vad_audio) -> Iterator[Transcription]:
        yield Transcription(
            text=self.echo_text,
            language_code="en",
            turn_id=vad_audio.turn_id,
            turn_revision=vad_audio.turn_revision,
            speech_stopped_at_s=vad_audio.created_at_s,
        )

Create the arguments dataclass in src/speech_to_speech/arguments_classes/echo_stt_arguments.py:

from dataclasses import dataclass

@dataclass
class EchoSTTHandlerArguments:
    echo_text: str = "hello world"

Register the backend in src/speech_to_speech/backend_registry.py:

BackendSpec(
    name="echo",
    kind="stt",
    config_type=EchoSTTHandlerArguments,
    create_handler=_simple_handler_factory(
        "speech_to_speech.STT.echo_stt_handler",
        "EchoSTTHandler",
        attach_speculative_turns=True,
    ),
    config_prefix="stt",
)

Usage:

python -m speech_to_speech.cli serve \
    --stt echo \
    --stt.echo_text "custom greeting"

Example 2: Custom Silence TTS Backend

This TTS backend generates a short silent audio buffer for any input text.

Create the handler in src/speech_to_speech/TTS/silence_tts_handler.py:

import numpy as np
from speech_to_speech.baseHandler import BaseHandler
from speech_to_speech.pipeline.handler_types import TTSIn, TTSOut

class SilenceTTSHandler(BaseHandler[TTSIn, TTSOut]):
    """Generates a short silent audio buffer for any input text."""
    def setup(self, *, sample_rate: int = 16000, duration_s: float = 0.5) -> None:
        self.sample_rate = sample_rate
        self.num_samples = int(sample_rate * duration_s)

    def process(self, tts_in) -> TTSOut:
        silent_audio = np.zeros(self.num_samples, dtype=np.float32)
        yield silent_audio

Create the arguments dataclass in src/speech_to_speech/arguments_classes/silence_tts_arguments.py:

from dataclasses import dataclass

@dataclass
class SilenceTTSHandlerArguments:
    sample_rate: int = 16000
    duration_s: float = 0.5

Register the backend in src/speech_to_speech/backend_registry.py:

BackendSpec(
    name="silence",
    kind="tts",
    config_type=SilenceTTSHandlerArguments,
    create_handler=_simple_handler_factory(
        "speech_to_speech.TTS.silence_tts_handler",
        "SilenceTTSHandler",
        setup_should_listen=True,
        context_kwargs=True,
    ),
    config_prefix="tts",
)

Usage:

python -m speech_to_speech.cli serve \
    --tts silence \
    --tts.duration_s 1.0

Declaring Optional Dependencies

If your backend requires third-party libraries, specify them in the required_extra field:

BackendSpec(
    name="gtts",
    kind="tts",
    config_type=GTTSTTSHandlerArguments,
    create_handler=_simple_handler_factory(...),
    required_extra="gtts",
)

Users must install with pip install "speech-to-speech[gtts]" to use this backend.

Key Source Files and Architecture

Understanding the following files is essential when implementing custom STT or TTS backends:

• src/speech_to_speech/backend_registry.py: Contains the BackendSpec definitions and builds the STT_BACKENDS, TTS_BACKENDS, and LLM_BACKENDS registries.
• src/speech_to_speech/arguments_classes/*.py: Dataclasses that define CLI-exposed configuration options for each backend.
• src/speech_to_speech/STT/*.py: Base class BaseSTTHandler and concrete STT implementations (e.g., whisper_stt_handler.py).
• src/speech_to_speech/TTS/*.py: Base class BaseHandler and concrete TTS implementations (e.g., pocket_tts_handler.py).
• src/speech_to_speech/s2s_pipeline.py: Parses CLI arguments, selects backends via select_backend, and wires handlers together.
• src/speech_to_speech/cli.py: Entry point that forwards CLI flags such as --stt and --tts to the pipeline.

Summary

• Backend specifications drive the speech-to-speech pipeline through the BackendSpec dataclass in backend_registry.py.
• Handler classes must inherit from BaseSTTHandler (for STT) or BaseHandler (for TTS) and implement setup() plus the appropriate processing method.
• Configuration dataclasses expose parameters to the CLI via the config_prefix mechanism.
• Registration occurs by appending a BackendSpec to STT_BACKENDS or TTS_BACKENDS, optionally using _simple_handler_factory for standard instantiation patterns.
• Optional dependencies are declared via required_extra to ensure users install necessary packages before using custom backends.

Frequently Asked Questions

What base class should I use for a custom STT backend?

You must inherit from BaseSTTHandler located in the speech_to_speech.STT module. Your subclass must implement the setup(**kwargs) method for initialization and the process(vad_audio) method that yields Transcription objects. The process method receives voice activity detection (VAD) audio segments and must return an iterator of transcription messages containing the recognized text and metadata.

How do I expose configuration options for my custom backend to the CLI?

Define a dataclass decorated with @dataclass that holds your configuration fields. Pass this class as the config_type parameter when creating your BackendSpec. Set the config_prefix parameter to "stt" or "tts" to namespace the CLI flags (e.g., --stt.model_path or --tts.sample_rate). The pipeline automatically converts these flags into a normalized dictionary passed to your handler's setup method.

Can I use external libraries like Google TTS or Azure Speech in my custom backend?

Yes. Implement your handler to import and use any third-party libraries. Declare the dependency in the required_extra field of your BackendSpec (e.g., required_extra="azure"). When users attempt to use your backend without installing the corresponding extra, the framework raises an informative ImportError suggesting the correct pip install command, such as pip install "speech-to-speech[azure]".

Where does the pipeline instantiate my custom backend?

The pipeline instantiates your backend in src/speech_to_speech/s2s_pipeline.py through the select_backend function. This function looks up your backend's name in the global registries defined in backend_registry.py, normalizes the configuration using your dataclass, and calls your create_handler factory function with a HandlerContext and the configuration dictionary.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}