# How to Integrate a Custom STT Backend into the Speech-to-Speech Backend Registry

> Integrate a custom STT backend into the Speech-to-Speech backend registry. Learn how to create a BackendSpec entry to link your dataclass and handler factory.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-08

---

**Integrate a custom STT backend by creating a `BackendSpec` entry that links your arguments dataclass and handler factory to the `STT_BACKENDS` registry in [`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py).**

The huggingface/speech-to-speech library routes all speech-to-text processing through a static backend registry that maps configuration names to handler instances. To integrate a custom STT backend into the backend registry system, you extend the `STT_BACKENDS` list with a new specification that defines how to construct your implementation.

## Understanding the Backend Registry Architecture

The registry system centers on the `BackendSpec` frozen dataclass defined in [`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py). Each spec stores a backend’s **name**, **kind** (`"stt"`, `"tts"`, or `"llm"`), **arguments class**, **factory function**, and optional metadata including configuration prefixes and capability flags.

When the pipeline initializes, the library executes three phases:

1. **Registry Construction** – The module imports trigger `build_backend_registry(kind, specs)` (lines [49‑59]), creating immutable mappings for `STT_BACKENDS`, `LLM_BACKENDS`, and `TTS_BACKENDS`.
2. **Backend Selection** – The `select_backend(registry, name, config)` function (lines [62‑68]) validates the requested name and normalizes configuration into a plain dictionary.
3. **Handler Instantiation** – The `create_backend_handler(selection, context)` function (lines [82‑92]) invokes the factory stored in the spec, passing the normalized config and handling optional-dependency import errors.

## Step-by-Step Integration Guide

### Step 1: Define the Arguments Dataclass

Create a dataclass in `src/speech_to_speech/arguments_classes/` that inherits from `EmptyBackendArguments`. This class declares the configuration fields your backend requires.

```python

# src/speech_to_speech/arguments_classes/my_custom_stt_arguments.py

from dataclasses import dataclass
from speech_to_speech.backend_registry import EmptyBackendArguments

@dataclass
class MyCustomSTTHandlerArguments(EmptyBackendArguments):
    model_path: str = "models/my_custom.pt"
    language: str = "en"
    chunk_size: int = 16000

```

Follow the pattern established by `WhisperSTTHandlerArguments` (lines [18‑24]) to ensure compatibility with the CLI parsing system.

### Step 2: Implement the Handler Class

Write a handler class in `src/speech_to_speech/STT/` that accepts the standard constructor signature used throughout the library: `__init__(stop_event, queue_in, queue_out, *, setup_args=…, setup_kwargs=…)`.

```python

# src/speech_to_speech/STT/my_custom_stt_handler.py

from typing import Any
from queue import Queue
from threading import Event

class MyCustomSTTHandler:
    def __init__(
        self,
        stop_event: Event,
        queue_in: Queue[Any],
        queue_out: Queue[Any],
        *,
        setup_kwargs: dict[str, Any],
    ):
        self.stop_event = stop_event
        self.queue_in = queue_in
        self.queue_out = queue_out
        self.model_path = setup_kwargs["model_path"]
        self.language = setup_kwargs["language"]
        self.chunk_size = setup_kwargs["chunk_size"]
        # Load model, initialize resources...

    def process(self, audio_chunk):
        # Perform transcription logic

        transcribed_text = self._transcribe(audio_chunk)
        self.queue_out.put(transcribed_text)

    def close(self):
        # Cleanup resources

        pass

```

Reference `WhisperSTTHandler` in [`src/speech_to_speech/STT/whisper_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/whisper_stt_handler.py) for the complete implementation pattern.

### Step 3: Create the Factory Function

Use `_simple_handler_factory` (lines [104‑131]) if your handler can be constructed directly from the normalized configuration dictionary. This factory returns a `HandlerFactory` that imports and instantiates your class.

```python
from speech_to_speech.backend_registry import _simple_handler_factory

factory = _simple_handler_factory(
    module_path="speech_to_speech.STT.my_custom_stt_handler",
    class_name="MyCustomSTTHandler",
    attach_speculative_turns=True,
)

```

For complex initialization logic requiring conditional imports or additional context, write a custom factory function similar to `_create_parakeet` instead.

### Step 4: Register the BackendSpec

Append a new `BackendSpec` to the `STT_BACKENDS` list near line [87] of [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py).

```python

# src/speech_to_speech/backend_registry.py

from speech_to_speech.arguments_classes.my_custom_stt_arguments import MyCustomSTTHandlerArguments

STT_BACKENDS = build_backend_registry(
    "stt",
    [
        # ... existing specs like whisper ...

        BackendSpec(
            name="my_custom",
            kind="stt",
            arg_class=MyCustomSTTHandlerArguments,
            factory=_simple_handler_factory(
                "speech_to_speech.STT.my_custom_stt_handler",
                "MyCustomSTTHandler",
                attach_speculative_turns=True,
            ),
            config_prefix="my_custom",
            # optional: required_extra="my_custom_dep"

        ),
    ],
)

```

The registry is static and builds once at import time. Adding the spec here makes `"my_custom"` selectable via `select_backend(STT_BACKENDS, "my_custom", config)`.

### Step 5: (Optional) Provide a Configuration Normalizer

If your backend requires special configuration handling or prefix stripping, implement a normalizer function and pass it via the `normalize_config=` parameter in `BackendSpec`.

```python
def _normalize_my_custom_config(config: dict) -> dict:
    # Transform config if needed

    return config

```

See `_normalize_facebook_mms_config` (lines [143‑147]) for the reference implementation pattern.

### Step 6: Expose via CLI

Update [`src/speech_to_speech/cli.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/cli.py) to recognize the new backend name and expose its arguments in the help text. Users can then select your backend:

```bash
speech-to-speech --stt my_custom --my_custom_model_path ./model.pt --my_custom_language fr

```

## Usage Example

Once registered, the pipeline automatically instantiates your handler when the user specifies your backend name. The `create_backend_handler` function resolves the spec, normalizes arguments, and invokes your factory:

```python
from speech_to_speech.backend_registry import select_backend, create_backend_handler, STT_BACKENDS

# Selection phase

selection = select_backend(STT_BACKENDS, "my_custom", user_config)

# Instantiation phase

handler = create_backend_handler(selection, handler_context)

```

The handler now receives audio chunks via `queue_in` and outputs transcriptions via `queue_out` according to the standard STT handler contract.

## Summary

- The **backend registry** in [`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py) uses immutable `BackendSpec` entries to map names to handler factories.
- To integrate a custom STT backend, you must provide an **arguments dataclass**, a **handler implementation**, and a **factory function**, then register them in `STT_BACKENDS`.
- The `_simple_handler_factory` utility handles standard instantiation patterns, while custom factories support complex initialization logic.
- Registration occurs at import time, making your backend immediately selectable via configuration files or CLI arguments.

## Frequently Asked Questions

### How does the registry handle missing optional dependencies?

The `create_backend_handler` function (lines [82‑92]) wraps factory execution in a try-except block that catches `ImportError` and `ModuleNotFoundError`. If your backend declares a `required_extra` in its `BackendSpec`, the error message directs users to install the corresponding package, preventing hard failures when optional dependencies are absent.

### Can I modify the registry at runtime after import?

No. The `STT_BACKENDS`, `LLM_BACKENDS`, and `TTS_BACKENDS` registries are constructed statically when [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py) is first imported. To add a backend, you must edit the source file and append a `BackendSpec` to the list passed to `build_backend_registry` before the module loads.

### What is the purpose of the `attach_speculative_turns` parameter in the factory?

This flag indicates whether the handler supports speculative transcription turns for low-latency response generation. When `True`, the pipeline may send partial audio chunks before the user finishes speaking, allowing the STT backend to warm up inference. Set this according to whether your implementation can safely handle overlapping or incomplete audio segments.