How to Integrate a Custom STT Backend into the Speech-to-Speech Backend Registry

Integrate a custom STT backend by creating a BackendSpec entry that links your arguments dataclass and handler factory to the STT_BACKENDS registry in src/speech_to_speech/backend_registry.py.

The huggingface/speech-to-speech library routes all speech-to-text processing through a static backend registry that maps configuration names to handler instances. To integrate a custom STT backend into the backend registry system, you extend the STT_BACKENDS list with a new specification that defines how to construct your implementation.

Understanding the Backend Registry Architecture

The registry system centers on the BackendSpec frozen dataclass defined in src/speech_to_speech/backend_registry.py. Each spec stores a backend’s name, kind ("stt", "tts", or "llm"), arguments class, factory function, and optional metadata including configuration prefixes and capability flags.

When the pipeline initializes, the library executes three phases:

1. Registry Construction – The module imports trigger build_backend_registry(kind, specs) (lines [49‑59]), creating immutable mappings for STT_BACKENDS, LLM_BACKENDS, and TTS_BACKENDS.
2. Backend Selection – The select_backend(registry, name, config) function (lines [62‑68]) validates the requested name and normalizes configuration into a plain dictionary.
3. Handler Instantiation – The create_backend_handler(selection, context) function (lines [82‑92]) invokes the factory stored in the spec, passing the normalized config and handling optional-dependency import errors.

Step-by-Step Integration Guide

Step 1: Define the Arguments Dataclass

Create a dataclass in src/speech_to_speech/arguments_classes/ that inherits from EmptyBackendArguments. This class declares the configuration fields your backend requires.

# src/speech_to_speech/arguments_classes/my_custom_stt_arguments.py

from dataclasses import dataclass
from speech_to_speech.backend_registry import EmptyBackendArguments

@dataclass
class MyCustomSTTHandlerArguments(EmptyBackendArguments):
    model_path: str = "models/my_custom.pt"
    language: str = "en"
    chunk_size: int = 16000

Follow the pattern established by WhisperSTTHandlerArguments (lines [18‑24]) to ensure compatibility with the CLI parsing system.

Step 2: Implement the Handler Class

Write a handler class in src/speech_to_speech/STT/ that accepts the standard constructor signature used throughout the library: __init__(stop_event, queue_in, queue_out, *, setup_args=…, setup_kwargs=…).

# src/speech_to_speech/STT/my_custom_stt_handler.py

from typing import Any
from queue import Queue
from threading import Event

class MyCustomSTTHandler:
    def __init__(
        self,
        stop_event: Event,
        queue_in: Queue[Any],
        queue_out: Queue[Any],
        *,
        setup_kwargs: dict[str, Any],
    ):
        self.stop_event = stop_event
        self.queue_in = queue_in
        self.queue_out = queue_out
        self.model_path = setup_kwargs["model_path"]
        self.language = setup_kwargs["language"]
        self.chunk_size = setup_kwargs["chunk_size"]
        # Load model, initialize resources...

    def process(self, audio_chunk):
        # Perform transcription logic

        transcribed_text = self._transcribe(audio_chunk)
        self.queue_out.put(transcribed_text)

    def close(self):
        # Cleanup resources

        pass

Reference WhisperSTTHandler in src/speech_to_speech/STT/whisper_stt_handler.py for the complete implementation pattern.

Step 3: Create the Factory Function

Use _simple_handler_factory (lines [104‑131]) if your handler can be constructed directly from the normalized configuration dictionary. This factory returns a HandlerFactory that imports and instantiates your class.

from speech_to_speech.backend_registry import _simple_handler_factory

factory = _simple_handler_factory(
    module_path="speech_to_speech.STT.my_custom_stt_handler",
    class_name="MyCustomSTTHandler",
    attach_speculative_turns=True,
)

For complex initialization logic requiring conditional imports or additional context, write a custom factory function similar to _create_parakeet instead.

Step 4: Register the BackendSpec

Append a new BackendSpec to the STT_BACKENDS list near line [87] of backend_registry.py.

# src/speech_to_speech/backend_registry.py

from speech_to_speech.arguments_classes.my_custom_stt_arguments import MyCustomSTTHandlerArguments

STT_BACKENDS = build_backend_registry(
    "stt",
    [
        # ... existing specs like whisper ...

        BackendSpec(
            name="my_custom",
            kind="stt",
            arg_class=MyCustomSTTHandlerArguments,
            factory=_simple_handler_factory(
                "speech_to_speech.STT.my_custom_stt_handler",
                "MyCustomSTTHandler",
                attach_speculative_turns=True,
            ),
            config_prefix="my_custom",
            # optional: required_extra="my_custom_dep"

        ),
    ],
)

The registry is static and builds once at import time. Adding the spec here makes "my_custom" selectable via select_backend(STT_BACKENDS, "my_custom", config).

Step 5: (Optional) Provide a Configuration Normalizer

If your backend requires special configuration handling or prefix stripping, implement a normalizer function and pass it via the normalize_config= parameter in BackendSpec.

def _normalize_my_custom_config(config: dict) -> dict:
    # Transform config if needed

    return config

See _normalize_facebook_mms_config (lines [143‑147]) for the reference implementation pattern.

Step 6: Expose via CLI

Update src/speech_to_speech/cli.py to recognize the new backend name and expose its arguments in the help text. Users can then select your backend:

speech-to-speech --stt my_custom --my_custom_model_path ./model.pt --my_custom_language fr

Usage Example

Once registered, the pipeline automatically instantiates your handler when the user specifies your backend name. The create_backend_handler function resolves the spec, normalizes arguments, and invokes your factory:

from speech_to_speech.backend_registry import select_backend, create_backend_handler, STT_BACKENDS

# Selection phase

selection = select_backend(STT_BACKENDS, "my_custom", user_config)

# Instantiation phase

handler = create_backend_handler(selection, handler_context)

The handler now receives audio chunks via queue_in and outputs transcriptions via queue_out according to the standard STT handler contract.

Summary

• The backend registry in src/speech_to_speech/backend_registry.py uses immutable BackendSpec entries to map names to handler factories.
• To integrate a custom STT backend, you must provide an arguments dataclass, a handler implementation, and a factory function, then register them in STT_BACKENDS.
• The _simple_handler_factory utility handles standard instantiation patterns, while custom factories support complex initialization logic.
• Registration occurs at import time, making your backend immediately selectable via configuration files or CLI arguments.

Frequently Asked Questions

How does the registry handle missing optional dependencies?

The create_backend_handler function (lines [82‑92]) wraps factory execution in a try-except block that catches ImportError and ModuleNotFoundError. If your backend declares a required_extra in its BackendSpec, the error message directs users to install the corresponding package, preventing hard failures when optional dependencies are absent.

Can I modify the registry at runtime after import?

No. The STT_BACKENDS, LLM_BACKENDS, and TTS_BACKENDS registries are constructed statically when backend_registry.py is first imported. To add a backend, you must edit the source file and append a BackendSpec to the list passed to build_backend_registry before the module loads.

What is the purpose of the attach_speculative_turns parameter in the factory?

This flag indicates whether the handler supports speculative transcription turns for low-latency response generation. When True, the pipeline may send partial audio chunks before the user finishes speaking, allowing the STT backend to warm up inference. Set this according to whether your implementation can safely handle overlapping or incomplete audio segments.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}