How to Implement a Custom STT Backend for the Speech-to-Speech Pipeline

To implement a custom STT backend for the speech-to-speech pipeline, create a configuration dataclass in src/speech_to_speech/arguments_classes/, implement a handler class inheriting from BaseSTTHandler in src/speech_to_speech/STT/, and register it via BackendSpec in src/speech_to_speech/backend_registry.py.

The Hugging Face speech-to-speech library constructs its pipeline from registered backends that follow a strict contract defined in the source code. To implement a custom STT backend for speech-to-speech pipeline integration, you must provide a configuration schema, a handler implementation, and a registry entry. This guide uses the actual file paths and class signatures from the repository to ensure your custom backend integrates seamlessly with speculative turn tracking and CLI argument parsing.

Step 1: Create the Configuration Dataclass

Define a dataclass in src/speech_to_speech/arguments_classes/ to declare your backend's hyperparameters. The registry uses this class to auto-generate CLI flags and normalize configuration dictionaries.

# src/speech_to_speech/arguments_classes/my_custom_stt_arguments.py

from dataclasses import dataclass, field

@dataclass
class MyCustomSTTHandlerArguments:
    """Configuration options for MyCustom STT backend."""
    model_path: str = field(
        default="my_org/my_model",
        metadata={"help": "Path or identifier of the model to load."}
    )
    language: str | None = field(
        default=None,
        metadata={"help": "Target language code (e.g., 'en', 'fr'). None enables auto-detection."}
    )
    gen_max_new_tokens: int = field(
        default=128, 
        metadata={"help": "Maximum tokens to generate during transcription."}
    )

The BackendSpec.normalize method (via normalize_dataclass_config in backend_registry.py) automatically collects fields prefixed with gen_ into a gen_kwargs sub-dictionary. This matches the pattern used by existing backends like Whisper, allowing you to pass generation parameters cleanly to your model.

Step 2: Implement the Handler Class

Create your handler in src/speech_to_speech/STT/ by inheriting from BaseSTTHandler. You must implement the setup method for one-time initialization and the process method for inference.

# src/speech_to_speech/STT/my_custom_stt_handler.py

from speech_to_speech.STT.base_stt_handler import BaseSTTHandler
from speech_to_speech.pipeline.messages import Transcription
from typing import Iterator

class MyCustomSTTHandler(BaseSTTHandler):
    """Custom STT backend implementation."""

    def setup(
        self, 
        model_path: str, 
        language: str | None = None, 
        gen_kwargs: dict | None = None
    ) -> None:
        """Load model resources once during pipeline construction."""
        self.model = load_my_model(model_path)  # User-defined loading logic

        self.language = language
        self.gen_kwargs = gen_kwargs or {}

    def process(self, vad_audio) -> Iterator[dict]:
        """Transcribe audio and yield Transcription objects."""
        # vad_audio.audio is a NumPy array prepared by BaseSTTHandler.prepare_model_inputs

        text, detected_lang = self.model.transcribe(
            vad_audio.audio, 
            language=self.language, 
            **self.gen_kwargs
        )
        
        yield Transcription(
            text=text,
            language_code=detected_lang,
            turn_id=vad_audio.turn_id,
            turn_revision=vad_audio.turn_revision,
            speech_stopped_at_s=vad_audio.created_at_s,
        )

Inheriting from BaseSTTHandler (defined in src/speech_to_speech/STT/base_stt_handler.py) provides automatic speculative-turn filtering and queue management logic. The process method receives vad_audio objects containing NumPy audio arrays and metadata, and must yield at least one Transcription instance to integrate with the pipeline's message passing system.

Step 3: Register the Backend

Add a BackendSpec entry to the STT_BACKENDS registry in src/speech_to_speech/backend_registry.py. Use the _simple_handler_factory helper to wire up your class, or provide a custom factory function for complex initialization.

# src/speech_to_speech/backend_registry.py

from speech_to_speech.arguments_classes.my_custom_stt_arguments import MyCustomSTTHandlerArguments
from speech_to_speech.STT.my_custom_stt_handler import MyCustomSTTHandler

STT_BACKENDS = build_backend_registry(
    "stt",
    [
        # ... existing backends ...

        BackendSpec(
            name="my-custom",
            kind="stt",
            config_type=MyCustomSTTHandlerArguments,
            create_handler=_simple_handler_factory(
                "speech_to_speech.STT.my_custom_stt_handler",
                "MyCustomSTTHandler",
                attach_speculative_turns=True,  # Required for proper turn ordering

            ),
            config_prefix="my_custom_stt",  # CLI flags become --my_custom_stt_model_path

            required_extra="my-custom-extra",  # Optional pip extra name

            capabilities=BackendCapabilities(supports_audio_input=False),
        ),
    ],
)

The registration fields serve specific purposes:

• name – CLI identifier used with --stt my-custom
• kind – Must be "stt" for speech-to-text backends
• config_type – Links to your arguments dataclass from Step 1
• create_handler – Factory function; _simple_handler_factory lazily imports your module and instantiates the class
• attach_speculative_turns – Grants access to SpeculativeTurnTracker for handling mid-stream revisions
• config_prefix – Prefixes CLI arguments to avoid namespace collisions

Because ModuleArguments in src/speech_to_speech/arguments_classes/module_arguments.py reads the registry to populate choices metadata, your backend appears automatically in the CLI help without additional code.

Step 4: Declare Optional Dependencies (Optional)

If your backend requires third-party libraries, declare them as pip extras to provide clear error messages when dependencies are missing.

Add the extra to pyproject.toml:

[project.optional-dependencies]
my-custom-extra = ["my-custom-stt-lib>=1.0"]

Set the required_extra field in your BackendSpec to match this name. When users run create_backend_handler without the package installed, the registry raises an informative error referencing the exact pip install command (pip install "speech-to-speech[my-custom-extra]").

Step 5: Verify the Integration

Validate your implementation by checking the registry and running a test command:

from speech_to_speech.backend_registry import STT_BACKENDS

assert "my-custom" in STT_BACKENDS
assert STT_BACKENDS["my-custom"].kind == "stt"

Test the full pipeline via CLI:

speech-to-speech serve \
  --stt my-custom \
  --my_custom_stt_model_path path/to/model \
  --my_custom_stt_language en

If the handler loads without import errors and returns Transcription objects, your custom STT backend is fully operational within the speech-to-speech pipeline.

Summary

• Configuration: Create a dataclass in src/speech_to_speech/arguments_classes/ with field() metadata for CLI integration.
• Handler: Inherit from BaseSTTHandler in src/speech_to_speech/STT/ and implement setup() for initialization and process() to yield Transcription objects.
• Registration: Add a BackendSpec to STT_BACKENDS in backend_registry.py using _simple_handler_factory with attach_speculative_turns=True.
• CLI Integration: The config_prefix field auto-generates prefixed arguments; no manual CLI code is required.
• Dependencies: Use required_extra in the spec and [project.optional-dependencies] in pyproject.toml for optional packages.

Frequently Asked Questions

What methods must a custom STT handler implement?

You must implement setup for one-time resource loading and process for inference. The setup method receives configuration values from your dataclass, while process receives vad_audio objects and must yield Transcription instances. Optional methods like prepare_model_inputs can be overridden for custom audio preprocessing.

How does the CLI discover my custom backend automatically?

The ModuleArguments class in src/speech_to_speech/arguments_classes/module_arguments.py inspects the STT_BACKENDS registry at import time to populate the --stt argument's choices metadata. Because the registry is built when backend_registry.py loads, your backend appears in the CLI immediately after registration without modifying argument parsing code.

Can I use a custom factory function instead of _simple_handler_factory?

Yes. Replace _simple_handler_factory with your own function accepting (context: HandlerContext, config: Mapping[str, Any]) and returning your handler instance. This is useful when you need to inject shared pipeline resources or perform complex initialization logic beyond simple class instantiation.

Why must I set attach_speculative_turns=True when registering the backend?

Setting attach_speculative_turns=True wires your handler to the SpeculativeTurnTracker, which manages mid-stream transcription revisions and turn ordering. Without this, your backend cannot properly handle overlapping speech or correction events, leading to out-of-order messages in the pipeline.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}