# How to Implement a Custom STT Backend for the Speech-to-Speech Pipeline

> Learn to implement a custom STT backend for the speech-to-speech pipeline. This guide details creating configuration dataclasses, handler classes, and registering your backend for seamless integration.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-10

---

**To implement a custom STT backend for the speech-to-speech pipeline, create a configuration dataclass in `src/speech_to_speech/arguments_classes/`, implement a handler class inheriting from `BaseSTTHandler` in `src/speech_to_speech/STT/`, and register it via `BackendSpec` in [`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py).**

The Hugging Face `speech-to-speech` library constructs its pipeline from registered backends that follow a strict contract defined in the source code. To implement a custom STT backend for speech-to-speech pipeline integration, you must provide a configuration schema, a handler implementation, and a registry entry. This guide uses the actual file paths and class signatures from the repository to ensure your custom backend integrates seamlessly with speculative turn tracking and CLI argument parsing.

## Step 1: Create the Configuration Dataclass

Define a **dataclass** in `src/speech_to_speech/arguments_classes/` to declare your backend's hyperparameters. The registry uses this class to auto-generate CLI flags and normalize configuration dictionaries.

```python

# src/speech_to_speech/arguments_classes/my_custom_stt_arguments.py

from dataclasses import dataclass, field

@dataclass
class MyCustomSTTHandlerArguments:
    """Configuration options for MyCustom STT backend."""
    model_path: str = field(
        default="my_org/my_model",
        metadata={"help": "Path or identifier of the model to load."}
    )
    language: str | None = field(
        default=None,
        metadata={"help": "Target language code (e.g., 'en', 'fr'). None enables auto-detection."}
    )
    gen_max_new_tokens: int = field(
        default=128, 
        metadata={"help": "Maximum tokens to generate during transcription."}
    )

```

The `BackendSpec.normalize` method (via `normalize_dataclass_config` in [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py)) automatically collects fields prefixed with `gen_` into a `gen_kwargs` sub-dictionary. This matches the pattern used by existing backends like Whisper, allowing you to pass generation parameters cleanly to your model.

## Step 2: Implement the Handler Class

Create your handler in `src/speech_to_speech/STT/` by inheriting from **`BaseSTTHandler`**. You must implement the `setup` method for one-time initialization and the `process` method for inference.

```python

# src/speech_to_speech/STT/my_custom_stt_handler.py

from speech_to_speech.STT.base_stt_handler import BaseSTTHandler
from speech_to_speech.pipeline.messages import Transcription
from typing import Iterator

class MyCustomSTTHandler(BaseSTTHandler):
    """Custom STT backend implementation."""

    def setup(
        self, 
        model_path: str, 
        language: str | None = None, 
        gen_kwargs: dict | None = None
    ) -> None:
        """Load model resources once during pipeline construction."""
        self.model = load_my_model(model_path)  # User-defined loading logic

        self.language = language
        self.gen_kwargs = gen_kwargs or {}

    def process(self, vad_audio) -> Iterator[dict]:
        """Transcribe audio and yield Transcription objects."""
        # vad_audio.audio is a NumPy array prepared by BaseSTTHandler.prepare_model_inputs

        text, detected_lang = self.model.transcribe(
            vad_audio.audio, 
            language=self.language, 
            **self.gen_kwargs
        )
        
        yield Transcription(
            text=text,
            language_code=detected_lang,
            turn_id=vad_audio.turn_id,
            turn_revision=vad_audio.turn_revision,
            speech_stopped_at_s=vad_audio.created_at_s,
        )

```

Inheriting from `BaseSTTHandler` (defined in [`src/speech_to_speech/STT/base_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/base_stt_handler.py)) provides automatic speculative-turn filtering and queue management logic. The `process` method receives `vad_audio` objects containing NumPy audio arrays and metadata, and must yield at least one `Transcription` instance to integrate with the pipeline's message passing system.

## Step 3: Register the Backend

Add a **`BackendSpec`** entry to the `STT_BACKENDS` registry in [`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py). Use the **`_simple_handler_factory`** helper to wire up your class, or provide a custom factory function for complex initialization.

```python

# src/speech_to_speech/backend_registry.py

from speech_to_speech.arguments_classes.my_custom_stt_arguments import MyCustomSTTHandlerArguments
from speech_to_speech.STT.my_custom_stt_handler import MyCustomSTTHandler

STT_BACKENDS = build_backend_registry(
    "stt",
    [
        # ... existing backends ...

        BackendSpec(
            name="my-custom",
            kind="stt",
            config_type=MyCustomSTTHandlerArguments,
            create_handler=_simple_handler_factory(
                "speech_to_speech.STT.my_custom_stt_handler",
                "MyCustomSTTHandler",
                attach_speculative_turns=True,  # Required for proper turn ordering

            ),
            config_prefix="my_custom_stt",  # CLI flags become --my_custom_stt_model_path

            required_extra="my-custom-extra",  # Optional pip extra name

            capabilities=BackendCapabilities(supports_audio_input=False),
        ),
    ],
)

```

The registration fields serve specific purposes:

- **`name`** – CLI identifier used with `--stt my-custom`
- **`kind`** – Must be `"stt"` for speech-to-text backends
- **`config_type`** – Links to your arguments dataclass from Step 1
- **`create_handler`** – Factory function; `_simple_handler_factory` lazily imports your module and instantiates the class
- **`attach_speculative_turns`** – Grants access to `SpeculativeTurnTracker` for handling mid-stream revisions
- **`config_prefix`** – Prefixes CLI arguments to avoid namespace collisions

Because `ModuleArguments` in [`src/speech_to_speech/arguments_classes/module_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/module_arguments.py) reads the registry to populate `choices` metadata, your backend appears automatically in the CLI help without additional code.

## Step 4: Declare Optional Dependencies (Optional)

If your backend requires third-party libraries, declare them as pip extras to provide clear error messages when dependencies are missing.

Add the extra to [`pyproject.toml`](https://github.com/huggingface/speech-to-speech/blob/main/pyproject.toml):

```toml
[project.optional-dependencies]
my-custom-extra = ["my-custom-stt-lib>=1.0"]

```

Set the **`required_extra`** field in your `BackendSpec` to match this name. When users run `create_backend_handler` without the package installed, the registry raises an informative error referencing the exact pip install command (`pip install "speech-to-speech[my-custom-extra]"`).

## Step 5: Verify the Integration

Validate your implementation by checking the registry and running a test command:

```python
from speech_to_speech.backend_registry import STT_BACKENDS

assert "my-custom" in STT_BACKENDS
assert STT_BACKENDS["my-custom"].kind == "stt"

```

Test the full pipeline via CLI:

```bash
speech-to-speech serve \
  --stt my-custom \
  --my_custom_stt_model_path path/to/model \
  --my_custom_stt_language en

```

If the handler loads without import errors and returns `Transcription` objects, your custom STT backend is fully operational within the speech-to-speech pipeline.

## Summary

- **Configuration**: Create a dataclass in `src/speech_to_speech/arguments_classes/` with `field()` metadata for CLI integration.
- **Handler**: Inherit from `BaseSTTHandler` in `src/speech_to_speech/STT/` and implement `setup()` for initialization and `process()` to yield `Transcription` objects.
- **Registration**: Add a `BackendSpec` to `STT_BACKENDS` in [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py) using `_simple_handler_factory` with `attach_speculative_turns=True`.
- **CLI Integration**: The `config_prefix` field auto-generates prefixed arguments; no manual CLI code is required.
- **Dependencies**: Use `required_extra` in the spec and `[project.optional-dependencies]` in [`pyproject.toml`](https://github.com/huggingface/speech-to-speech/blob/main/pyproject.toml) for optional packages.

## Frequently Asked Questions

### What methods must a custom STT handler implement?

You must implement **`setup`** for one-time resource loading and **`process`** for inference. The `setup` method receives configuration values from your dataclass, while `process` receives `vad_audio` objects and must yield `Transcription` instances. Optional methods like `prepare_model_inputs` can be overridden for custom audio preprocessing.

### How does the CLI discover my custom backend automatically?

The **`ModuleArguments`** class in [`src/speech_to_speech/arguments_classes/module_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/module_arguments.py) inspects the `STT_BACKENDS` registry at import time to populate the `--stt` argument's `choices` metadata. Because the registry is built when [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py) loads, your backend appears in the CLI immediately after registration without modifying argument parsing code.

### Can I use a custom factory function instead of `_simple_handler_factory`?

Yes. Replace `_simple_handler_factory` with your own function accepting `(context: HandlerContext, config: Mapping[str, Any])` and returning your handler instance. This is useful when you need to inject shared pipeline resources or perform complex initialization logic beyond simple class instantiation.

### Why must I set `attach_speculative_turns=True` when registering the backend?

Setting **`attach_speculative_turns=True`** wires your handler to the `SpeculativeTurnTracker`, which manages mid-stream transcription revisions and turn ordering. Without this, your backend cannot properly handle overlapping speech or correction events, leading to out-of-order messages in the pipeline.