# How to Contribute to the HuggingFace Speech-to-Speech Project: A Complete Guide

> Learn how to contribute to the HuggingFace speech-to-speech project. Follow our guide to set up your environment, understand the pipeline, and submit pull requests to enhance this open-source initiative.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-02

---

**Contributing to the HuggingFace speech-to-speech project involves setting up a local development environment, understanding the modular VAD→STT→LLM→TTS pipeline architecture, and submitting focused pull requests that add backends, fix issues, or improve documentation.**

The **HuggingFace speech-to-speech** repository is a production-ready, low-latency voice assistant pipeline built entirely in Python. Whether you want to add a new TTS backend, optimize queue handling, or improve documentation, this guide walks you through the contribution workflow using actual source paths and patterns from the codebase.

## Setting Up Your Development Environment

Before writing code, you need the repository cloned with dependencies installed and the test suite passing.

```bash
git clone https://github.com/huggingface/speech-to-speech.git
cd speech-to-speech
uv sync                 # installs in editable mode from pyproject.toml

pytest                  # run unit tests

ruff check              # linting check

```

This setup (lines 44‑51 of the README) ensures your changes integrate cleanly with the existing CI pipeline.

## Understanding the Core Architecture

The **speech-to-speech** pipeline uses three fundamental concepts that every contributor must understand:

| Concept | Purpose | Location |
|---------|---------|----------|
| **Argument classes** | Typed `dataclass` objects exposing every CLI flag, parsed with `HfArgumentParser` | `src/speech_to_speech/arguments_classes/*` |
| **Handlers** | `BaseHandler` subclasses encoding each pipeline stage (VAD, STT, LLM, TTS) | `src/speech_to_speech/VAD/*`, `STT/*`, `LLM/*`, `TTS/*` |
| **Pipeline builder** | Orchestrates queues, events, and handler chains across four deployment modes | [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) |

### Pipeline Execution Flow

In [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py), the `build_pipeline` function constructs the communication graph:

- **Argument pre‑parsing** (lines 57‑70): Determines which LLM argument class to register based on `--lm_model` selection
- **Handler chain construction** via `_build_pipeline_handlers` (lines 78‑94): Wires `VAD → STT → TranscriptionNotifier → LLM → LMOutputProcessor → TTS`
- **Mode‑specific setup** (lines 106‑140): Selects between `local`, `raw-websocket`, `realtime`, or `socket` modes

## Contributing a New Backend Handler

Adding support for a new speech-to-text or text-to-speech model follows a consistent pattern.

### Adding a Custom STT Handler

1. **Create your handler class** in `src/speech_to_speech/STT/`:

```python

# src/speech_to_speech/STT/my_stt_handler.py

from speech_to_speech.baseHandler import BaseHandler
from speech_to_speech.pipeline.handler_types import STTIn, STTOut

class MySTTHandler(BaseHandler[STTIn, STTOut]):
    def __init__(self, stop_event, queue_in, queue_out, *, model_name: str):
        super().__init__(stop_event, queue_in, queue_out)
        self.model_name = model_name
        self._load_model()

    def _run_once(self, item: STTIn) -> None:
        transcription = self.model.transcribe(item.audio_bytes)
        self.queue_out.put(STTOut(text=transcription, language=item.language))

```

2. **Add an arguments dataclass** in [`src/speech_to_speech/arguments_classes/stt_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/stt_arguments.py) (or create a new module).

3. **Register in the pipeline builder** at lines 80‑78 of [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py):

```python
elif module_kwargs.stt == "my_custom":
    from speech_to_speech.STT.my_stt_handler import MySTTHandler
    return MySTTHandler(
        stop_event,
        queue_in=audio_chunks_queue,
        queue_out=transcription_queue,
        model_name=module_kwargs.stt_model_name,
    )

```

### Adding a Custom TTS Handler

The pattern is identical for TTS in `src/speech_to_speech/TTS/`:

```python

# Register in get_tts_handler (lines 49-63 of s2s_pipeline.py)

elif module_kwargs.tts == "my_custom":
    from speech_to_speech.TTS.my_custom_tts_handler import MyCustomTTSHandler
    return MyCustomTTSHandler(
        stop_event,
        queue_in=lm_response_queue,
        queue_out=send_audio_chunks_queue,
        voice=module_kwargs.tts_voice,
    )

```

## Testing Your Changes Locally

Before submitting, verify your implementation against the actual pipeline.

### Run the Realtime Server Demo

```bash

# Terminal 1: Start the WebSocket server

export OPENAI_API_KEY=sk-...   # or leave unset for local LLM

speech-to-speech                # starts at ws://localhost:8765/v1/realtime

# Terminal 2: Connect with the demo client

python scripts/listen_and_play_realtime.py --host 127.0.0.1 --port 8765

```

### Test Local Microphone Mode

```bash
speech-to-speech --local_mac_optimal_settings

```

### Write a Unit Test for Your Handler

```python
def test_my_stt_handler():
    from speech_to_speech.STT.my_stt_handler import MySTTHandler
    from speech_to_speech.utils.thread_manager import ThreadManager
    from threading import Event
    from queue import Queue

    in_q = Queue()
    out_q = Queue()
    stop_evt = Event()
    
    handler = MySTTHandler(stop_evt, in_q, out_q, model_name="test-model")
    tm = ThreadManager([handler])
    tm.start()
    
    in_q.put(STTIn(audio_bytes=b"test audio data", sample_rate=16000))
    result = out_q.get(timeout=5)
    
    assert result.text == "expected transcription"
    tm.stop()

```

## Suggested First-Time Contributions to the Speech-to-Speech Project

| Area | Specific Tasks |
|------|--------------|
| **Documentation** | Expand README sections on custom backend development; add queue flow diagrams |
| **Test coverage** | Add edge‑case tests for empty audio chunks and graceful shutdown in all four modes |
| **Handler refactor** | Extract common `BaseHandler` boilerplate into shared mix‑ins |
| **New backend** | Implement a lightweight TTS option (e.g., `gTTS`) with `--tts=gtts` flag |
| **Performance** | Benchmark `--num_pipelines` scaling and profile queue bottlenecks |

## Submitting Your Pull Request

Follow the workflow defined in the README (lines 72‑75):

1. **Fork** the repository and create a feature branch: `git checkout -b add-my-backend`
2. **Make focused changes** — limit each PR to one logical addition or fix
3. **Validate locally**: `pytest && ruff check`
4. **Update documentation** for any new CLI flags or API changes
5. **Open a PR** targeting `main` with clear description: "Closes #123: Adds MyCustom TTS backend"
6. **Monitor CI** — GitHub Actions runs tests and style checks automatically

The HuggingFace team prefers **small, focused PRs** over large refactorings. For substantial architectural changes, open an issue first to discuss approach.

## Key Files Every Contributor Should Know

| File | Function |
|------|----------|
| [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) | Central orchestration: argument parsing, handler wiring, mode selection |
| [`src/speech_to_speech/arguments_classes/module_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/module_arguments.py) | Global CLI options and backend registry |
| [`src/speech_to_speech/baseHandler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/baseHandler.py) | Abstract base class for all pipeline stage handlers |
| [`src/speech_to_speech/utils/thread_manager.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/utils/thread_manager.py) | Thread lifecycle management for handler processes |
| [`src/speech_to_speech/VAD/vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/vad_handler.py) | Reference Silero VAD implementation |
| [`src/speech_to_speech/STT/parakeet_tdt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/parakeet_tdt_handler.py) | Default local STT backend pattern |
| [`src/speech_to_speech/TTS/qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/qwen3_tts_handler.py) | Default TTS backend pattern |
| [`README.md`](https://github.com/huggingface/speech-to-speech/blob/main/README.md) | Installation, quick‑start, and contribution guidelines |

## Summary

- **Clone and install** with `uv sync`, then run `pytest` and `ruff check` to validate your environment
- **Understand the three pillars**: argument classes, `BaseHandler` subclasses, and the `build_pipeline` orchestrator in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py)
- **Add backends** by implementing handlers in the appropriate `VAD/`, `STT/`, `LLM/`, or `TTS/` directory, registering them in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py), and adding corresponding argument classes
- **Test locally** using `speech-to-speech` with the demo client or `--local_mac_optimal_settings`
- **Submit focused PRs** referencing issues, with passing CI and updated documentation

## Frequently Asked Questions

### What skills do I need to contribute to the HuggingFace speech-to-speech project?

You need intermediate Python experience, familiarity with `asyncio` or threading, and basic understanding of audio processing concepts. The codebase uses `dataclasses`, `HfArgumentParser`, and typed queues extensively — patterns common in modern ML engineering but approachable with standard Python knowledge.

### Can I add a commercial API backend like Azure Speech or AWS Polly?

Yes. Create a handler subclass in the appropriate `STT/` or `TTS/` directory that calls the commercial API in `_run_once()`. Add your API key handling to the arguments classes, and register the backend in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py). Follow the pattern in existing handlers like [`qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/qwen3_tts_handler.py) for queue management.

### How does the `ThreadManager` ensure clean shutdown?

The `ThreadManager` (in [`src/speech_to_speech/utils/thread_manager.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/utils/thread_manager.py)) collects all handler threads and provides `start()` and `stop()` methods. On `stop()`, it sets the shared `stop_event` that each handler checks in its main loop, allowing graceful termination rather than forced thread killing.

### Should I write tests for my new backend contribution?

Yes — include unit tests that instantiate your handler with `ThreadManager`, feed sample data through the input queue, and verify expected output on the result queue. Test edge cases like empty inputs and rapid start/stop cycles to match the project's quality standards.