# Common Issues and How to Troubleshoot Them in the Speech-to-Speech Pipeline

> Troubleshoot common speech-to-speech pipeline issues like concurrency bugs, dependency mismatches, and protocol errors. Learn diagnostic patterns and targeted logging for the huggingface/speech-to-speech repository.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-05

---

**Most runtime problems in the `huggingface/speech-to-speech` pipeline stem from concurrency bugs (MLX lock contention, stale turn tracking), dependency mismatches (CUDA wheels, NumPy versions), or protocol mismatches (audio format, control messages), all of which can be diagnosed through targeted logging and the debugging patterns shown below.**

The `huggingface/speech-to-speech` repository implements a real-time speech-to-speech pipeline that chains Voice Activity Detection (VAD), Speech-to-Text (STT), Large Language Model (LLM), and Text-to-Speech (TTS) stages. Understanding common issues and how to troubleshoot them in the speech-to-speech pipeline requires familiarity with its threaded architecture, message-passing contracts, and platform-specific constraints.

---

## Pipeline Architecture Overview

The system runs as a **four-stage cascade** (VAD → STT → LLM → TTS) where each stage executes in its own thread and communicates via typed messages defined in [`src/speech_to_speech/pipeline/messages.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/messages.py). Control flow signals—such as session termination—use lightweight dataclasses in [`src/speech_to_speech/pipeline/control.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/control.py). The [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) module orchestrates everything, parsing CLI arguments and launching the main event loop.

Three concurrency primitives demand particular attention:

- **`SpeculativeTurnTracker`** ([`pipeline/speculative_turns.py`](https://github.com/huggingface/speech-to-speech/blob/main/pipeline/speculative_turns.py)): Tracks turn revisions to abort stale work and prevent duplicate output when turns reopen
- **`CancelScope`** ([`pipeline/cancel_scope.py`](https://github.com/huggingface/speech-to-speech/blob/main/pipeline/cancel_scope.py)): Provides generation-based cancellation flags that worker threads check before emitting output
- **`MLXLockContext`** ([`utils/mlx_lock.py`](https://github.com/huggingface/speech-to-speech/blob/main/utils/mlx_lock.py)): A global re-entrant lock required by all MLX-based handlers on Apple Silicon to prevent Metal command-buffer races

---

## Deadlock or Freeze on Apple Silicon

**Symptom:** The pipeline becomes unresponsive shortly after startup or during model loading on macOS with Apple Silicon.

**Root cause:** The MLX lock was acquired but not released, often due to a handler exiting abnormally without calling `release_mlx_lock`. The lock depth remains greater than zero, and logs warn about "MLX lock release requested by non-owner thread."

**Diagnosis:** Search console output for warnings containing "non-owner thread" or check [`utils/mlx_lock.py`](https://github.com/huggingface/speech-to-speech/blob/main/utils/mlx_lock.py) for lock depth diagnostics.

**Fix:** Ensure every MLX model invocation uses the context manager pattern:

```python
from speech_to_speech.utils.mlx_lock import MLXLockContext

def run_mlx_model(handler_name: str):
    with MLXLockContext(handler_name=handler_name, timeout=30) as acquired:
        if not acquired:
            raise RuntimeError(f"{handler_name}: could not obtain MLX lock")
        # Model inference occurs here

        # Lock releases automatically on context exit

```

Enable `INFO`-level logging to see acquisition times and previous owners if you suspect lock contention.

---

## Duplicate Responses After Turn Reopening

**Symptom:** Users hear the same response twice, or stale LLM output plays after they interrupt and restart speaking.

**Root cause:** The `SpeculativeTurnTracker` was not notified of the new revision, or a handler committed output before verifying `commit_if_latest_after_pending_reopen`. When a turn reopens, the revision number increments, but pending work from the previous revision may still complete and emit audio.

**Diagnosis:** Verify that your handler calls `tracker.observe(turn_id, revision)` for every new revision. Inspect `tracker._latest_revision` (an `OrderedDict`) at runtime to confirm revision tracking state.

**Fix:** Implement the observe-then-guard pattern:

```python
from speech_to_speech.pipeline.speculative_turns import SpeculativeTurnTracker

tracker = SpeculativeTurnTracker()

def on_new_audio(turn_id: str, revision: int, audio):
    # Register revision before any async work begins

    tracker.observe(turn_id, revision)
    
    # Later, before final output commitment:

    if tracker.is_latest_after_pending_reopen(turn_id, revision):
        commit_response(turn_id, revision, audio)
    else:
        logger.debug("Stale turn %s rev %d ignored", turn_id, revision)

```

---

## Qwen3-TTS Generation Errors on Linux

**Symptom:** Import failures or symbol resolution errors when loading the Qwen3-TTS backend, particularly messages referencing `qwentts-cpp-python`.

**Root cause:** The default `qwentts-cpp-python` wheel targets CUDA 12.8, but the host system has CUDA 12.4 or no CUDA at all. This mismatch manifests in [`TTS/qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/TTS/qwen3_tts_handler.py) during startup.

**Diagnosis:** Check startup logs for missing symbol errors originating from the TTS handler.

**Fix:** Install the wheel matching your CUDA version per the *CUDA Note for Qwen3-TTS* section in the README. As a defensive coding pattern, wrap the import:

```python
try:
    from speech_to_speech.TTS.qwen3_tts_handler import Qwen3TTSHandler
except ImportError as exc:
    logger.error(
        "Failed to load Qwen3-TTS backend – check CUDA wheel compatibility. %s",
        exc,
    )
    raise

```

---

## Session End Not Propagating

**Symptom:** Clients remain connected after sending a session termination signal, or handlers continue processing stale audio.

**Root cause:** Control messages use `SESSION_END` from [`pipeline/control.py`](https://github.com/huggingface/speech-to-speech/blob/main/pipeline/control.py), but `is_control_message` was invoked with a mismatched `ControlKind`. The message passes through unrecognized.

**Diagnosis:** Add debug logging to print the received message class and its `kind` attribute before calling `is_control_message`.

**Fix:** Ensure consistent use of `ControlKind.SESSION_END` when constructing and checking control messages in [`pipeline/control.py`](https://github.com/huggingface/speech-to-speech/blob/main/pipeline/control.py).

---

## Audio Format Mismatches

**Symptom:** Garbled output, silence, or codec errors when streaming audio via WebSocket or raw socket connections.

**Root cause:** The server expects raw PCM at 16 kHz, mono, 16-bit signed integer (int16). Clients sending different sample rates, bit depths, or compressed formats trigger misalignment in [`connections/socket_receiver.py`](https://github.com/huggingface/speech-to-speech/blob/main/connections/socket_receiver.py) or [`connections/websocket_streamer.py`](https://github.com/huggingface/speech-to-speech/blob/main/connections/websocket_streamer.py).

**Diagnosis:** Verify client encoding with command-line tools before debugging server code.

**Fix:** Standardize client output using SoX or equivalent:

```bash
sox input.wav -r 16000 -e signed-integer -b 16 -t raw -

```

---

## Stale STT Results Blocking Downstream Work

**Symptom:** New transcriptions are ignored after a session restart, as if the pipeline believes they belong to a cancelled previous session.

**Root cause:** `CancelScope` maintains a generation counter that increments on cancellation. If not reset when a new session begins, downstream handlers treat fresh transcriptions as stale based on `is_stale` checks in [`pipeline/cancel_scope.py`](https://github.com/huggingface/speech-to-speech/blob/main/pipeline/cancel_scope.py).

**Diagnosis:** Log the generation counter at session boundaries; unexpectedly high values indicate failure to reset.

**Fix:** Explicitly reset state on session creation:

```python
from speech_to_speech.pipeline.cancel_scope import CancelScope

cancel_scope = CancelScope()

def on_session_start():
    cancel_scope.reset()
    logger.info("CancelScope reset for new session")

```

---

## Missing Optional Dependencies

**Symptom:** `ImportError` at startup when both DeepFilterNet and Pocket TTS extras are installed.

**Root cause:** **NumPy version conflict**—DeepFilterNet requires `numpy<2` while Pocket TTS requires `numpy>=2`. The README documents this incompatibility.

**Diagnosis:** Check the traceback for version-related import failures.

**Fix:** Install only the needed extra, never both:

```bash

# For DeepFilterNet noise suppression

pip install "speech-to-speech[deepfilter]"

# OR for Pocket TTS

pip install "speech-to-speech[pocket]"

```

---

## Key Files for Debugging

| File | Purpose |
|------|---------|
| [`src/speech_to_speech/pipeline/messages.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/messages.py) | Typed message contracts for inter-stage communication |
| [`src/speech_to_speech/pipeline/control.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/control.py) | Control message enums and `is_control_message` helper |
| [`src/speech_to_speech/pipeline/speculative_turns.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/speculative_turns.py) | `SpeculativeTurnTracker` and turn revision logic |
| [`src/speech_to_speech/pipeline/cancel_scope.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/cancel_scope.py) | `CancelScope` with generation-based staleness checks |
| [`src/speech_to_speech/utils/mlx_lock.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/utils/mlx_lock.py) | `MLXLockContext` for Apple Silicon thread safety |
| [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) | CLI entry point and pipeline orchestration |
| [`README.md`](https://github.com/huggingface/speech-to-speech/blob/main/README.md) | Installation notes and dependency caveats |

---

## Summary

- **Apple Silicon deadlocks:** Always use `MLXLockContext` with timeout handling; watch for non-owner release warnings
- **Duplicate responses:** Ensure `SpeculativeTurnTracker.observe()` precedes any `is_latest_after_pending_reopen()` checks
- **CUDA errors:** Match `qwentts-cpp-python` wheel to host CUDA version; defensively wrap imports
- **Session control:** Verify `ControlKind` consistency across message construction and validation
- **Audio glitches:** Enforce 16 kHz mono int16 PCM on client side before server debugging
- **Stale processing:** Reset `CancelScope` at every session boundary to clear generation counters
- **Dependency conflicts:** Never install both `[deepfilter]` and `[pocket]` extras due to NumPy version incompatibility

---

## Frequently Asked Questions

### Why does the pipeline freeze on my Mac but work on Linux?

Apple Silicon requires exclusive access to the MLX framework through a global lock. If any handler acquires `MLXLockContext` and exits without releasing it—due to an unhandled exception or improper context usage—the lock remains held and blocks all subsequent MLX operations. Linux uses CUDA or CPU backends that do not share this constraint.

### How do I prevent duplicate audio when a user interrupts and restarts speaking?

Implement proper **speculative turn tracking**. Each new audio segment must call `tracker.observe(turn_id, revision)` with an incremented revision number. Before committing any TTS output, verify `tracker.is_latest_after_pending_reopen(turn_id, revision)` returns true; otherwise discard the stale result. See [`pipeline/speculative_turns.py`](https://github.com/huggingface/speech-to-speech/blob/main/pipeline/speculative_turns.py) for the implementation.

### What CUDA version do I need for Qwen3-TTS?

The default wheel targets CUDA 12.8. If your system has CUDA 12.4 or lacks CUDA entirely, install the appropriate alternative wheel documented in the README's *CUDA Note for Qwen3-TTS* section. Mismatched wheels produce symbol resolution errors during [`qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/qwen3_tts_handler.py) initialization.