# How Speculative Turns Handle Multiple In-Flight Responses in Hugging Face Speech-to-Speech

> Discover how Hugging Face's speculative turns manage multiple in-flight responses with a thread-safe tracker, ensuring reliable speech-to-speech processing and preventing race conditions.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: internals
- Published: 2026-07-08

---

**The speculative turns feature coordinates concurrent responses using a thread-safe `SpeculativeTurnTracker` that enforces single-candidate semantics and atomic revision tracking to prevent race conditions.**

The `huggingface/speech-to-speech` repository implements a sophisticated turn-taking system to manage overlapping voice activity detection results. At its core, the `SpeculativeTurnTracker` class in [`src/speech_to_speech/pipeline/speculative_turns.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/speculative_turns.py) maintains revision histories and synchronizes access to ensure that multiple in-flight responses—whether partial transcriptions or final results—resolve correctly even when they arrive out of order.

## Core Architecture of the SpeculativeTurnTracker

### Thread-Safe State Management

The tracker protects all state mutations behind a `Condition` variable (`_condition`), ensuring atomic updates across concurrent threads. It maintains four critical mappings to coordinate turn state:

- **`_latest_revision`** – Stores the most recent revision number observed for each turn. The `observe()` method updates this map when a new revision supersedes the previous one.
- **`_committed_revision`** – Tracks which revision has been accepted as final for a turn, accessed via `commit()` and `is_committed()`.
- **`_pending_reopen`** – Holds the candidate revision currently being evaluated for a speculative reopen. This map enforces the single-candidate rule.
- **`_reopen_grace`** – Manages a temporary window after confirmation during which late responses are ignored.

### Memory Pruning

To prevent unbounded memory growth, the `_prune_tracked_turns()` method automatically removes the oldest turn entries once the total count exceeds the configurable `max_tracked_turns` limit.

## Handling Concurrent In-Flight Responses

When multiple partial results arrive simultaneously, the system coordinates them through a strict protocol:

1. **Observation** – Each incoming response calls `tracker.observe(turn_id, revision)`. If the revision is newer than the stored value in `_latest_revision`, the method updates the map and `is_latest()` returns `True`.

2. **Candidate Initiation** – The first response that detects a potential turn reopen invokes `begin_reopen_candidate(turn_id, revision)`. If another response attempts to start a candidate for the same base revision while one is active, the method returns the existing `candidate_revision`. If the request targets a different revision while a candidate is pending, it returns `None`, preventing overlapping candidates.

3. **Non-Blocking Queries** – Callers that cannot afford to block use the `try_*` variants:
   - `try_is_latest_after_pending_reopen()` returns `None` while a candidate is pending, otherwise a boolean.
   - `try_commit_if_latest_after_pending_reopen()` attempts a commit and returns `None` if a pending reopen exists.

4. **Confirmation or Cancellation** – Once the system validates the reopen, it calls `confirm_reopen_candidate()`, which atomically updates `_latest_revision`, removes the pending entry, and notifies waiting threads via the condition variable. If the reopen condition fails, `cancel_reopen_candidate()` clears the pending entry.

5. **Grace Window** – After confirmation, `start_reopen_grace()` opens a brief timeout window. During this period, `is_latest_after_reopen_grace()` and its `try_*` counterpart block or return `None`, ensuring that late-arriving partial results do not overwrite the committed revision.

6. **Safe Commit** – The `commit_if_latest_after_pending_reopen()` method only finalizes a turn after the pending candidate resolves—either through confirmation or cancellation—guaranteeing that a commit never races against an in-flight reopen.

## Implementation Example in VADHandler

The `VADHandler` class in [`src/speech_to_speech/VAD/vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/vad_handler.py) demonstrates the typical consumer pattern:

```python

# Inside VADHandler._ensure_turn_for_speech_start()

turn_id, rev = self._current_turn_id, self._current_turn_revision

# 1. Observe the incoming audio fragment

self.speculative_turns.observe(turn_id, rev)

# 2. If we think the turn should be reopened, start a candidate

candidate = self.speculative_turns.begin_reopen_candidate(turn_id, rev)

if candidate is not None:
    # 3. Confirm the candidate once we have enough evidence

    if self._enough_audio_for_reopen():
        self.speculative_turns.confirm_reopen_candidate(turn_id, rev, candidate)
    else:
        # 4. Or cancel it if the condition disappears

        self.speculative_turns.cancel_reopen_candidate(turn_id, candidate)

# 5. When committing the final result, wait for any pending reopen

self.speculative_turns.commit_if_latest_after_pending_reopen(turn_id, rev)

```

This pattern appears throughout the test suite, including `test_vad_direct_reopen_path_uses_tracker_candidate_protocol` in [`tests/test_speculative_turns.py`](https://github.com/huggingface/speech-to-speech/blob/main/tests/test_speculative_turns.py), which validates the correct behavior under concurrent access.

## Summary

- The `SpeculativeTurnTracker` class in [`src/speech_to_speech/pipeline/speculative_turns.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/speculative_turns.py) uses a `Condition` lock and atomic revision maps to coordinate multiple in-flight responses safely.
- **Only one pending reopen candidate** is permitted per turn; subsequent attempts receive `None` or the existing candidate, preventing overlapping speculative reopens.
- Non-blocking `try_*` methods allow callers to query state without waiting for pending operations to resolve.
- The system protects committed revisions through a grace period mechanism and prunes stale state based on the `max_tracked_turns` configuration.
- Integration tests in [`tests/test_speculative_turns.py`](https://github.com/huggingface/speech-to-speech/blob/main/tests/test_speculative_turns.py) and OpenAI-compatible realtime endpoints validate these thread-safety guarantees under load.

## Frequently Asked Questions

### What happens when two responses attempt to reopen the same turn simultaneously?

The first caller to invoke `begin_reopen_candidate()` creates an entry in `_pending_reopen` and receives a candidate revision. Subsequent callers targeting the same base revision receive the existing candidate identifier, while those targeting different revisions receive `None`. This ensures that only one speculative reopen candidate exists per turn at any time.

### How does the tracker prevent memory leaks during long-running sessions?

The `_prune_tracked_turns()` method automatically removes the oldest entries from `_latest_revision`, `_committed_revision`, and other internal maps once the count exceeds the configurable `max_tracked_turns` parameter. This bounds memory usage regardless of how many turns occur during the session.

### Can a response commit while another is still evaluating a reopen candidate?

No. The `commit_if_latest_after_pending_reopen()` method blocks until any pending candidate is resolved via `confirm_reopen_candidate()` or `cancel_reopen_candidate()`. The non-blocking variant, `try_commit_if_latest_after_pending_reopen()`, returns `None` immediately if a candidate is pending, allowing the caller to defer the commit operation without blocking the thread.

### What is the purpose of the reopen grace period?

The `start_reopen_grace()` method opens a brief window after confirming a reopen candidate. During this window, `is_latest_after_reopen_grace()` prevents late-arriving partial results from overwriting the newly confirmed revision. This absorbs network jitter and processing delays that might cause stale responses to arrive after the system has already accepted a speculative reopen.