# What Is the Speculative Turns Feature for Handling Turn Revisions in Speech-to-Speech?

> Explore speculative turns for speech-to-speech pipelines. This feature efficiently manages audio turn revisions in real-time, ensuring only the latest valid audio is processed.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: deep-dive
- Published: 2026-08-07

---

**The speculative turns feature is a thread-safe revision tracker that enables real-time speech-to-speech pipelines to manage transient audio "turns" that may be revised or superseded while processing is ongoing, ensuring only the latest valid revision is committed downstream.**

The **speculative turns** mechanism in the `huggingface/speech-to-speech` repository solves a critical latency problem in conversational AI: when a voice activity detector (VAD) reopens a supposedly finished utterance, the system must decide whether to discard already-processed output or risk showing stale data. By speculatively tracking multiple revisions of the same conversational turn, the pipeline maintains low latency while guaranteeing correctness.

## Core Architecture of the Speculative Turn Tracker

At the heart of this feature lies the **`SpeculativeTurnTracker`** class defined in [[`src/speech_to_speech/pipeline/speculative_turns.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/speculative_turns.py)](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/speculative_turns.py#L24-L46). This thread-safe structure coordinates state across four internal data structures protected by a `Condition` lock:

- **`_latest_revision`** (OrderedDict): Tracks the most recent revision number for each active `turn_id`.
- **`_committed_revision`** (dict): Stores the highest revision safely emitted to downstream handlers.
- **`_pending_reopen`** (dict): Holds pending reopen requests when a later revision is expected but not yet confirmed.
- **`_reopen_grace`** (dict): Implements a grace period allowing downstream components to finish processing old revisions before the turn officially updates.

### Why "Speculative" Processing Matters

When speech is captured in real-time, the VAD may determine that audio after a pause still belongs to the previous utterance. Rather than immediately invalidating processed output, the pipeline **speculatively** assumes an upcoming revision will replace the current one. This design enables:

1. **Non-blocking processing**: Handlers continue working on the current revision while awaiting potential newer revisions.
2. **Graceful updates**: A configurable grace window gives language models and TTS engines time to finish rendering before accepting replacements.
3. **Deterministic commits**: Only the latest revision at the moment of commit is accepted, preventing outdated audio from reaching users.

## The Speculative Reopen Workflow

The tracker manages turn revisions through a strict lifecycle implemented across lines 38-92 of the core module:

1. **Observation**: As audio frames arrive, handlers call `tracker.observe(turn_id, revision)` to record the newest revision.

2. **Reopen Candidate**: When VAD detects possible continuation, `begin_reopen_candidate(turn_id, revision)` registers a pending candidate (revision + 1) in `_pending_reopen`.

3. **Confirmation**: Once new audio is confirmed, `confirm_reopen_candidate(turn_id, base_revision, candidate_revision)` updates `_latest_revision`, clears the pending entry, and notifies waiting threads.

4. **Grace Period**: If downstream handlers need time to finish, `start_reopen_grace(turn_id, revision, grace_s)` creates a deadline after which older revisions become obsolete.

5. **Commit**: Handlers call `commit_if_latest_after_reopen_grace(turn_id, revision)` to atomically record a revision only if it remains the latest.

6. **Pruning**: The `_prune_tracked_turns()` method (lines 46-47) automatically discards oldest entries when tracked turns exceed `_MAX_TRACKED_TURNS` (default 2048), bounding memory usage.

All waiting logic uses bounded timeouts (default 2 seconds) via private helpers like `_wait_for_pending_reopen_locked` and `_wait_for_reopen_gate_locked`, ensuring threads return `None` rather than deadlock when reopens are pending.

## Integration Points in the Speech-to-Speech Pipeline

The tracker is instantiated per pipeline unit in [[`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py)](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py#L45-L46) and injected into multiple handlers to coordinate the entire VAD → STT → LM → TTS chain.

### VAD Handler Integration

In [[`src/speech_to_speech/VAD/vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/vad_handler.py)](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/vad_handler.py), the VAD passes the `speculative_turns` tracker to signal possible reopen events:

```python
class VADHandler:
    def __init__(self, speculative_turns: SpeculativeTurnTracker):
        self.tracker = speculative_turns
        self.current_turn = "turn_1"
        self.revision = 0

    def on_speech_detected(self, audio):
        # Record the latest revision each time voice activity continues

        self.tracker.observe(self.current_turn, self.revision)

    def on_possible_reopen(self):
        # VAD thinks the utterance may continue – request a speculative reopen

        cand = self.tracker.begin_reopen_candidate(self.current_turn, self.revision)
        if cand:
            # Wait for the new audio segment…

            new_audio = self.wait_for_next_chunk()
            # Confirm the reopen once we have the extra audio

            self.tracker.confirm_reopen_candidate(
                self.current_turn, self.revision, cand
            )
            self.revision = cand

```

### STT and Language Model Coordination

Each STT handler in `src/speech_to_speech/STT/` receives the `speculative_turns` parameter to determine whether a transcription should replace a previous version. Similarly, [[`src/speech_to_speech/LM/lm_output_processor.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LM/lm_output_processor.py)](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LM/lm_output_processor.py) checks the tracker before forwarding language model responses, guaranteeing only the newest turn revision reaches the TTS stage.

## Practical Implementation Examples

### Basic Tracker Usage

Create and manage speculative turns using the core API:

```python
from speech_to_speech.pipeline.speculative_turns import SpeculativeTurnTracker

# Create a tracker (one per pipeline)

tracker = SpeculativeTurnTracker()

# 1️⃣ Record the first revision of a turn

turn_id = "turn_1"
tracker.observe(turn_id, revision=0)

# 2️⃣ Begin a speculative reopen – expect a newer revision

candidate_rev = tracker.begin_reopen_candidate(turn_id, revision=0)
print(f"Pending candidate revision: {candidate_rev}")   # → 1

# 3️⃣ Verify current revision is still latest

assert tracker.is_latest(turn_id, 0)  # True

# 4️⃣ Confirm the reopen once new audio chunk arrives

confirmed = tracker.confirm_reopen_candidate(
    turn_id, base_revision=0, candidate_revision=1
)
print(f"Reopen confirmed: {confirmed}")   # → True

# 5️⃣ Start grace period (e.g., give UI time to finish rendering)

tracker.start_reopen_grace(turn_id, revision=1, grace_s=0.8)

# 6️⃣ Commit only if revision 1 is still the latest

if tracker.commit_if_latest_after_reopen_grace(turn_id, revision=1):
    print("Revision 1 committed")
else:
    print("Revision 1 was superseded")

```

### Non-Blocking "Try" API

Avoid deadlocks in high-throughput handlers using the non-blocking variant:

```python

# Check if safe to proceed without blocking

if tracker.try_is_latest_after_pending_reopen(turn_id, revision) is None:
    # A pending reopen is still in flight – retry later

    schedule_retry()
else:
    # Safe to proceed with current revision

    process_turn(turn_id, revision)

```

This pattern prevents pipeline stalls when VAD uncertainty creates temporary revision conflicts.

## Summary

- The **speculative turns** feature uses a `SpeculativeTurnTracker` class to manage real-time revisions of conversational turns without blocking the pipeline.
- Thread-safe state tracking across `_latest_revision`, `_committed_revision`, `_pending_reopen`, and `_reopen_grace` enables deterministic commit semantics.
- The reopen workflow (observe → begin candidate → confirm → grace period → commit) allows VAD, STT, and LM handlers to process audio speculatively while maintaining correctness.
- Integration spans [[`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py)](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py), VAD handlers, STT modules, and LM output processors to coordinate the entire speech-to-speech chain.
- Memory usage is bounded by automatic pruning at `_MAX_TRACKED_TURNS` (2048 entries), with all blocking operations subject to 2-second timeouts.

## Frequently Asked Questions

### How does the speculative turns feature prevent showing outdated audio to users?

The tracker requires handlers to call `commit_if_latest_after_reopen_grace()` before emitting output. This method atomically verifies the revision is still current after any grace periods expire. If a newer revision arrived during processing, the commit fails and the outdated audio is discarded, ensuring users only see the most recent valid transcription.

### What happens when the VAD detects a reopen but no new audio arrives?

If `begin_reopen_candidate()` registers a pending reopen that never gets confirmed via `confirm_reopen_candidate()`, the tracker leaves the revision in `_pending_reopen`. Subsequent calls to `try_is_latest_after_pending_reopen()` return `None`, signaling handlers to retry later. The 2-second timeout on blocking operations ensures the system eventually resolves stale pending states without permanent deadlock.

### Why use an OrderedDict for `_latest_revision` instead of a regular dictionary?

The `OrderedDict` maintains insertion order, enabling the `_prune_tracked_turns()` method to efficiently discard the oldest entries when the tracker exceeds `_MAX_TRACKED_TURNS` (2048). This FIFO eviction strategy ensures memory usage remains bounded while preserving recent turn history necessary for speculative processing.

### Can multiple handlers commit different revisions of the same turn simultaneously?

No. The `Condition` lock protecting all internal structures ensures atomic commit semantics. When handlers call `commit_if_latest_after_reopen_grace()`, the method acquires the lock, verifies the revision is still latest, updates `_committed_revision`, and releases the lock. This serializes commits so only the first valid commit succeeds; subsequent attempts on superseded revisions fail safely.