What Is the Speculative Turns Feature for Handling Turn Revisions in Speech-to-Speech?

The speculative turns feature is a thread-safe revision tracker that enables real-time speech-to-speech pipelines to manage transient audio "turns" that may be revised or superseded while processing is ongoing, ensuring only the latest valid revision is committed downstream.

The speculative turns mechanism in the huggingface/speech-to-speech repository solves a critical latency problem in conversational AI: when a voice activity detector (VAD) reopens a supposedly finished utterance, the system must decide whether to discard already-processed output or risk showing stale data. By speculatively tracking multiple revisions of the same conversational turn, the pipeline maintains low latency while guaranteeing correctness.

Core Architecture of the Speculative Turn Tracker

At the heart of this feature lies the SpeculativeTurnTracker class defined in [src/speech_to_speech/pipeline/speculative_turns.py](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/speculative_turns.py#L24-L46). This thread-safe structure coordinates state across four internal data structures protected by a Condition lock:

• _latest_revision (OrderedDict): Tracks the most recent revision number for each active turn_id.
• _committed_revision (dict): Stores the highest revision safely emitted to downstream handlers.
• _pending_reopen (dict): Holds pending reopen requests when a later revision is expected but not yet confirmed.
• _reopen_grace (dict): Implements a grace period allowing downstream components to finish processing old revisions before the turn officially updates.

Why "Speculative" Processing Matters

When speech is captured in real-time, the VAD may determine that audio after a pause still belongs to the previous utterance. Rather than immediately invalidating processed output, the pipeline speculatively assumes an upcoming revision will replace the current one. This design enables:

1. Non-blocking processing: Handlers continue working on the current revision while awaiting potential newer revisions.
2. Graceful updates: A configurable grace window gives language models and TTS engines time to finish rendering before accepting replacements.
3. Deterministic commits: Only the latest revision at the moment of commit is accepted, preventing outdated audio from reaching users.

The Speculative Reopen Workflow

The tracker manages turn revisions through a strict lifecycle implemented across lines 38-92 of the core module:

1. Observation: As audio frames arrive, handlers call tracker.observe(turn_id, revision) to record the newest revision.

2. Reopen Candidate: When VAD detects possible continuation, begin_reopen_candidate(turn_id, revision) registers a pending candidate (revision + 1) in _pending_reopen.

3. Confirmation: Once new audio is confirmed, confirm_reopen_candidate(turn_id, base_revision, candidate_revision) updates _latest_revision, clears the pending entry, and notifies waiting threads.

4. Grace Period: If downstream handlers need time to finish, start_reopen_grace(turn_id, revision, grace_s) creates a deadline after which older revisions become obsolete.

5. Commit: Handlers call commit_if_latest_after_reopen_grace(turn_id, revision) to atomically record a revision only if it remains the latest.

6. Pruning: The _prune_tracked_turns() method (lines 46-47) automatically discards oldest entries when tracked turns exceed _MAX_TRACKED_TURNS (default 2048), bounding memory usage.

All waiting logic uses bounded timeouts (default 2 seconds) via private helpers like _wait_for_pending_reopen_locked and _wait_for_reopen_gate_locked, ensuring threads return None rather than deadlock when reopens are pending.

Integration Points in the Speech-to-Speech Pipeline

The tracker is instantiated per pipeline unit in [src/speech_to_speech/s2s_pipeline.py](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py#L45-L46) and injected into multiple handlers to coordinate the entire VAD → STT → LM → TTS chain.

VAD Handler Integration

In [src/speech_to_speech/VAD/vad_handler.py](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/vad_handler.py), the VAD passes the speculative_turns tracker to signal possible reopen events:

class VADHandler:
    def __init__(self, speculative_turns: SpeculativeTurnTracker):
        self.tracker = speculative_turns
        self.current_turn = "turn_1"
        self.revision = 0

    def on_speech_detected(self, audio):
        # Record the latest revision each time voice activity continues

        self.tracker.observe(self.current_turn, self.revision)

    def on_possible_reopen(self):
        # VAD thinks the utterance may continue – request a speculative reopen

        cand = self.tracker.begin_reopen_candidate(self.current_turn, self.revision)
        if cand:
            # Wait for the new audio segment…

            new_audio = self.wait_for_next_chunk()
            # Confirm the reopen once we have the extra audio

            self.tracker.confirm_reopen_candidate(
                self.current_turn, self.revision, cand
            )
            self.revision = cand

STT and Language Model Coordination

Each STT handler in src/speech_to_speech/STT/ receives the speculative_turns parameter to determine whether a transcription should replace a previous version. Similarly, [src/speech_to_speech/LM/lm_output_processor.py](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LM/lm_output_processor.py) checks the tracker before forwarding language model responses, guaranteeing only the newest turn revision reaches the TTS stage.

Practical Implementation Examples

Basic Tracker Usage

Create and manage speculative turns using the core API:

from speech_to_speech.pipeline.speculative_turns import SpeculativeTurnTracker

# Create a tracker (one per pipeline)

tracker = SpeculativeTurnTracker()

# 1️⃣ Record the first revision of a turn

turn_id = "turn_1"
tracker.observe(turn_id, revision=0)

# 2️⃣ Begin a speculative reopen – expect a newer revision

candidate_rev = tracker.begin_reopen_candidate(turn_id, revision=0)
print(f"Pending candidate revision: {candidate_rev}")   # → 1

# 3️⃣ Verify current revision is still latest

assert tracker.is_latest(turn_id, 0)  # True

# 4️⃣ Confirm the reopen once new audio chunk arrives

confirmed = tracker.confirm_reopen_candidate(
    turn_id, base_revision=0, candidate_revision=1
)
print(f"Reopen confirmed: {confirmed}")   # → True

# 5️⃣ Start grace period (e.g., give UI time to finish rendering)

tracker.start_reopen_grace(turn_id, revision=1, grace_s=0.8)

# 6️⃣ Commit only if revision 1 is still the latest

if tracker.commit_if_latest_after_reopen_grace(turn_id, revision=1):
    print("Revision 1 committed")
else:
    print("Revision 1 was superseded")

Non-Blocking "Try" API

Avoid deadlocks in high-throughput handlers using the non-blocking variant:

# Check if safe to proceed without blocking

if tracker.try_is_latest_after_pending_reopen(turn_id, revision) is None:
    # A pending reopen is still in flight – retry later

    schedule_retry()
else:
    # Safe to proceed with current revision

    process_turn(turn_id, revision)

This pattern prevents pipeline stalls when VAD uncertainty creates temporary revision conflicts.

Summary

• The speculative turns feature uses a SpeculativeTurnTracker class to manage real-time revisions of conversational turns without blocking the pipeline.
• Thread-safe state tracking across _latest_revision, _committed_revision, _pending_reopen, and _reopen_grace enables deterministic commit semantics.
• The reopen workflow (observe → begin candidate → confirm → grace period → commit) allows VAD, STT, and LM handlers to process audio speculatively while maintaining correctness.
• Integration spans [src/speech_to_speech/s2s_pipeline.py](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py), VAD handlers, STT modules, and LM output processors to coordinate the entire speech-to-speech chain.
• Memory usage is bounded by automatic pruning at _MAX_TRACKED_TURNS (2048 entries), with all blocking operations subject to 2-second timeouts.

Frequently Asked Questions

How does the speculative turns feature prevent showing outdated audio to users?

The tracker requires handlers to call commit_if_latest_after_reopen_grace() before emitting output. This method atomically verifies the revision is still current after any grace periods expire. If a newer revision arrived during processing, the commit fails and the outdated audio is discarded, ensuring users only see the most recent valid transcription.

What happens when the VAD detects a reopen but no new audio arrives?

If begin_reopen_candidate() registers a pending reopen that never gets confirmed via confirm_reopen_candidate(), the tracker leaves the revision in _pending_reopen. Subsequent calls to try_is_latest_after_pending_reopen() return None, signaling handlers to retry later. The 2-second timeout on blocking operations ensures the system eventually resolves stale pending states without permanent deadlock.

Why use an OrderedDict for _latest_revision instead of a regular dictionary?

The OrderedDict maintains insertion order, enabling the _prune_tracked_turns() method to efficiently discard the oldest entries when the tracker exceeds _MAX_TRACKED_TURNS (2048). This FIFO eviction strategy ensures memory usage remains bounded while preserving recent turn history necessary for speculative processing.

Can multiple handlers commit different revisions of the same turn simultaneously?

No. The Condition lock protecting all internal structures ensures atomic commit semantics. When handlers call commit_if_latest_after_reopen_grace(), the method acquires the lock, verifies the revision is still latest, updates _committed_revision, and releases the lock. This serializes commits so only the first valid commit succeeds; subsequent attempts on superseded revisions fail safely.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}