How Speculative Turns Allow Early LLM Processing in Real-Time Speech-to-Speech Systems

Speculative turns reduce latency by letting the pipeline start LLM inference on partial transcriptions while a user is still speaking, then discarding the output if the transcription later changes.

Real-time speech-to-speech systems face a fundamental tension: users want instant responses, but the transcription of their speech isn't finalized until they stop talking. The huggingface/speech-to-speech repository solves this with speculative turns—a mechanism that pipelines the LLM stage early while guaranteeing correctness through revision tracking and grace-period validation.

What Problem Speculative Turns Solve

Without speculation, the pipeline would wait for Voice Activity Detection (VAD) to confirm a turn is complete before invoking the Speech-to-Text (STT) and Large Language Model (LLM) stages. This sequential processing adds unnecessary latency. With speculative turns, the system starts LLM processing on the current best-guess transcription immediately, then validates whether that guess was correct before emitting any output to the user.

The implementation centers on SpeculativeTurnTracker in src/speech_to_speech/pipeline/speculative_turns.py.

Core Mechanism: Revision Tracking

Every audio turn receives unique identification and version control:

• Turn ID: A UUID (turn_id) that persists across revisions of the same logical turn
• Revision counter: An integer (turn_revision) incremented each time the VAD or STT updates the transcription

This pairing allows the system to distinguish between "same turn, new information" versus "entirely new turn."

The Speculative Reopen Workflow

The tracker coordinates six distinct phases through specific method calls:

Phase	Trigger	Tracker Method
Observation	New audio arrives for a turn	observe(turn_id, revision)
Reopen candidate	VAD detects possible pause, may resume	begin_reopen_candidate(base_revision, candidate_revision)
Grace period	Audio stops, waiting to confirm completion	start_reopen_grace(turn_id, revision, grace_s)
Confirmation	Turn genuinely finished, accept newer revision	confirm_reopen_candidate()
Cancellation	Turn was noise/trailing audio, revert	cancel_reopen_candidate()
Commit attempt	LLM/TTS wants to emit output	commit_if_latest_after_reopen_grace()

The reopen grace window (speculative_reopen_ms, default 800ms) is the critical parameter. During this window, LLM output is generated speculatively. If a newer revision arrives before expiration, the speculative work is discarded. If the grace expires with no new revisions, the commit succeeds.

Integration Across Pipeline Components

VAD Handler (src/speech_to_speech/VAD/vad_handler.py)

The VAD drives the speculative state machine. When audio arrives:

# vad_handler.py – tracking an active turn

if self.speculative_turns:
    self.speculative_turns.observe(self._current_turn_id,
                                   self._current_turn_revision)

When the VAD suspects the user paused:

# Starting the grace period before finalizing

if self.speculative_turns:
    self.speculative_turns.start_reopen_grace(
        self._current_turn_id,
        self._current_turn_revision,
        grace_s=self._speculative_reopen_ms / 1000.0,
    )

LLM Response Handler (src/speech_to_speech/api/openai_realtime/handlers/response.py)

Generated tokens don't reach the user until validated:

# response.py – conditional commit after generation

if self._service.speculative_turns:
    commit_ok = self._service.speculative_turns.commit_if_latest_after_reopen_grace(
        turn_id, turn_revision
    )
    if not commit_ok:
        # Turn was superseded by newer revision; discard output

        return

Service Layer (src/speech_to_speech/api/openai_realtime/service.py)

Resets tracker state on session boundaries and gates event forwarding based on is_latest_after_reopen_grace() checks.

Pipeline Assembly (src/speech_to_speech/s2s_pipeline.py)

A single tracker instance shares state across all components:

# s2s_pipeline.py – dependency injection

speculative_turns = SpeculativeTurnTracker()   # one per session

vars(vad_kw)["speculative_turns"] = speculative_turns
vars(kw)["speculative_turns"] = speculative_turns   # LLM, TTS handlers

Enabling and Configuring Speculative Turns

Activate via CLI flag with custom grace duration:

speech-to-speech \
  --mode realtime \
  --speculative_reopen_ms 800

The value propagates through VADHandlerArguments to VADHandler.__init__ and ultimately to start_reopen_grace() as seconds.

Thread Safety and Correctness Guarantees

SpeculativeTurnTracker uses internal locking to coordinate concurrent access:

• VAD thread: Updates revisions, starts grace periods
• LLM threads: Query latest status, attempt commits
• TTS threads: Same validation pattern as LLM

The try_commit_if_latest_after_reopen_grace() variant provides non-blocking behavior where appropriate.

Key correctness properties:

• A commit only succeeds if is_latest_after_reopen_grace() returns True
• Pending reopen candidates block premature finalization
• Revision monotonicity prevents time-of-check/time-of-use races

Performance Characteristics

Scenario	Behavior
User speaks continuously	LLM pipelines behind real-time transcription; minimal wasted work
User pauses briefly then resumes	Grace period absorbs hesitation; no restart needed
User stops definitively	Grace expires, commit proceeds with minimal final latency
Transcription corrects mid-speech	Older speculative LLM work discarded; fresh start on new revision

The latency win comes from overlapping LLM computation with the final moments of user speech and the grace period itself—typically hundreds of milliseconds saved per turn.

Summary

• Speculative turns start LLM processing on incomplete transcriptions to reduce perceived latency
• Revision counters (turn_id + turn_revision) track transcription stability across pipeline stages
• Reopen grace periods (speculative_reopen_ms) define the validation window for speculative work
• SpeculativeTurnTracker in src/speech_to_speech/pipeline/speculative_turns.py provides thread-safe coordination
• Commits through commit_if_latest_after_reopen_grace() guarantee only final-transcription-aligned output reaches users
• VAD, LLM, and TTS handlers all participate via dependency injection from s2s_pipeline.py

Frequently Asked Questions

What happens if the transcription changes after the LLM starts but before the grace period ends?

The tracker detects the revision mismatch via is_latest. When commit_if_latest_after_reopen_grace() is called, it returns False and the handler discards the generated output. The pipeline continues with the newer revision as the speculative base.

How does SpeculativeTurnTracker handle concurrent access from multiple pipeline stages?

The implementation uses internal locks to protect revision state, pending reopen candidates, and grace period deadlines. Both blocking (commit_if_latest_after_reopen_grace) and non-blocking (try_commit_if_latest_after_reopen_grace) commit methods are provided for flexibility.

Can speculative turns be disabled, and what is the latency cost?

Set --speculative_reopen_ms 0 to disable. The system then waits for VAD finalization before starting LLM inference, adding approximately the full STT latency plus any VAD decision delay—typically 200-500ms additional per turn depending on audio characteristics.

What's the relationship between reopen candidates and the grace period?

begin_reopen_candidate registers that a newer revision might arrive, creating a pending state. start_reopen_grace begins the actual timer during which speculative output can be validated. A candidate must be confirmed or cancelled; the grace period must expire for commits to proceed.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}