# How Speculative Turns Allow Early LLM Processing in Real-Time Speech-to-Speech Systems

> Discover how speculative turns enable early LLM processing in real-time speech-to-speech systems by starting inference on partial transcriptions to reduce latency.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: deep-dive
- Published: 2026-08-06

---

**Speculative turns reduce latency by letting the pipeline start LLM inference on partial transcriptions while a user is still speaking, then discarding the output if the transcription later changes.**

Real-time speech-to-speech systems face a fundamental tension: users want instant responses, but the transcription of their speech isn't finalized until they stop talking. The **huggingface/speech-to-speech** repository solves this with **speculative turns**—a mechanism that pipelines the LLM stage early while guaranteeing correctness through revision tracking and grace-period validation.

## What Problem Speculative Turns Solve

Without speculation, the pipeline would wait for Voice Activity Detection (VAD) to confirm a turn is complete before invoking the Speech-to-Text (STT) and Large Language Model (LLM) stages. This sequential processing adds unnecessary latency. With **speculative turns**, the system starts LLM processing on the current best-guess transcription immediately, then validates whether that guess was correct before emitting any output to the user.

The implementation centers on `SpeculativeTurnTracker` in [`src/speech_to_speech/pipeline/speculative_turns.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/speculative_turns.py).

## Core Mechanism: Revision Tracking

Every audio turn receives unique identification and version control:

- **Turn ID**: A UUID (`turn_id`) that persists across revisions of the same logical turn
- **Revision counter**: An integer (`turn_revision`) incremented each time the VAD or STT updates the transcription

This pairing allows the system to distinguish between "same turn, new information" versus "entirely new turn."

## The Speculative Reopen Workflow

The tracker coordinates six distinct phases through specific method calls:

| Phase | Trigger | Tracker Method |
|-------|---------|----------------|
| **Observation** | New audio arrives for a turn | `observe(turn_id, revision)` |
| **Reopen candidate** | VAD detects possible pause, may resume | `begin_reopen_candidate(base_revision, candidate_revision)` |
| **Grace period** | Audio stops, waiting to confirm completion | `start_reopen_grace(turn_id, revision, grace_s)` |
| **Confirmation** | Turn genuinely finished, accept newer revision | `confirm_reopen_candidate()` |
| **Cancellation** | Turn was noise/trailing audio, revert | `cancel_reopen_candidate()` |
| **Commit attempt** | LLM/TTS wants to emit output | `commit_if_latest_after_reopen_grace()` |

The **reopen grace window** (`speculative_reopen_ms`, default 800ms) is the critical parameter. During this window, LLM output is generated speculatively. If a newer revision arrives before expiration, the speculative work is discarded. If the grace expires with no new revisions, the commit succeeds.

## Integration Across Pipeline Components

### VAD Handler ([`src/speech_to_speech/VAD/vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/vad_handler.py))

The VAD drives the speculative state machine. When audio arrives:

```python

# vad_handler.py – tracking an active turn

if self.speculative_turns:
    self.speculative_turns.observe(self._current_turn_id,
                                   self._current_turn_revision)

```

When the VAD suspects the user paused:

```python

# Starting the grace period before finalizing

if self.speculative_turns:
    self.speculative_turns.start_reopen_grace(
        self._current_turn_id,
        self._current_turn_revision,
        grace_s=self._speculative_reopen_ms / 1000.0,
    )

```

### LLM Response Handler ([`src/speech_to_speech/api/openai_realtime/handlers/response.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/handlers/response.py))

Generated tokens don't reach the user until validated:

```python

# response.py – conditional commit after generation

if self._service.speculative_turns:
    commit_ok = self._service.speculative_turns.commit_if_latest_after_reopen_grace(
        turn_id, turn_revision
    )
    if not commit_ok:
        # Turn was superseded by newer revision; discard output

        return

```

### Service Layer ([`src/speech_to_speech/api/openai_realtime/service.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/service.py))

Resets tracker state on session boundaries and gates event forwarding based on `is_latest_after_reopen_grace()` checks.

### Pipeline Assembly ([`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py))

A single tracker instance shares state across all components:

```python

# s2s_pipeline.py – dependency injection

speculative_turns = SpeculativeTurnTracker()   # one per session

vars(vad_kw)["speculative_turns"] = speculative_turns
vars(kw)["speculative_turns"] = speculative_turns   # LLM, TTS handlers

```

## Enabling and Configuring Speculative Turns

Activate via CLI flag with custom grace duration:

```bash
speech-to-speech \
  --mode realtime \
  --speculative_reopen_ms 800

```

The value propagates through `VADHandlerArguments` to `VADHandler.__init__` and ultimately to `start_reopen_grace()` as seconds.

## Thread Safety and Correctness Guarantees

`SpeculativeTurnTracker` uses internal locking to coordinate concurrent access:

- **VAD thread**: Updates revisions, starts grace periods
- **LLM threads**: Query latest status, attempt commits
- **TTS threads**: Same validation pattern as LLM

The `try_commit_if_latest_after_reopen_grace()` variant provides non-blocking behavior where appropriate.

Key correctness properties:

- A commit only succeeds if `is_latest_after_reopen_grace()` returns `True`
- Pending reopen candidates block premature finalization
- Revision monotonicity prevents time-of-check/time-of-use races

## Performance Characteristics

| Scenario | Behavior |
|----------|----------|
| User speaks continuously | LLM pipelines behind real-time transcription; minimal wasted work |
| User pauses briefly then resumes | Grace period absorbs hesitation; no restart needed |
| User stops definitively | Grace expires, commit proceeds with minimal final latency |
| Transcription corrects mid-speech | Older speculative LLM work discarded; fresh start on new revision |

The latency win comes from overlapping LLM computation with the final moments of user speech and the grace period itself—typically hundreds of milliseconds saved per turn.

## Summary

- **Speculative turns** start LLM processing on incomplete transcriptions to reduce perceived latency
- **Revision counters** (`turn_id` + `turn_revision`) track transcription stability across pipeline stages
- **Reopen grace periods** (`speculative_reopen_ms`) define the validation window for speculative work
- `SpeculativeTurnTracker` in [`src/speech_to_speech/pipeline/speculative_turns.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/speculative_turns.py) provides thread-safe coordination
- Commits through `commit_if_latest_after_reopen_grace()` guarantee only final-transcription-aligned output reaches users
- VAD, LLM, and TTS handlers all participate via dependency injection from [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py)

## Frequently Asked Questions

### What happens if the transcription changes after the LLM starts but before the grace period ends?

The tracker detects the revision mismatch via `is_latest`. When `commit_if_latest_after_reopen_grace()` is called, it returns `False` and the handler discards the generated output. The pipeline continues with the newer revision as the speculative base.

### How does `SpeculativeTurnTracker` handle concurrent access from multiple pipeline stages?

The implementation uses internal locks to protect revision state, pending reopen candidates, and grace period deadlines. Both blocking (`commit_if_latest_after_reopen_grace`) and non-blocking (`try_commit_if_latest_after_reopen_grace`) commit methods are provided for flexibility.

### Can speculative turns be disabled, and what is the latency cost?

Set `--speculative_reopen_ms 0` to disable. The system then waits for VAD finalization before starting LLM inference, adding approximately the full STT latency plus any VAD decision delay—typically 200-500ms additional per turn depending on audio characteristics.

### What's the relationship between reopen candidates and the grace period?

`begin_reopen_candidate` registers that a newer revision *might* arrive, creating a pending state. `start_reopen_grace` begins the actual timer during which speculative output can be validated. A candidate must be confirmed or cancelled; the grace period must expire for commits to proceed.