# Optimizing Latency with Speculative Turn Processing and Barge-In Handling in Speech-to-Speech

> Optimize speech-to-speech latency with speculative turn processing and barge-in handling. Achieve sub-second responses in real-time dialogue. Learn how at huggingface/speech-to-speech.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: performance
- Published: 2026-08-09

---

**The huggingface/speech-to-speech repository minimizes conversational latency by speculating on upcoming user turns and rapidly discarding stale output when users interrupt, achieving sub-second response times in real-time dialogue.**

Optimizing latency with speculative turn processing and barge-in handling is critical for building responsive voice AI systems. The `huggingface/speech-to-speech` codebase implements a sophisticated concurrency model that processes audio chunks speculatively while maintaining the ability to instantly cancel outdated responses when users interject. This architecture enables full-duplex conversation where the system predicts user intent without waiting for turn completion.

## How Speculative Turn Processing Reduces Latency

### The SpeculativeTurnTracker Architecture

At the core of the latency optimization lies the `SpeculativeTurnTracker` class defined in [`src/speech_to_speech/pipeline/speculative_turns.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/speculative_turns.py). This thread-safe revision manager tracks the latest revision of each audio turn and allows the system to create a *reopen* candidate while the current turn is still being processed.

The tracker is instantiated per pipeline unit in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) (line 79) and injected into every handler via the `HandlerContext`. For example, the VAD handler receives the `speculative_turns` instance at construction (lines 82‑84), enabling distributed access across the processing chain.

### Key Methods for Turn Management

The tracker exposes several atomic operations that handlers invoke as audio chunks flow through the pipeline:

- **`observe(turn_id, revision)`** – Records the newest revision for a turn. Called by VAD, STT, LM, and TTS handlers as chunks are processed.
- **`begin_reopen_candidate(turn_id, revision)`** – Reserves the next revision as a possible speculative turn when the system predicts the user may start speaking before the previous turn commits.
- **`commit_if_latest_after_pending_reopen(turn_id, revision)`** – Atomically commits a revision only if it remains the newest after any pending reopen resolves, preventing race conditions during finalization.
- **`has_pending_reopen_or_grace(turn_id)`** – Queries whether a turn has an unresolved reopen or active grace period, allowing downstream components to defer output until the turn state stabilizes.

### Parallel Processing Benefits

By decoupling turn detection from response generation, the system achieves **parallel processing** gains. While the language model streams a response, the VAD can detect new speech segments and initialize a speculative turn via `begin_reopen_candidate`. If the user confirms the new turn, the system discards the stale LM output immediately rather than waiting for the full generation to complete. A configurable grace period managed by `start_reopen_grace` filters transient noise, ensuring that brief interruptions do not trigger premature turn closures.

## Handling Barge-In Events for Immediate Interruption

### Client-Side Event Processing

Barge-in handling operates primarily within [`src/speech_to_speech/api/openai_realtime/websocket_router.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/websocket_router.py). The router processes two critical event types:

1. **`ResponseCancelEvent`** – Signals an explicit cancellation request.
2. **`SpeechStartedEvent`** – Indicates the user has begun speaking, potentially interrupting the current response.

When `SpeechStartedEvent` arrives (lines 66‑87), the code evaluates the runtime configuration to determine if interruption is permitted. If allowed, the system flushes both audio and text output queues, clears the `response_playing` flag, and logs the interruption (lines 73‑85). The `_send_loop_for` function mirrors this logic (lines 66‑81), ensuring that in-flight audio is discarded promptly regardless of which component detects the barge-in.

### Queue Flushing with Sentinel Preservation

The flush mechanism preserves essential control messages to maintain session integrity. Two helper functions determine which items survive the purge:

```python
def _keep_audio_sentinel(item: Any) -> bool:
    # Ensure SESSION_END survives a barge-in flush.

    return _is_audio_done(item) or is_control_message(item, SESSION_END.kind)

def _keep_user_text_event(item: Any) -> bool:
    # Preserve user-visible text events during a flush.

    return isinstance(item, (SpeechStoppedEvent,
                             PartialTranscriptionEvent,
                             TranscriptionCompletedEvent,
                             AudioInputCompletedEvent,
                             TokenUsageEvent))

```

The `_flush_queue` implementation (lines 10‑17) removes all items from the queue while collecting preserved events, then re-inserts them at the **front** of the queue using `appendleft`. This guarantees that sentinel markers and partial transcriptions are processed before any new items handlers might enqueue during the flush window.

## End-to-End Latency Optimization Flow

The complete latency reduction pipeline follows this sequence:

1. **Audio ingestion** – Incoming audio triggers the VAD, which calls `speculative_turns.observe(turn_id, rev)` to register the initial revision.
2. **Speculative initialization** – As STT processes chunks, it may invoke `begin_reopen_candidate` if it detects prosodic cues suggesting the user will speak again before the current turn finalizes.
3. **Response generation** – The LM streams output events into the queue while the TTS handler checks `has_pending_reopen_or_grace` to avoid emitting audio that might be immediately discarded.
4. **Barge-in detection** – A `SpeechStartedEvent` triggers the flush logic in [`websocket_router.py`](https://github.com/huggingface/speech-to-speech/blob/main/websocket_router.py), preserving the `SESSION_END` sentinel and user text events while discarding stale audio.
5. **Atomic commitment** – Once the reopen is confirmed, `commit_if_latest_after_pending_reopen` finalizes the new revision, allowing the pipeline to commit to the new turn state.

This flow enables the system to **speculate** on upcoming turns while maintaining the ability to **immediately discard** obsolete output, achieving sub-second latency in full-duplex scenarios.

## Implementation Example

The following demonstrates initializing the tracker and handling barge-in events:

```python
from speech_to_speech.pipeline.speculative_turns import SpeculativeTurnTracker
from queue import Queue, Empty

# 1. Create a speculative turn tracker (per pipeline unit)

tracker = SpeculativeTurnTracker(max_tracked_turns=2048)

# 2. Record a new revision for a turn (called by VAD/STT)

turn_id = "turn-42"
revision = 3
tracker.observe(turn_id, revision)

# 3. Begin a speculative reopen candidate

candidate_rev = tracker.begin_reopen_candidate(turn_id, revision)

# Returns revision + 1 if allowed, otherwise None

# 4. Commit only if still the newest revision

if tracker.commit_if_latest_after_pending_reopen(turn_id, revision):
    print("Turn committed – safe to emit response")
else:
    print("Turn superseded – discard output")

# 5. Barge-in handling – flush queues while preserving essentials

def _flush_queue(q: Queue, *, preserve=None):
    preserved = []
    while True:
        try:
            item = q.get_nowait()
            if preserve and preserve(item):
                preserved.append(item)
        except Empty:
            break
    if preserved:
        with q.mutex:
            for i in reversed(preserved):
                q.queue.appendleft(i)
            q.not_empty.notify(len(preserved))

# Usage during interrupt:

_flush_queue(unit.output_queue, preserve=_keep_audio_sentinel)
_flush_queue(unit.text_output_queue, preserve=_keep_user_text_event)

```

## Summary

- **SpeculativeTurnTracker** in [`src/speech_to_speech/pipeline/speculative_turns.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/speculative_turns.py) provides thread-safe revision management that allows parallel processing of predicted user turns.
- **Barge-in handling** in [`websocket_router.py`](https://github.com/huggingface/speech-to-speech/blob/main/websocket_router.py) implements immediate queue flushing with sentinel preservation to prevent session state corruption during interruptions.
- **Atomic commit operations** ensure that only the latest revision is finalized, eliminating race conditions when users interrupt mid-response.
- **Grace periods** managed by `start_reopen_grace` filter transient noise while maintaining responsiveness to genuine interruptions.

## Frequently Asked Questions

### How does the SpeculativeTurnTracker prevent race conditions during turn finalization?

The tracker uses `commit_if_latest_after_pending_reopen` to perform an atomic comparison of the requested revision against the current state. If a speculative reopen occurred while the handler was processing, the commit fails and the handler discards its output, ensuring only the most recent turn state is ever finalized.

### What happens to audio already queued for playback when a barge-in occurs?

The `_send_loop_for` function in [`websocket_router.py`](https://github.com/huggingface/speech-to-speech/blob/main/websocket_router.py) immediately flushes the `output_queue` when it detects a `SpeechStartedEvent`. The flush preserves only the `SESSION_END` sentinel (via `_keep_audio_sentinel`) and user-visible text events, discarding all pending audio chunks to prevent the user from hearing the bot speak over them.

### Why is the SpeculativeTurnTracker instantiated per pipeline unit rather than globally?

Instantiating the tracker per unit in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) (line 79) ensures isolation between concurrent conversation sessions. Each pipeline unit maintains its own turn state through the `HandlerContext`, preventing cross-contamination of revision numbers and allowing independent speculation strategies for different users or conversation contexts.

### How does the system distinguish between genuine barge-in and background noise?

The `start_reopen_grace` method establishes a short timeout window after a potential reopen event. During this grace period, the system waits for confirmation from the VAD and STT handlers before committing to the new turn. Transient sounds that do not generate sustained speech activity expire within the grace period, allowing the original turn to proceed without interruption.