How to Enable Live Transcription with Partial Transcript Streaming in Hugging Face Speech-to-Speech

Enabling live transcription requires setting --enable_live_transcription and optionally adjusting --live_transcription_update_interval to control how frequently partial transcripts emit from the STT handler to the UI.

Live transcription provides real-time feedback while users speak, displaying partial results before final transcription completes. The huggingface/speech-to-speech library implements this through a progressive streaming architecture built into the Parakeet TDT STT handler, with configurable update intervals and clean separation between preview and final results.

Core Components of Live Transcription

Live transcription relies on three interconnected components that flow from audio input through to client display.

Module Arguments (Configuration Layer)

In src/speech_to_speech/arguments_classes/module_arguments.py, the ModuleArguments class defines the CLI flags that control live transcription behavior:

• --enable_live_transcription — Boolean flag to toggle the feature (default: True)
• --live_transcription_update_interval — Float specifying seconds between partial updates (default: 0.5)

These arguments propagate through s2s_pipeline.py into the pipeline configuration.

Parakeet TDT Handler (Generation Layer)

The ParakeetTDTSTTHandler in src/speech_to_speech/STT/parakeet_tdt_handler.py implements the actual progressive streaming. When enable_live_transcription is True and incoming audio has mode == "progressive" (indicating ongoing speech from the VAD), the handler instantiates a SmartProgressiveStreamingHandler that produces partial transcripts at the configured interval.

The handler's process() method yields PartialTranscription objects containing:

• text: The incremental transcript chunk
• turn_id: Identifier for the current speech turn
• turn_revision: Sequence counter for ordering partial results

Transcription Notifier (Dispatch Layer)

src/speech_to_speech/STT/transcription_notifier.py receives PartialTranscription objects and converts them into PartialTranscriptionEvent instances placed on text_output_queue. This ensures the realtime service receives properly formatted events without blocking the STT handler.

Architectural Flow

VAD Audio ──► ParakeetTDTSTTHandler ──► PartialTranscription
                                              │
                                              ▼
                         TranscriptionNotifier (event conversion)
                                              │
                                              ▼
                    RealtimeService / WebSocket ──► Client UI
                                              │
                    (final result arrives)    ▼
                         TranscriptionCompletedEvent

Critical design constraint: PartialTranscription messages route only to the UI, not to the LLM. This keeps language model payloads compact by sending complete transcripts only upon finalization.

Step-by-Step Activation

1. Enable via Command Line

Launch with explicit flags for fine-grained control:

python -m speech_to_speech \
    --enable_live_transcription \
    --live_transcription_update_interval 0.3 \
    --stt_handler parakeet_tdt

The --stt_handler parakeet_tdt selection is required—this handler contains the progressive streaming implementation.

2. Enable Programmatically

from speech_to_speech.s2s_pipeline import s2s_pipeline
from speech_to_speech.arguments_classes.module_arguments import ModuleArguments

args = ModuleArguments(
    enable_live_transcription=True,
    live_transcription_update_interval=0.5,  # 500ms updates

)

pipeline = s2s_pipeline(args)
pipeline.run()

The s2s_pipeline() function automatically injects these flags into handler initialization.

3. Consume Partial Events Client-Side

WebSocket clients receive structured events with type discrimination:

socket.addEventListener('message', (event) => {
    const data = JSON.parse(event.data);
    
    if (data.type === 'partial_transcription') {
        // Render incremental text: data.delta contains new content
        appendToLiveDisplay(data.delta);
    } else if (data.type === 'transcription_completed') {
        // Replace live preview with finalized transcript
        finalizeDisplay(data.transcript);
        clearLiveIndicator();
    }
});

Implementing Custom STT Handlers with Live Transcription

When extending BaseSTTHandler, mirror the Parakeet TDT pattern:

from speech_to_speech.STT.base_stt_handler import BaseSTTHandler
from speech_to_speech.STT.transcription_messages import PartialTranscription

class CustomSTTHandler(BaseSTTHandler):
    def __init__(self, enable_live_transcription=False, update_interval=0.5, **kwargs):
        super().__init__(**kwargs)
        self.enable_live_transcription = enable_live_transcription
        self.update_interval = update_interval
        self._streaming_handler = None  # Your progressive implementation

        
    def process(self, vad_audio):
        # Progressive path: user still speaking

        if self.enable_live_transcription and vad_audio.mode == "progressive":
            partial = self._streaming_handler.transcribe_chunk(vad_audio.audio)
            if partial:
                yield PartialTranscription(
                    text=partial,
                    turn_id=vad_audio.turn_id,
                    turn_revision=vad_audio.turn_revision,
                )
            return
            
        # Final path: speech segment complete

        final = self.transcribe_complete(vad_audio.audio)
        yield final

Key requirements:

• Check both enable_live_transcription flag and vad_audio.mode == "progressive"
• Include turn_id and turn_revision for event ordering
• Yield PartialTranscription only; let TranscriptionNotifier handle event conversion

Configuration Reference

Parameter	Default	Valid Range	Effect
enable_live_transcription	True	Boolean	Master toggle for progressive streaming
live_transcription_update_interval	0.5	> 0.0 seconds	Frequency of partial transcript emission

Lower intervals improve perceived responsiveness but increase message volume. Values below 0.1 may cause UI thrashing; values above 1.0 reduce the "live" feel.

Source File Locations

File	Purpose
src/speech_to_speech/arguments_classes/module_arguments.py	CLI argument definitions for live transcription flags
src/speech_to_speech/STT/parakeet_tdt_handler.py	Primary implementation with SmartProgressiveStreamingHandler
src/speech_to_speech/STT/transcription_notifier.py	Event conversion from PartialTranscription to PartialTranscriptionEvent
src/speech_to_speech/STT/transcription_messages.py	Message class definitions (PartialTranscription, TranscriptionCompletedEvent)
src/speech_to_speech/s2s_pipeline.py	Pipeline assembly and flag propagation

Summary

• Enable live transcription with --enable_live_transcription or ModuleArguments(enable_live_transcription=True)
• Adjust refresh rate via --live_transcription_update_interval (default 500ms)
• Requires Parakeet TDT handler — other STT handlers must implement progressive streaming manually
• Partial results never reach the LLM — they exist solely for UI feedback
• Final transcription triggers cleanup — handlers reset state and emit TranscriptionCompletedEvent

Frequently Asked Questions

Can I use live transcription with Whisper or other STT handlers?

No. According to the huggingface/speech-to-speech source code, only the Parakeet TDT handler implements SmartProgressiveStreamingHandler with progressive audio support. Whisper and other handlers emit complete transcripts only after silence detection. To add live transcription to another handler, implement the BaseSTTHandler pattern shown above with incremental transcription logic.

What happens if I disable live transcription mid-conversation?

The enable_live_transcription flag is global and pipeline-scoped, not per-turn. Disabling it requires pipeline restart—there is no runtime toggle. However, the VAD can suppress progressive mode for individual turns, effectively falling back to final-only transcription when the user pauses.

Why are partial transcripts not sent to the language model?

The architecture intentionally isolates partial transcripts from LLM input to prevent context fragmentation and reduce token costs. PartialTranscription objects route through text_output_queue to realtime consumers only, while TranscriptionCompletedEvent triggers the full LLM request with complete, finalized text.

How do I handle connection drops during live transcript streaming?

The turn_id and turn_revision fields in PartialTranscription enable client-side reconstruction. On reconnection, request the current turn_id from the server and discard any cached partials with mismatched IDs. The demo implementation in demo/README.md shows optional server-side buffering for brief disconnections.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}