How to Enable Live Transcription During Voice Conversations with Hugging Face Speech-to-Speech

Enable live transcription in Hugging Face Speech-to-Speech by setting enable_live_transcription=True in your ModuleArguments, which activates realtime VAD buffering and streams partial transcriptions through TranscriptionNotifier to your client.

The Hugging Face speech-to-speech repository provides a production-ready pipeline for real-time voice conversations. By enabling live transcription, you can display subtitles while the user is still speaking, dramatically improving the conversational experience. This feature is implemented through three coordinated components that progressively process audio without waiting for speech completion.

Understanding the Live Transcription Architecture

Live transcription in speech-to-speech works by intercepting audio at the Voice Activity Detection (VAD) layer and forwarding partial results before the final utterance ends. This requires tight coordination between the VAD handler, STT handler, and a dedicated notifier component.

The implementation spans four critical source files:

File	Responsibility
src/speech_to_speech/arguments_classes/module_arguments.py	Defines the enable_live_transcription CLI/API flag
src/speech_to_speech/VAD/vad_handler.py	Implements progressive audio yielding via _process_realtime
src/speech_to_speech/STT/transcription_notifier.py	Bridges partial transcriptions to text_output_queue
src/speech_to_speech/s2s_pipeline.py	Wires components together and propagates configuration

Step 1: Configure Live Transcription via CLI

The fastest way to enable live transcription is through command-line arguments. In module_arguments.py (line 61), the ModuleArguments dataclass exposes:

from speech_to_speech.arguments_classes.module_arguments import ModuleArguments

# Or via CLI:

# python -m speech_to_speech.server --enable_live_transcription

Recommended CLI invocation:

python -m speech_to_speech.server \
  --enable_live_transcription \
  --live_transcription_update_interval 0.5 \
  --model_name gpt-4o

The live_transcription_update_interval parameter (default 0.5 seconds) controls how frequently progressive audio chunks are emitted. Lower values provide more responsive subtitles but increase STT computation.

Step 2: Enable Live Transcription Programmatically

For embedded applications, configure ModuleArguments directly and pass to S2SPipeline:

from speech_to_speech.arguments_classes.module_arguments import ModuleArguments
from speech_to_speech.s2s_pipeline import S2SPipeline

# Build configuration with live transcription enabled

module_args = ModuleArguments(
    enable_live_transcription=True,
    live_transcription_update_interval=0.3,  # 300ms updates

)

# Pipeline initializes VADHandler with realtime flag

pipeline = S2SPipeline(module_kwargs=module_args)
pipeline.run()

In s2s_pipeline.py (lines 579–585), the constructor propagates this flag:

if module_kwargs.enable_live_transcription:
    vad_kw.enable_realtime_transcription = True
    vad_kw.realtime_processing_pause = module_kwargs.live_transcription_update_interval

Step 3: How VADHandler Processes Realtime Audio

The VADHandler in vad_handler.py (line 70) implements the core realtime logic. When enable_realtime_transcription=True, it switches to _process_realtime mode:

# From vad_handler.py - _process_realtime excerpt

if (current_time - self.last_process_time) >= progressive_pause:
    yield VADAudio(
        audio=progressive_buffer,
        mode="progressive",  # Signals partial transcription expected

        timestamp=timestamp,
    )

Key behaviors of progressive mode:

• mode="progressive" — Tells downstream STT handlers this is intermediate audio
• SpeechStartedEvent / SpeechStoppedEvent — Keeps UI synchronized with speech state
• Periodic yielding — Controlled by realtime_processing_pause interval

Step 4: Consuming Partial Transcription Events

The TranscriptionNotifier (line 42 in transcription_notifier.py) receives PartialTranscription objects from STT handlers and forwards them:

self.text_output_queue.put(
    PartialTranscriptionEvent(delta=str(transcription.text), ...)
)

To consume live transcriptions in your client:

import queue
from speech_to_speech.pipeline.queue_types import TextEventItem
from speech_to_speech.STT.transcription_notifier import (
    PartialTranscriptionEvent,
    TranscriptionCompletedEvent,
)

output_q: queue.Queue[TextEventItem] = pipeline.text_output_queue

while True:
    event = output_q.get()
    
    if isinstance(event, PartialTranscriptionEvent):
        # Update subtitle display in real time

        print(f"Live: {event.delta}", end="\r")
        
    elif isinstance(event, TranscriptionCompletedEvent):
        # Final transcription ready for LLM processing

        print("\nFinal:", event.transcript)
        break

This event-driven design decouples transcription display from LLM response generation—you get immediate visual feedback without blocking the conversation flow.

Step 5: STT Handler Configuration for Progressive Mode

STT handlers must explicitly support progressive transcription. The ParakeetTDTSTTHandler (and WhisperSTTHandler) implement this via enable_live_transcription:

from speech_to_speech.STT.parakeet_tdt_handler import ParakeetTDTSTTHandler

stt = ParakeetTDTSTTHandler(
    enable_live_transcription=True,
    # Additional: model_path, language_detection, etc.

)

pipeline = S2SPipeline(
    module_kwargs=module_args,
    stt_handler=stt,
)
pipeline.run()

When enabled, the handler's process method invokes _show_progressive_transcription for each chunk, yielding lightweight PartialTranscription objects rather than waiting for endpoint detection.

Performance Considerations for Live Transcription

Update interval trade-offs:

Interval	Latency	CPU Usage	Best For
0.2s	Lowest	Higher	Real-time subtitles, typing indicators
0.5s (default)	Balanced	Moderate	General conversational UIs
1.0s	Higher	Lower	Low-power devices, batch-like feel

Memory implications: Progressive mode retains audio buffers longer than standard VAD. The VADHandler manages this via progressive_buffer accumulation, automatically flushed on SpeechStoppedEvent.

Full Pipeline Flow Summary

1. User speaks → Microphone captures audio
2. VADHandler detects voice activity, yields VADAudio(mode="progressive") every N seconds
3. STT Handler (Parakeet/Whisper) processes chunk → emits PartialTranscription
4. TranscriptionNotifier wraps in PartialTranscriptionEvent → posts to text_output_queue
5. Client UI displays live subtitle from queue
6. Speech ends → VADHandler yields final VADAudio, STT emits full Transcription
7. TranscriptionNotifier sends TranscriptionCompletedEvent, then GenerateResponseRequest to LLM

Summary

• enable_live_transcription in ModuleArguments activates the realtime code path
• VADHandler switches to _process_realtime, yielding progressive audio chunks
• TranscriptionNotifier bridges partial results to text_output_queue via PartialTranscriptionEvent
• live_transcription_update_interval tunes the subtitle refresh rate (0.2–1.0s typical)
• Event-driven consumption lets you display subtitles without blocking LLM response generation

Frequently Asked Questions

What happens if I enable live transcription but my STT handler doesn't support it?

The pipeline will fail gracefully with an attribute error or silently ignore progressive chunks depending on handler implementation. Always verify your chosen STT handler implements enable_live_transcription—ParakeetTDTSTTHandler and WhisperSTTHandler are confirmed compatible.

Can I disable live transcription mid-conversation?

No. The enable_live_transcription flag is read at pipeline construction time in S2SPipeline.__init__. To change modes, you must reconstruct the pipeline with new ModuleArguments. Dynamic reconfiguration is not supported in the current architecture.

How does live_transcription_update_interval affect STT accuracy?

Smaller intervals increase partial transcription noise—earlier chunks lack acoustic context. However, final transcriptions (sent to the LLM) use complete audio buffers, so interval choice only impacts subtitle quality, not downstream language model accuracy.

Where should I connect my UI to receive live transcriptions?

Read from pipeline.text_output_queue and filter for PartialTranscriptionEvent instances. The queue is thread-safe and populated by TranscriptionNotifier.process. For web applications, wrap this in a WebSocket forwarder; for desktop apps, use a direct UI update callback.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}