Debugging Live Transcription Streaming and TranscriptionNotifier Events in Hugging Face Speech-to-Speech

Live transcription streaming in the huggingface/speech-to-speech library emits PartialTranscriptionEvent objects for real-time updates and TranscriptionCompletedEvent objects for final results, with the TranscriptionNotifier class acting as the bridge between STT handlers and the queue-based event system.

The huggingface/speech-to-speech pipeline processes real-time audio through Voice Activity Detection (VAD), Speech-to-Text (STT), and Large Language Model (LLM) stages. Debugging live transcription streaming requires understanding how the TranscriptionNotifier converts STT outputs into protocol-neutral queue events that drive the realtime service.

Pipeline Architecture Overview

The speech-to-speech library transforms raw microphone data into LLM requests through a sequence of typed handlers:

• VAD produces VADAudio messages consumed by STT handlers
• STT yields PartialTranscription (live) or Transcription (final) objects
• TranscriptionNotifier converts these into PartialTranscriptionEvent or TranscriptionCompletedEvent placed on a Queue[TextEventItem] defined in pipeline/queue_types.py
• RealtimeService consumes these queue items via WebSocket routers

This architecture separates protocol-neutral pipeline messages from transport-specific events, enabling flexible debugging of the live transcription stream independent of the WebSocket layer.

How TranscriptionNotifier Processes Events

The TranscriptionNotifier class in src/speech_to_speech/STT/transcription_notifier.py implements the BaseHandler[STTOut, LLMIn] interface and serves as the critical junction between STT outputs and downstream consumers.

Setup and Configuration

The notifier requires explicit initialization of queue and event objects:

def setup(self, text_output_queue=None, should_listen=None):
    self.text_output_queue = text_output_queue
    self.should_listen = should_listen

The text_output_queue receives protocol-neutral events, while should_listen is a threading.Event that controls microphone listening state when transcripts are empty.

Handling Partial Transcriptions

When processing a PartialTranscription, the notifier emits live updates without triggering LLM generation:

def process(self, transcription: STTOut) -> Iterator[LLMIn]:
    if isinstance(transcription, PartialTranscription):
        if self.text_output_queue and transcription.text:
            self.text_output_queue.put(
                PartialTranscriptionEvent(
                    delta=str(transcription.text),
                    turn_id=transcription.turn_id,
                    turn_revision=transcription.turn_revision,
                )
            )
        return  # No LLM input emitted

Key behavior: Partial transcriptions never produce LLM input—the method returns early after queueing the event.

Handling Final Transcriptions

Final Transcription objects always generate a TranscriptionCompletedEvent (lines 63-71). Even when the final transcript is empty, the notifier still emits the completion event if a partial was previously sent, as verified by test_empty_final_transcription_still_emits_completion_after_partial.

If the transcript is empty and should_listen was provided, the notifier sets the event (lines 76-78) to resume audio capture:

if not transcription.text and self.should_listen:
    self.should_listen.set()

Implementing Live Transcription Streaming

The ParakeetTDTSTTHandler in src/speech_to_speech/STT/parakeet_tdt_handler.py implements live streaming through progressive transcription.

Enabling Live Mode

Live transcription is controlled by the enable_live_transcription flag (default True):

handler = ParakeetTDTSTTHandler()
handler.enable_live_transcription = True
handler.streaming_handler = SmartProgressiveStreamingHandler(model=my_parakeet_model)

Progressive Streaming Mechanics

The handler calls self.streaming_handler.transcribe_incremental(audio) repeatedly, yielding PartialTranscription objects containing both fixed (confirmed) and active (growing) text segments. The SmartProgressiveStreamingHandler manages windowing with default parameters of max_window_size=15.0 seconds and sentence_buffer=2.0 seconds.

Console output is handled by _print_live_transcription, which clears previous lines using _clear_live_transcription_line to create the streaming effect.

Common Debugging Scenarios

No PartialTranscriptionEvent appears in the queue Likely causes include empty transcription.text or uninitialized text_output_queue. Verify by adding logger.debug before line 35 in transcription_notifier.py or inspecting queue contents after processing.

Final transcription never triggers downstream processing If the Transcription object is empty and no partial preceded it, the notifier may skip emission. Run test_empty_final_transcription_still_emits_completion_after_partial to verify expected behavior—completion events require a preceding partial when the final text is empty.

Service fails to resume listening after silence Check that should_listen was passed during notifier setup via _notifier(..., should_listen=event). Without this event reference, empty final transcripts cannot re-enable the microphone.

Live console output flickers or truncates The _print_live_transcription method truncates text with ellipsis (\u2026) when exceeding terminal width. Verify FakeConsole.width in tests or increase terminal dimensions.

Partial transcripts fail to fix after 15 seconds Review SmartProgressiveStreamingHandler configuration. The default max_window_size=15.0 forces text finalization; exceeding this window without sentence breaks may cause unexpected behavior.

Practical Code Examples

Instantiating TranscriptionNotifier

from queue import Queue
from threading import Event
from speech_to_speech.STT.transcription_notifier import TranscriptionNotifier

q = Queue()
listen_evt = Event()
notifier = TranscriptionNotifier()
notifier.setup(text_output_queue=q, should_listen=listen_evt)

Feeding Partial and Final Transcriptions

from speech_to_speech.pipeline.messages import PartialTranscription, Transcription

# Live partial update

partial = PartialTranscription(text="Hello wor", turn_id="t1", turn_revision=0)
list(notifier.process(partial))  # Returns [] (no LLM input)

# Final empty transcript

final = Transcription(text="", language_code="en", speech_stopped_at_s=12.3)
list(notifier.process(final))  # Returns [] but emits completion event

# Retrieve events from queue

from speech_to_speech.pipeline.events import PartialTranscriptionEvent, TranscriptionCompletedEvent

partial_evt = q.get_nowait()   # PartialTranscriptionEvent

complete_evt = q.get_nowait()  # TranscriptionCompletedEvent

Processing VAD Audio with Live Transcription

from speech_to_speech.pipeline.messages import VADAudio

vad_msg = VADAudio(audio=my_audio_np, mode="progressive")
for msg in handler.process(vad_msg):
    if isinstance(msg, PartialTranscription):
        print(f"Live: {msg.text}")
    elif isinstance(msg, Transcription):
        print(f"Final: {msg.text}")

Inspecting Queue Events for Debugging

while not q.empty():
    ev = q.get()
    if isinstance(ev, PartialTranscriptionEvent):
        print(f"[LIVE] delta='{ev.delta}' turn={ev.turn_id}")
    elif isinstance(ev, TranscriptionCompletedEvent):
        print(f"[DONE] transcript='{ev.transcript}' lang={ev.language_code}")

Key Source Files

• src/speech_to_speech/STT/transcription_notifier.py – Core bridge converting STT outputs to queue events; handles empty-transcript edge cases and listening state management.
• src/speech_to_speech/pipeline/events.py – Pydantic models defining PartialTranscriptionEvent and TranscriptionCompletedEvent.
• src/speech_to_speech/pipeline/messages.py – Typed pipeline messages including PartialTranscription and Transcription.
• src/speech_to_speech/STT/parakeet_tdt_handler.py – STT handler implementing live transcription streaming with console output.
• src/speech_to_speech/STT/smart_progressive_streaming.py – Incremental transcription algorithm producing live updates with configurable windowing.
• tests/test_transcription_notifier.py – Unit tests specifying behavior for empty versus non-empty final transcriptions.
• tests/test_parakeet_transcription_events.py – Tests verifying live console handling and progressive transcription mechanics.

Summary

• Partial transcriptions generate only PartialTranscriptionEvent queue items and never trigger LLM processing.
• Final transcriptions always emit TranscriptionCompletedEvent, even when empty, provided a partial was previously sent.
• Empty final transcripts can re-enable microphone listening via the should_listen event if configured during notifier setup.
• Live streaming relies on SmartProgressiveStreamingHandler with 15-second default windows and console line-clearing for real-time display.
• Debug verification should inspect the text_output_queue contents and confirm should_listen event propagation in transcription_notifier.py.

Frequently Asked Questions

Why are my partial transcriptions not appearing in the WebSocket feed?

Partial transcriptions require both a configured text_output_queue and non-empty text content. Verify that TranscriptionNotifier.setup() received the queue instance and that the STT handler is yielding PartialTranscription objects with populated text fields. The notifier silently drops empty partials to reduce noise.

How does the system handle completely silent final transcriptions?

When a final Transcription arrives with empty text but was preceded by partial updates, the TranscriptionNotifier still emits a TranscriptionCompletedEvent to signal turn completion. If should_listen was provided during setup, the notifier sets this event to resume audio capture, preventing the pipeline from stalling on silence.

What causes the live transcription console output to flicker or disappear?

The ParakeetTDTSTTHandler uses carriage return sequences to overwrite lines for the streaming effect. If terminal width is insufficient, the _print_live_transcription method truncates text with ellipsis characters, potentially causing visual artifacts. Ensure your terminal width accommodates the expected transcription length or modify the handler's console width configuration.

How can I verify that TranscriptionNotifier is emitting the correct event types?

Inject a debug loop inspecting the text_output_queue after processing messages, as shown in the practical examples. Valid live transcriptions should produce PartialTranscriptionEvent objects with delta strings, while final transcriptions produce TranscriptionCompletedEvent objects containing the full transcript and metadata. Unit tests in tests/test_transcription_notifier.py demonstrate the expected event sequences for various edge cases.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}