How to Use Direct Audio Input Mode in speech-to-speech to Send Audio Directly to LLM (Skipping STT)

The speech-to-speech library supports a direct audio input mode that bypasses Speech-to-Text entirely by setting audio_input_mode="direct" in RuntimeConfig and routing VADAudio messages through AudioInputNotifier instead of a transcription handler.

Direct audio input mode eliminates the STT latency bottleneck by feeding raw audio from the Voice Activity Detector straight into the Language Model. This guide explains the architecture, configuration options, and implementation details based on the Hugging Face speech-to-speech source code.

Architecture: How Direct Audio Input Works

The VADAudio Message Type

The pipeline uses VADAudio messages to transport audio between stages. The critical field controlling routing is mode, defined in src/speech_to_speech/pipeline/messages.py:

class VADAudio(PipelineMessage):
    tag: Literal["vad_audio"] = "vad_audio"
    audio: np.ndarray
    mode: Literal["progressive", "final"] | None = None

• mode="progressive" — Audio chunks for streaming STT transcription
• mode=None or mode="final" — Complete audio segments eligible for direct LLM consumption

AudioInputNotifier: The STT Bypass Handler

When audio_input_mode="direct" is configured, the pipeline instantiates AudioInputNotifier from src/speech_to_speech/LLM/audio_input_notifier.py instead of a TranscriptionHandler:

class AudioInputNotifier(BaseHandler[VADAudio, LLMIn]):
    """Bridge final VAD audio directly into the LLM stage."""
    def should_process_input(self, item: VADAudio) -> bool:
        # "progressive" is used for streaming STT; everything else goes through

        if item.mode == "progressive":
            return False
        ...

This handler creates an AudioInputCompletedEvent containing the raw audio, sample rate, and timing metadata—delivered directly to the LLM handler without any text transcription.

Pipeline Construction Logic

In src/speech_to_speech/s2s_pipeline.py, the audio_input_mode flag determines handler selection:

if runtime_config.audio_input_mode == "direct":
    pipeline.add_handler(AudioInputNotifier())
else:
    pipeline.add_handler(TranscriptionHandler())  # normal STT path

Enabling Direct Audio Input Mode

Method 1: Python API Configuration

Create a RuntimeConfig with audio_input_mode="direct" and pass it to SpeechToSpeechPipeline:

from speech_to_speech.s2s_pipeline import SpeechToSpeechPipeline
from speech_to_speech.api.openai_realtime.runtime_config import RuntimeConfig

config = RuntimeConfig(
    # ... other parameters like temperature, model, max_tokens ...

    audio_input_mode="direct",
)

pipeline = SpeechToSpeechPipeline(runtime_config=config)
pipeline.run()

The RuntimeConfig class in src/speech_to_speech/api/openai_realtime/runtime_config.py validates this field and propagates it throughout the pipeline initialization.

Method 2: CLI with the Demo Client

The synthetic conversation demo supports direct audio mode via command-line flag:

python -m scripts.synthetic_conversation_realtime_client \
    --audio-input-mode direct \
    --model-id openai/whisper-1  # ignored in direct mode, but required by parser

The flag sets the runtime configuration before pipeline construction, triggering the same AudioInputNotifier path as the Python API.

Method 3: Manual VADAudio Creation

For testing or custom integrations, construct VADAudio with mode=None:

from speech_to_speech.pipeline.messages import VADAudio
import numpy as np

# 1 second of 16kHz audio (common for Whisper models)

audio_chunk = np.zeros(16000, dtype=np.float32)
vad_msg = VADAudio(audio=audio_chunk, mode=None)  # direct-audio eligible

When processed by a pipeline with direct audio mode enabled, this message bypasses all transcription logic.

Key Implementation Details

Why Progressive Mode Gets Filtered

The should_process_input logic explicitly excludes progressive chunks to prevent double-processing:

if item.mode == "progressive":
    return False

This ensures that streaming STT configurations and direct audio modes can coexist in the same codebase without interference.

Message Flow Comparison

Stage	Standard STT Mode	Direct Audio Input Mode
VAD output	VADAudio(mode="progressive")	VADAudio(mode=None)
Next handler	TranscriptionHandler	AudioInputNotifier
LLM input	Transcription text object	AudioInputCompletedEvent with raw audio
Latency	STT inference time + LLM time	Audio transmission time + LLM time

Compatible LLM Backends

Direct audio input mode requires an LLM handler that accepts AudioInputCompletedEvent messages. The built-in OpenAI Realtime API handler in src/speech_to_speech/LLM/ supports this natively. Custom LLM handlers must implement:

class MyAudioCapableLLM(BaseHandler[AudioInputCompletedEvent, LLMOut]):
    def process(self, event: AudioInputCompletedEvent) -> LLMOut:
        # Access event.audio (np.ndarray), event.sample_rate, event.timestamp

        ...

Summary

• Set audio_input_mode="direct" in RuntimeConfig to skip STT entirely
• AudioInputNotifier in src/speech_to_speech/LLM/audio_input_notifier.py bridges VAD audio directly to the LLM
• VADAudio messages with mode=None or "final" trigger the direct path; "progressive" is filtered out
• Pipeline construction conditionally adds handlers based on the runtime configuration flag
• Latency reduction eliminates STT inference time, ideal for pre-processed audio or specialized audio-understanding models

Frequently Asked Questions

Does direct audio input mode work with any LLM backend?

No, the LLM handler must explicitly support audio input events. The Hugging Face speech-to-speech library's OpenAI Realtime API integration handles this natively. For custom backends, your handler must accept AudioInputCompletedEvent objects containing the raw np.ndarray audio buffer, sample rate, and timing metadata.

What audio format does the LLM receive in direct mode?

The AudioInputCompletedEvent carries the original NumPy array from VADAudio.audio plus metadata. For OpenAI Realtime compatibility, the downstream handler typically converts this to 16kHz 16-bit PCM. Check your specific LLM handler's preprocessing requirements in src/speech_to_speech/LLM/.

Can I switch between STT and direct audio modes at runtime?

No, the audio_input_mode is evaluated once during pipeline construction in s2s_pipeline.py. To change modes, you must create a new SpeechToSpeechPipeline instance with a modified RuntimeConfig. The mode is not designed for dynamic switching within a single conversation session.

Why would I use direct audio input instead of STT?

Use direct audio input when your downstream LLM has native audio understanding capabilities (like OpenAI's GPT-4o Realtime models), when operating on pre-processed audio features, or when minimizing latency is critical. Standard STT mode remains preferable for text-based LLMs or when you need human-readable transcriptions for logging or debugging.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}