# How to Use Direct Audio Input Mode in speech-to-speech to Send Audio Directly to LLM (Skipping STT)

> Learn how to use direct audio input mode in speech-to-speech to send audio directly to LLM skipping STT. Configure RuntimeConfig for seamless audio routing.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-07

---

**The speech-to-speech library supports a direct audio input mode that bypasses Speech-to-Text entirely by setting `audio_input_mode="direct"` in `RuntimeConfig` and routing `VADAudio` messages through `AudioInputNotifier` instead of a transcription handler.**

Direct audio input mode eliminates the STT latency bottleneck by feeding raw audio from the Voice Activity Detector straight into the Language Model. This guide explains the architecture, configuration options, and implementation details based on the Hugging Face speech-to-speech source code.

## Architecture: How Direct Audio Input Works

### The VADAudio Message Type

The pipeline uses `VADAudio` messages to transport audio between stages. The critical field controlling routing is `mode`, defined in [`src/speech_to_speech/pipeline/messages.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/messages.py):

```python
class VADAudio(PipelineMessage):
    tag: Literal["vad_audio"] = "vad_audio"
    audio: np.ndarray
    mode: Literal["progressive", "final"] | None = None

```

- **`mode="progressive"`** — Audio chunks for streaming STT transcription
- **`mode=None`** or **`mode="final"`** — Complete audio segments eligible for direct LLM consumption

### AudioInputNotifier: The STT Bypass Handler

When `audio_input_mode="direct"` is configured, the pipeline instantiates `AudioInputNotifier` from [`src/speech_to_speech/LLM/audio_input_notifier.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/audio_input_notifier.py) instead of a `TranscriptionHandler`:

```python
class AudioInputNotifier(BaseHandler[VADAudio, LLMIn]):
    """Bridge final VAD audio directly into the LLM stage."""
    def should_process_input(self, item: VADAudio) -> bool:
        # "progressive" is used for streaming STT; everything else goes through

        if item.mode == "progressive":
            return False
        ...

```

This handler creates an `AudioInputCompletedEvent` containing the raw audio, sample rate, and timing metadata—delivered directly to the LLM handler without any text transcription.

### Pipeline Construction Logic

In [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py), the `audio_input_mode` flag determines handler selection:

```python
if runtime_config.audio_input_mode == "direct":
    pipeline.add_handler(AudioInputNotifier())
else:
    pipeline.add_handler(TranscriptionHandler())  # normal STT path

```

## Enabling Direct Audio Input Mode

### Method 1: Python API Configuration

Create a `RuntimeConfig` with `audio_input_mode="direct"` and pass it to `SpeechToSpeechPipeline`:

```python
from speech_to_speech.s2s_pipeline import SpeechToSpeechPipeline
from speech_to_speech.api.openai_realtime.runtime_config import RuntimeConfig

config = RuntimeConfig(
    # ... other parameters like temperature, model, max_tokens ...

    audio_input_mode="direct",
)

pipeline = SpeechToSpeechPipeline(runtime_config=config)
pipeline.run()

```

The `RuntimeConfig` class in [`src/speech_to_speech/api/openai_realtime/runtime_config.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/runtime_config.py) validates this field and propagates it throughout the pipeline initialization.

### Method 2: CLI with the Demo Client

The synthetic conversation demo supports direct audio mode via command-line flag:

```bash
python -m scripts.synthetic_conversation_realtime_client \
    --audio-input-mode direct \
    --model-id openai/whisper-1  # ignored in direct mode, but required by parser

```

The flag sets the runtime configuration before pipeline construction, triggering the same `AudioInputNotifier` path as the Python API.

### Method 3: Manual VADAudio Creation

For testing or custom integrations, construct `VADAudio` with `mode=None`:

```python
from speech_to_speech.pipeline.messages import VADAudio
import numpy as np

# 1 second of 16kHz audio (common for Whisper models)

audio_chunk = np.zeros(16000, dtype=np.float32)
vad_msg = VADAudio(audio=audio_chunk, mode=None)  # direct-audio eligible

```

When processed by a pipeline with direct audio mode enabled, this message bypasses all transcription logic.

## Key Implementation Details

### Why Progressive Mode Gets Filtered

The `should_process_input` logic explicitly excludes progressive chunks to prevent double-processing:

```python
if item.mode == "progressive":
    return False

```

This ensures that streaming STT configurations and direct audio modes can coexist in the same codebase without interference.

### Message Flow Comparison

| Stage | Standard STT Mode | Direct Audio Input Mode |
|-------|-------------------|------------------------|
| VAD output | `VADAudio(mode="progressive")` | `VADAudio(mode=None)` |
| Next handler | `TranscriptionHandler` | `AudioInputNotifier` |
| LLM input | `Transcription` text object | `AudioInputCompletedEvent` with raw audio |
| Latency | STT inference time + LLM time | Audio transmission time + LLM time |

### Compatible LLM Backends

Direct audio input mode requires an LLM handler that accepts `AudioInputCompletedEvent` messages. The built-in OpenAI Realtime API handler in `src/speech_to_speech/LLM/` supports this natively. Custom LLM handlers must implement:

```python
class MyAudioCapableLLM(BaseHandler[AudioInputCompletedEvent, LLMOut]):
    def process(self, event: AudioInputCompletedEvent) -> LLMOut:
        # Access event.audio (np.ndarray), event.sample_rate, event.timestamp

        ...

```

## Summary

- **Set `audio_input_mode="direct"`** in `RuntimeConfig` to skip STT entirely
- **`AudioInputNotifier`** in [`src/speech_to_speech/LLM/audio_input_notifier.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/audio_input_notifier.py) bridges VAD audio directly to the LLM
- **`VADAudio` messages with `mode=None`** or `"final"` trigger the direct path; `"progressive"` is filtered out
- **Pipeline construction** conditionally adds handlers based on the runtime configuration flag
- **Latency reduction** eliminates STT inference time, ideal for pre-processed audio or specialized audio-understanding models

## Frequently Asked Questions

### Does direct audio input mode work with any LLM backend?

No, the LLM handler must explicitly support audio input events. The Hugging Face speech-to-speech library's OpenAI Realtime API integration handles this natively. For custom backends, your handler must accept `AudioInputCompletedEvent` objects containing the raw `np.ndarray` audio buffer, sample rate, and timing metadata.

### What audio format does the LLM receive in direct mode?

The `AudioInputCompletedEvent` carries the original NumPy array from `VADAudio.audio` plus metadata. For OpenAI Realtime compatibility, the downstream handler typically converts this to 16kHz 16-bit PCM. Check your specific LLM handler's preprocessing requirements in `src/speech_to_speech/LLM/`.

### Can I switch between STT and direct audio modes at runtime?

No, the `audio_input_mode` is evaluated once during pipeline construction in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py). To change modes, you must create a new `SpeechToSpeechPipeline` instance with a modified `RuntimeConfig`. The mode is not designed for dynamic switching within a single conversation session.

### Why would I use direct audio input instead of STT?

Use direct audio input when your downstream LLM has native audio understanding capabilities (like OpenAI's GPT-4o Realtime models), when operating on pre-processed audio features, or when minimizing latency is critical. Standard STT mode remains preferable for text-based LLMs or when you need human-readable transcriptions for logging or debugging.