# Implementing Audio Input Directly to LLM Without STT: The Hugging Face Speech-to-Speech Architecture

> Bypass STT in Hugging Face speech-to-speech. Route raw audio directly to LLM with --stt none and an audio-capable backend for efficient audio input.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: architecture
- Published: 2026-08-09

---

**You can bypass traditional speech-to-text transcription in the Hugging Face speech-to-speech repository by setting `--stt none` with an audio-capable LLM backend like `chat-completions`, which routes raw audio directly from the VAD handler to the LLM via the `AudioInputNotifier` class.**

The `huggingface/speech-to-speech` repository provides a real-time voice pipeline that traditionally converts speech to text before processing with a language model. For implementing audio input directly to LLM without STT, the framework offers a specialized pathway that feeds raw audio directly to multimodal LLMs supporting native audio understanding. This architecture eliminates transcription latency and reduces computational overhead by removing the STT module entirely from the handler chain.

## Backend Capability Declaration

The system declares audio input capabilities through the `BackendCapabilities` dataclass in [`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py) (lines 4-6). This dataclass includes the boolean flag `supports_audio_input`, which backends must set to `True` to receive raw audio streams.

The **chat-completions** backend explicitly enables this capability (lines 404-406), making it compatible with the `--stt none` configuration. When you query the `LLM_BACKENDS` registry, only backends advertising `supports_audio_input=True` appear as valid options for direct audio processing.

## CLI Validation and Safety Checks

Before constructing the pipeline, the system validates compatibility in [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) (lines 321-323). The validation logic ensures that `--stt none` is only permitted when the selected LLM backend can handle raw audio:

```python

# s2s_pipeline.py – lines 321-323

if module_kwargs.stt == "none" and not llm_backend.spec.capabilities.supports_audio_input:
    supported = ", ".join(
        name for name, spec in LLM_BACKENDS.items()
        if spec.capabilities.supports_audio_input
    )
    raise ValueError(
        f"--stt none requires an audio-input LLM backend; choose one of: {supported}."
    )

```

If the check fails, the system raises a `ValueError` listing compatible backends such as `chat-completions`, preventing runtime configuration errors.

## Audio Input Handler Construction

When validation passes, the pipeline instantiates an `AudioInputNotifier` instead of a traditional STT handler. Located in [`src/speech_to_speech/LLM/audio_input_notifier.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/audio_input_notifier.py), this class converts incoming audio chunks into the format required by the LLM (e.g., raw PCM data or streaming tensors).

The factory function `_create_audio_input` in [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py) (lines 34-46) handles this instantiation:

```python

# backend_registry.py – lines 34-46

def _create_audio_input(context: HandlerContext, _config: Mapping[str, Any]) -> Any:
    handler_class = _load_handler(
        "speech_to_speech.LLM.audio_input_notifier", "AudioInputNotifier"
    )
    return handler_class(
        context.stop_event,
        queue_in=context.queue_in,
        queue_out=context.queue_out,
        setup_kwargs={
            "sample_rate": context.sample_rate,
            "speculative_turns": context.speculative_turns,
            "text_output_queue": context.text_output_queue,
        },
    )

```

This handler receives the `sample_rate` and `speculative_turns` parameters from the pipeline context, ensuring the audio format matches the LLM's expectations.

## Data Flow Through the Pipeline

When operating in STT-bypass mode, audio flows through this specific handler chain:

1. **VADHandler** ([`src/speech_to_speech/VAD/vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/vad_handler.py)) detects voice activity and segments audio into chunks.
2. **AudioInputNotifier** receives these chunks directly, bypassing transcription.
3. **LLM Backend** (e.g., chat-completions) processes the raw audio tensors or PCM data.
4. **LMOutputProcessor** handles the language model's text or audio output.
5. **TTS** generates the final spoken response if needed.

This flow preserves turn-taking semantics through the voice activity detection while eliminating the latency introduced by intermediate text transcription.

## Practical Implementation

### Command-Line Configuration

To activate the direct audio pathway, specify `--stt none` alongside an audio-capable LLM backend:

```bash
python -m speech_to_speech run serve \
  --stt none \
  --llm_backend chat-completions \
  --tts qwen3 \
  --enable_llm_proxy

```

For local testing without a server, replace `run serve` with `run local`. The system validates that the selected backend supports audio input before initializing the pipeline.

### Programmatic Pipeline Construction

You can also instantiate this configuration programmatically:

```python
from speech_to_speech.s2s_pipeline import run_pipeline_command

# Initialize direct audio-to-LLM pipeline

run_pipeline_command(
    command="local",
    argv=[
        "--stt", "none",
        "--llm_backend", "chat-completions",
        "--tts", "qwen3",
        "--enable_llm_proxy"
    ],
)

```

## When to Use Direct Audio Input

Implementing audio input directly to LLM without STT benefits specific deployment scenarios:

- **Low-latency voice agents** where eliminating the transcription step reduces round-trip time for real-time conversations, crucial when the LLM natively understands speech (e.g., OpenAI's audio-enabled chat-completions).
- **Resource-constrained environments** where removing the STT model reduces memory and CPU/GPU requirements, allowing deployment on edge devices.
- **End-to-end audio quality** preservation, maintaining paralinguistic features (tone, emotion, non-speech sounds) that might be lost in text transcription.

## Summary

- The **Hugging Face speech-to-speech** repository supports direct audio input through the `--stt none` CLI flag combined with audio-capable LLM backends.
- **BackendCapabilities** in [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py) advertises audio support via the `supports_audio_input` boolean, enforced during pipeline initialization in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) (lines 321-323).
- The **AudioInputNotifier** class in [`src/speech_to_speech/LLM/audio_input_notifier.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/audio_input_notifier.py) handles raw audio chunk forwarding from the VAD to the LLM, replacing traditional STT handlers.
- This architecture enables **low-latency voice agents** that process speech end-to-end without intermediate text representation, reducing both latency and computational overhead.

## Frequently Asked Questions

### Which LLM backends support direct audio input in the speech-to-speech repository?

The **chat-completions** backend explicitly supports direct audio input by setting `supports_audio_input=True` in its `BackendCapabilities` declaration (lines 404-406 in [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py)). You can verify support programmatically by checking the `LLM_BACKENDS` registry for backends where `spec.capabilities.supports_audio_input` evaluates to true.

### What happens if I set `--stt none` with an incompatible LLM backend?

The pipeline raises a `ValueError` during initialization in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) (lines 321-323). The error message lists all compatible backends that advertise audio input capabilities, preventing runtime failures from unsupported configurations.

### Does bypassing STT affect voice activity detection (VAD)?

No, the **VADHandler** remains active in the pipeline regardless of STT configuration. The voice activity detection still segments incoming audio to determine turn boundaries, but instead of forwarding chunks to an STT model, it sends them directly to the `AudioInputNotifier` for LLM processing.

### Can I use the audio-direct mode with local LLM inference?

Yes, provided your local LLM backend implementation sets `supports_audio_input=True` in its capabilities and can process the audio format provided by `AudioInputNotifier`. The repository's modular design allows custom backends to register for direct audio processing as long as they implement the required handler interfaces and accept the `sample_rate` and audio chunk formats passed through the handler context.