# How to Skip STT and Send Audio Directly to an Audio-Capable LLM in Hugging Face Speech-to-Speech

> Learn how to skip STT and send audio directly to audio-capable LLMs in Hugging Face Speech-to-Speech by setting --stt none. Stream raw audio for efficient processing.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-11

---

**Set `--stt none` with `--llm_backend chat-completions` to bypass transcription and stream raw audio to a multimodal LLM.**

The Hugging Face `speech-to-speech` toolkit builds a modular processing chain: **VAD → STT → LLM → TTS**. By default, every spoken input passes through automatic speech recognition before reaching the language model. However, modern multimodal LLMs accept raw audio directly, eliminating latency and enabling end-to-end audio understanding. This guide explains how to skip the STT stage and route PCM audio straight to an audio-capable backend.

## Understanding the Pipeline Architecture

The `speech-to-speech` repository implements a handler-based pipeline where each stage communicates via queues. In [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py), the system validates component compatibility during startup. When you request STT bypass mode, the framework checks that your chosen LLM can receive audio input natively.

The validation logic (lines 21–24 of [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py)) enforces this constraint:

```python

# If --stt none is used, the LLM must advertise supports_audio_input=True

# Otherwise, ValueError is raised with list of compatible backends

```

This guardrail prevents pipeline misconfiguration and surfaces actionable error messages.

## Selecting the STT "none" Backend

The special STT backend **`none`** is registered in [`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py) (lines 90–95). Unlike standard STT handlers that invoke Whisper or similar models, this backend:

- Instantiates `AudioInputNotifier` instead of a transcription handler
- Forwards raw audio bytes unchanged to the LLM stage
- Adds zero processing overhead

```python

# Conceptual flow with --stt none

VAD detected audio → [no transcription] → AudioInputNotifier → LLM (raw PCM)

```

## Choosing an Audio-Capable LLM Backend

Not all LLM backends accept audio input. The capability matrix is defined in [`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py) (lines 96–105). Currently, only **`chat-completions`** exposes `supports_audio_input=True` in its `BackendCapabilities`.

| Backend | `supports_audio_input` | Audio Handling |
|---------|------------------------|----------------|
| `chat-completions` | ✅ `True` | Forwards PCM to remote API (OpenAI-compatible `/v1/audio/*` endpoints) |
| `transformers` | ❌ `False` | Text-only, requires STT transcription |
| `mlx-lm` | ❌ `False` | Text-only, requires STT transcription |
| `llama-cpp` | ❌ `False` | Text-only, requires STT transcription |

The `chat-completions` backend (implemented in [`src/speech_to_speech/LLM/chat_completions_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/chat_completions_language_model.py)) packages raw audio into the request payload for multimodal endpoints. This enables integration with:

- OpenAI's GPT-4o audio preview
- Custom audio-aware chat APIs
- Vision-language models with audio tokenizers

## CLI Commands to Skip STT

### Server Mode (Real-time Streaming)

Deploy a WebSocket server that accepts client audio and bypasses transcription:

```bash
speech-to-speech serve \
  --stt none \
  --llm_backend chat-completions \
  --tts qwen3 \
  --log_level info

```

The server expects clients to stream PCM audio. Each VAD-detected utterance routes directly to the configured chat-completions endpoint.

### Local Mode (Microphone Loopback)

Run a standalone client that captures from your microphone:

```bash
speech-to-speech local \
  --stt none \
  --llm_backend chat-completions \
  --tts qwen3 \
  --local-audio-input-device 0 \
  --local-audio-output-device 1 \
  --local-audio-chunk-size 1024

```

Adjust `--local-audio-input-device` and `--local-audio-output-device` to match your hardware indices (list with `arecord -l` or `sox` on Linux/macOS).

## Programmatic Configuration

For embedded applications or custom orchestration, construct arguments programmatically:

```python
from speech_to_speech.s2s_pipeline import parse_arguments, prepare_all_args, run_pipeline_command

args = parse_arguments(
    argv=[
        "--stt", "none",
        "--llm_backend", "chat-completions",
        "--tts", "qwen3",
        "--chat-completions-api-url", "https://api.openai.com/v1/chat/completions",
        "--chat-completions-api-key", os.getenv("OPENAI_API_KEY"),
    ],
    command="serve",
)
prepare_all_args(args)
run_pipeline_command("serve", [])

```

This pattern is useful for:
- Dynamic configuration based on runtime environment
- Testing with mock audio endpoints
- Integration into larger application frameworks

## Key Files and Their Roles

Understanding these source files helps with debugging and extension:

- **[`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py)** — Pipeline orchestration, argument parsing, and the `none` STT compatibility check
- **[`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py)** — Backend capability declarations, registration of `none` STT and `chat-completions` LLM
- **[`src/speech_to_speech/LLM/chat_completions_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/chat_completions_language_model.py)** — Audio-capable LLM handler implementation
- **[`src/speech_to_speech/STT/base_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/base_stt_handler.py)** — Base class hierarchy; `AudioInputNotifier` subclass enables pass-through mode

## Performance and Latency Considerations

Bypassing STT provides measurable benefits:

1. **Reduced end-to-end latency** — Eliminates transcription network round-trip or local inference time
2. **Preserved prosody** — Models receive raw audio containing tone, emotion, and non-verbal cues
3. **Unified context** — Single model handles audio comprehension and response generation

Trade-offs include:
- Higher LLM API costs (audio tokens vs. text tokens)
- Dependency on remote multimodal endpoints
- Reduced transparency (harder to debug audio vs. inspecting transcripts)

## Summary

- **Disable STT** with `--stt none` to route raw audio through the pipeline
- **Use `chat-completions` LLM backend** as the only built-in option with `supports_audio_input=True`
- **Validate configuration** — mismatched settings raise `ValueError` with compatible backend suggestions
- **Apply to server or local modes** using identical flag patterns
- **Consult [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py)** when extending with custom audio-capable LLMs

## Frequently Asked Questions

### What happens if I use `--stt none` with a non-audio LLM backend?

The pipeline raises a `ValueError` during startup in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py), listing LLM backends that support audio input. This prevents silent failures where audio would be dropped or misinterpreted.

### Can I use `--stt none` with local LLMs like `llama-cpp` or `mlx-lm`?

Not with the built-in backends. These register `supports_audio_input=False` in [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py). To use local audio-capable models, you would need to implement a custom LLM backend that advertises audio support and handles PCM payload encoding.

### What audio format does the `chat-completions` backend expect?

Raw PCM samples from the VAD stage, typically 16-bit 16kHz mono. The [`chat_completions_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/chat_completions_language_model.py) handler encodes these into the API-specific format (e.g., base64 for OpenAI) before transmission.

### Is there a quality difference between STT-first and raw audio routing?

Raw audio preserves acoustic information lost in transcription, potentially improving performance on emotion recognition, speaker identification, and noisy environments. However, the LLM's audio tokenizer quality and API latency become critical factors in perceived responsiveness.