How to Skip STT and Send Audio Directly to an Audio-Capable LLM in Hugging Face Speech-to-Speech

Set --stt none with --llm_backend chat-completions to bypass transcription and stream raw audio to a multimodal LLM.

The Hugging Face speech-to-speech toolkit builds a modular processing chain: VAD → STT → LLM → TTS. By default, every spoken input passes through automatic speech recognition before reaching the language model. However, modern multimodal LLMs accept raw audio directly, eliminating latency and enabling end-to-end audio understanding. This guide explains how to skip the STT stage and route PCM audio straight to an audio-capable backend.

Understanding the Pipeline Architecture

The speech-to-speech repository implements a handler-based pipeline where each stage communicates via queues. In src/speech_to_speech/s2s_pipeline.py, the system validates component compatibility during startup. When you request STT bypass mode, the framework checks that your chosen LLM can receive audio input natively.

The validation logic (lines 21–24 of s2s_pipeline.py) enforces this constraint:

# If --stt none is used, the LLM must advertise supports_audio_input=True

# Otherwise, ValueError is raised with list of compatible backends

This guardrail prevents pipeline misconfiguration and surfaces actionable error messages.

Selecting the STT "none" Backend

The special STT backend none is registered in src/speech_to_speech/backend_registry.py (lines 90–95). Unlike standard STT handlers that invoke Whisper or similar models, this backend:

• Instantiates AudioInputNotifier instead of a transcription handler
• Forwards raw audio bytes unchanged to the LLM stage
• Adds zero processing overhead

# Conceptual flow with --stt none

VAD detected audio → [no transcription] → AudioInputNotifier → LLM (raw PCM)

Choosing an Audio-Capable LLM Backend

Not all LLM backends accept audio input. The capability matrix is defined in src/speech_to_speech/backend_registry.py (lines 96–105). Currently, only chat-completions exposes supports_audio_input=True in its BackendCapabilities.

Backend	supports_audio_input	Audio Handling
chat-completions	✅ True	Forwards PCM to remote API (OpenAI-compatible /v1/audio/* endpoints)
transformers	❌ False	Text-only, requires STT transcription
mlx-lm	❌ False	Text-only, requires STT transcription
llama-cpp	❌ False	Text-only, requires STT transcription

The chat-completions backend (implemented in src/speech_to_speech/LLM/chat_completions_language_model.py) packages raw audio into the request payload for multimodal endpoints. This enables integration with:

• OpenAI's GPT-4o audio preview
• Custom audio-aware chat APIs
• Vision-language models with audio tokenizers

CLI Commands to Skip STT

Server Mode (Real-time Streaming)

Deploy a WebSocket server that accepts client audio and bypasses transcription:

speech-to-speech serve \
  --stt none \
  --llm_backend chat-completions \
  --tts qwen3 \
  --log_level info

The server expects clients to stream PCM audio. Each VAD-detected utterance routes directly to the configured chat-completions endpoint.

Local Mode (Microphone Loopback)

Run a standalone client that captures from your microphone:

speech-to-speech local \
  --stt none \
  --llm_backend chat-completions \
  --tts qwen3 \
  --local-audio-input-device 0 \
  --local-audio-output-device 1 \
  --local-audio-chunk-size 1024

Adjust --local-audio-input-device and --local-audio-output-device to match your hardware indices (list with arecord -l or sox on Linux/macOS).

Programmatic Configuration

For embedded applications or custom orchestration, construct arguments programmatically:

from speech_to_speech.s2s_pipeline import parse_arguments, prepare_all_args, run_pipeline_command

args = parse_arguments(
    argv=[
        "--stt", "none",
        "--llm_backend", "chat-completions",
        "--tts", "qwen3",
        "--chat-completions-api-url", "https://api.openai.com/v1/chat/completions",
        "--chat-completions-api-key", os.getenv("OPENAI_API_KEY"),
    ],
    command="serve",
)
prepare_all_args(args)
run_pipeline_command("serve", [])

This pattern is useful for:

• Dynamic configuration based on runtime environment
• Testing with mock audio endpoints
• Integration into larger application frameworks

Key Files and Their Roles

Understanding these source files helps with debugging and extension:

• src/speech_to_speech/s2s_pipeline.py — Pipeline orchestration, argument parsing, and the none STT compatibility check
• src/speech_to_speech/backend_registry.py — Backend capability declarations, registration of none STT and chat-completions LLM
• src/speech_to_speech/LLM/chat_completions_language_model.py — Audio-capable LLM handler implementation
• src/speech_to_speech/STT/base_stt_handler.py — Base class hierarchy; AudioInputNotifier subclass enables pass-through mode

Performance and Latency Considerations

Bypassing STT provides measurable benefits:

1. Reduced end-to-end latency — Eliminates transcription network round-trip or local inference time
2. Preserved prosody — Models receive raw audio containing tone, emotion, and non-verbal cues
3. Unified context — Single model handles audio comprehension and response generation

Trade-offs include:

• Higher LLM API costs (audio tokens vs. text tokens)
• Dependency on remote multimodal endpoints
• Reduced transparency (harder to debug audio vs. inspecting transcripts)

Summary

• Disable STT with --stt none to route raw audio through the pipeline
• Use chat-completions LLM backend as the only built-in option with supports_audio_input=True
• Validate configuration — mismatched settings raise ValueError with compatible backend suggestions
• Apply to server or local modes using identical flag patterns
• Consult backend_registry.py when extending with custom audio-capable LLMs

Frequently Asked Questions

What happens if I use --stt none with a non-audio LLM backend?

The pipeline raises a ValueError during startup in s2s_pipeline.py, listing LLM backends that support audio input. This prevents silent failures where audio would be dropped or misinterpreted.

Can I use --stt none with local LLMs like llama-cpp or mlx-lm?

Not with the built-in backends. These register supports_audio_input=False in backend_registry.py. To use local audio-capable models, you would need to implement a custom LLM backend that advertises audio support and handles PCM payload encoding.

What audio format does the chat-completions backend expect?

Raw PCM samples from the VAD stage, typically 16-bit 16kHz mono. The chat_completions_language_model.py handler encodes these into the API-specific format (e.g., base64 for OpenAI) before transmission.

Is there a quality difference between STT-first and raw audio routing?

Raw audio preserves acoustic information lost in transcription, potentially improving performance on emotion recognition, speaker identification, and noisy environments. However, the LLM's audio tokenizer quality and API latency become critical factors in perceived responsiveness.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}