Implementing Audio Input Directly to LLM Without STT: The Hugging Face Speech-to-Speech Architecture

You can bypass traditional speech-to-text transcription in the Hugging Face speech-to-speech repository by setting --stt none with an audio-capable LLM backend like chat-completions, which routes raw audio directly from the VAD handler to the LLM via the AudioInputNotifier class.

The huggingface/speech-to-speech repository provides a real-time voice pipeline that traditionally converts speech to text before processing with a language model. For implementing audio input directly to LLM without STT, the framework offers a specialized pathway that feeds raw audio directly to multimodal LLMs supporting native audio understanding. This architecture eliminates transcription latency and reduces computational overhead by removing the STT module entirely from the handler chain.

Backend Capability Declaration

The system declares audio input capabilities through the BackendCapabilities dataclass in src/speech_to_speech/backend_registry.py (lines 4-6). This dataclass includes the boolean flag supports_audio_input, which backends must set to True to receive raw audio streams.

The chat-completions backend explicitly enables this capability (lines 404-406), making it compatible with the --stt none configuration. When you query the LLM_BACKENDS registry, only backends advertising supports_audio_input=True appear as valid options for direct audio processing.

CLI Validation and Safety Checks

Before constructing the pipeline, the system validates compatibility in src/speech_to_speech/s2s_pipeline.py (lines 321-323). The validation logic ensures that --stt none is only permitted when the selected LLM backend can handle raw audio:

# s2s_pipeline.py – lines 321-323

if module_kwargs.stt == "none" and not llm_backend.spec.capabilities.supports_audio_input:
    supported = ", ".join(
        name for name, spec in LLM_BACKENDS.items()
        if spec.capabilities.supports_audio_input
    )
    raise ValueError(
        f"--stt none requires an audio-input LLM backend; choose one of: {supported}."
    )

If the check fails, the system raises a ValueError listing compatible backends such as chat-completions, preventing runtime configuration errors.

Audio Input Handler Construction

When validation passes, the pipeline instantiates an AudioInputNotifier instead of a traditional STT handler. Located in src/speech_to_speech/LLM/audio_input_notifier.py, this class converts incoming audio chunks into the format required by the LLM (e.g., raw PCM data or streaming tensors).

The factory function _create_audio_input in backend_registry.py (lines 34-46) handles this instantiation:

# backend_registry.py – lines 34-46

def _create_audio_input(context: HandlerContext, _config: Mapping[str, Any]) -> Any:
    handler_class = _load_handler(
        "speech_to_speech.LLM.audio_input_notifier", "AudioInputNotifier"
    )
    return handler_class(
        context.stop_event,
        queue_in=context.queue_in,
        queue_out=context.queue_out,
        setup_kwargs={
            "sample_rate": context.sample_rate,
            "speculative_turns": context.speculative_turns,
            "text_output_queue": context.text_output_queue,
        },
    )

This handler receives the sample_rate and speculative_turns parameters from the pipeline context, ensuring the audio format matches the LLM's expectations.

Data Flow Through the Pipeline

When operating in STT-bypass mode, audio flows through this specific handler chain:

1. VADHandler (src/speech_to_speech/VAD/vad_handler.py) detects voice activity and segments audio into chunks.
2. AudioInputNotifier receives these chunks directly, bypassing transcription.
3. LLM Backend (e.g., chat-completions) processes the raw audio tensors or PCM data.
4. LMOutputProcessor handles the language model's text or audio output.
5. TTS generates the final spoken response if needed.

This flow preserves turn-taking semantics through the voice activity detection while eliminating the latency introduced by intermediate text transcription.

Practical Implementation

Command-Line Configuration

To activate the direct audio pathway, specify --stt none alongside an audio-capable LLM backend:

python -m speech_to_speech run serve \
  --stt none \
  --llm_backend chat-completions \
  --tts qwen3 \
  --enable_llm_proxy

For local testing without a server, replace run serve with run local. The system validates that the selected backend supports audio input before initializing the pipeline.

Programmatic Pipeline Construction

You can also instantiate this configuration programmatically:

from speech_to_speech.s2s_pipeline import run_pipeline_command

# Initialize direct audio-to-LLM pipeline

run_pipeline_command(
    command="local",
    argv=[
        "--stt", "none",
        "--llm_backend", "chat-completions",
        "--tts", "qwen3",
        "--enable_llm_proxy"
    ],
)

When to Use Direct Audio Input

Implementing audio input directly to LLM without STT benefits specific deployment scenarios:

• Low-latency voice agents where eliminating the transcription step reduces round-trip time for real-time conversations, crucial when the LLM natively understands speech (e.g., OpenAI's audio-enabled chat-completions).
• Resource-constrained environments where removing the STT model reduces memory and CPU/GPU requirements, allowing deployment on edge devices.
• End-to-end audio quality preservation, maintaining paralinguistic features (tone, emotion, non-speech sounds) that might be lost in text transcription.

Summary

• The Hugging Face speech-to-speech repository supports direct audio input through the --stt none CLI flag combined with audio-capable LLM backends.
• BackendCapabilities in backend_registry.py advertises audio support via the supports_audio_input boolean, enforced during pipeline initialization in s2s_pipeline.py (lines 321-323).
• The AudioInputNotifier class in src/speech_to_speech/LLM/audio_input_notifier.py handles raw audio chunk forwarding from the VAD to the LLM, replacing traditional STT handlers.
• This architecture enables low-latency voice agents that process speech end-to-end without intermediate text representation, reducing both latency and computational overhead.

Frequently Asked Questions

Which LLM backends support direct audio input in the speech-to-speech repository?

The chat-completions backend explicitly supports direct audio input by setting supports_audio_input=True in its BackendCapabilities declaration (lines 404-406 in backend_registry.py). You can verify support programmatically by checking the LLM_BACKENDS registry for backends where spec.capabilities.supports_audio_input evaluates to true.

What happens if I set --stt none with an incompatible LLM backend?

The pipeline raises a ValueError during initialization in s2s_pipeline.py (lines 321-323). The error message lists all compatible backends that advertise audio input capabilities, preventing runtime failures from unsupported configurations.

Does bypassing STT affect voice activity detection (VAD)?

No, the VADHandler remains active in the pipeline regardless of STT configuration. The voice activity detection still segments incoming audio to determine turn boundaries, but instead of forwarding chunks to an STT model, it sends them directly to the AudioInputNotifier for LLM processing.

Can I use the audio-direct mode with local LLM inference?

Yes, provided your local LLM backend implementation sets supports_audio_input=True in its capabilities and can process the audio format provided by AudioInputNotifier. The repository's modular design allows custom backends to register for direct audio processing as long as they implement the required handler interfaces and accept the sample_rate and audio chunk formats passed through the handler context.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}