How to Configure Session Instructions, Tools, and Voice Settings at Runtime in Hugging Face Speech-to-Speech

The Hugging Face speech-to-speech library enables runtime session configuration through typed dataclass arguments that populate a ParsedArguments object, allowing dynamic assembly of STT, LLM, and TTS handlers via S2SPipeline.from_dict() or CLI parsing.

The huggingface/speech-to-speech repository implements a modular pipeline architecture where session configuration is decoupled from processing logic. This design allows you to adjust voice settings, language model instructions, and available tools at runtime without modifying the underlying handler implementations in src/speech_to_speech/.

Understanding the Argument Layer Architecture

The configuration system rests on a typed argument layer that uses Python dataclasses to define valid parameters for every pipeline component.

Typed Configuration Classes

Each handler exposes a dedicated arguments class in src/speech_to_speech/arguments_classes/. For example, WhisperSTTHandlerArguments configures speech-to-text parameters, while Qwen3TTSHandlerArguments controls voice synthesis settings. These dataclasses integrate with HfArgumentParser to validate inputs before the pipeline starts.

The LanguageModelHandlerArguments class manages session instructions and tool definitions for the LLM component, allowing you to specify system prompts and function calling capabilities programmatically.

ParsedArguments and Handler Graph Assembly

The S2SPipeline class in src/speech_to_speech/s2s_pipeline.py collects all argument instances into a single ParsedArguments object. During initialization, the pipeline inspects this object to instantiate concrete handlers—such as WhisperSTTHandler or ChatCompletionsLanguageModelHandler—and wire their input/output queues together.

This dynamic assembly means changing from Whisper to Paraformer STT requires only swapping the argument class reference, with no changes to the orchestration logic.

Configuring Voice Settings and Audio Parameters

Voice characteristics and audio processing parameters are controlled through TTS and STT handler arguments.

TTS Handler Configuration

The Qwen3TTSHandlerArguments dataclass in the arguments layer exposes parameters for voice selection, speed, and output format. When passed to S2SPipeline, these settings configure the Qwen3TTSHandler in src/speech_to_speech/TTS/qwen3_tts_handler.py to generate audio with specific acoustic properties.

from speech_to_speech.s2s_pipeline import S2SPipeline
from speech_to_speech.arguments_classes import Qwen3TTSHandlerArguments

# Configure voice settings at runtime

tts_args = Qwen3TTSHandlerArguments(
    qwen3_tts_model_name="Qwen/Qwen3-tts",
    # Additional voice parameters specific to the handler

)

pipeline = S2SPipeline.from_dict({
    "qwen3_tts_model_name": "Qwen/Qwen3-tts",
    "whisper_model_name": "openai/whisper-base"
})

STT and VAD Configuration

Speech recognition settings are managed through WhisperSTTHandlerArguments and VADHandlerArguments. These control model selection, language detection, and voice activity detection sensitivity. The BaseHandler class in src/speech_to_speech/baseHandler.py consumes these configurations when spawning worker threads for audio processing.

Session Instructions and Tool Integration

The pipeline supports dynamic configuration of language model behavior through dedicated argument classes.

LLM Handler Arguments

LanguageModelHandlerArguments defines session-level instructions and available tools for the ChatCompletionsLanguageModelHandler. This includes system prompts, maximum token limits, and tool schemas for function calling. The SpeculativeTurnTracker in src/speech_to_speech/pipeline/speculative_turns.py uses these settings to optimize response latency while respecting the configured instructions.

# Configure instructions and tools programmatically

config = {
    "language_model_name": "meta-llama/Meta-Llama-3-8B-Instruct",
    "system_prompt": "You are a helpful voice assistant. Use tools when necessary.",
    # Tool definitions would be passed through the appropriate argument fields

}

pipeline = S2SPipeline.from_dict(config)
pipeline.run()

Runtime Configuration Patterns

You can inject configuration data at runtime through two primary mechanisms.

Programmatic Configuration with from_dict

The S2SPipeline.from_dict() method accepts a dictionary mapping argument names to values, instantiating the necessary dataclasses internally. This approach is ideal for applications that need to switch configurations based on user profiles or external APIs.

from speech_to_speech.s2s_pipeline import S2SPipeline

# Runtime session configuration

session_config = {
    "whisper_model_name": "openai/whisper-base",
    "qwen3_tts_model_name": "Qwen/Qwen3-tts",
    "language_model_name": "meta-llama/Meta-Llama-3-8B-Instruct",
    "use_vad": True,
}

pipeline = S2SPipeline.from_dict(session_config)
pipeline.run()

Command-Line Interface

For server deployments, the pipeline uses HfArgumentParser to aggregate all argument classes into a unified CLI. You can pass configuration files or individual flags to override defaults defined in the dataclasses.

python -m speech_to_speech.s2s_pipeline \
    --whisper_model_name openai/whisper-base \
    --qwen3_tts_model_name Qwen/Qwen3-tts \
    --language_model_name meta-llama/Meta-Llama-3-8B-Instruct \
    --use_vad True

Session Lifecycle and Control

Runtime configuration includes mechanisms for graceful session termination and state management.

Control Messages and Session Termination

The ControlKind enum in src/speech_to_speech/pipeline/control.py defines END_SESSION, a special control message that signals all handlers to cease processing. When BaseHandler instances receive this message through their input queues, they shut down their worker threads cleanly, allowing you to reconfigure and restart the pipeline with new parameters without restarting the Python process.

Monitoring Configuration via Events

The pipeline emits typed events defined in src/speech_to_speech/pipeline/events.py that allow external observers to track configuration changes and processing state.

from speech_to_speech.s2s_pipeline import S2SPipeline
from speech_to_speech.pipeline.events import TranscriptionCompletedEvent, SpeechStartedEvent

pipeline = S2SPipeline.from_dict(config)

# Monitor session events

pipeline.register_event_handler(
    TranscriptionCompletedEvent, 
    lambda ev: print(f"Transcription: {ev.transcription}")
)
pipeline.register_event_handler(
    SpeechStartedEvent,
    lambda ev: print("Speech detected, processing...")
)

pipeline.run()

Summary

• Typed dataclasses like Qwen3TTSHandlerArguments and LanguageModelHandlerArguments encapsulate configuration for each pipeline layer in src/speech_to_speech/arguments_classes/.
• S2SPipeline.from_dict() enables runtime configuration injection without CLI dependency, supporting dynamic voice settings and instruction updates.
• ParsedArguments aggregates all handler configurations, allowing the pipeline to construct the appropriate processing graph at initialization.
• Control messages such as `ControlKind.END_SESSION** manage session lifecycle, enabling clean shutdowns and configuration refreshes.
• Event system in src/speech_to_speech/pipeline/events.py provides observability into how runtime configuration affects processing.

Frequently Asked Questions

How do I change voice settings mid-session in the Hugging Face speech-to-speech pipeline?

You cannot modify TTS parameters like voice or speed while the pipeline is actively running because handlers initialize their models during S2SPipeline.run(). To change voice settings, send a ControlKind.END_SESSION message to trigger graceful shutdown, then instantiate a new S2SPipeline with updated Qwen3TTSHandlerArguments via from_dict().

What is the difference between S2SPipeline() and S2SPipeline.from_dict() for configuration?

S2SPipeline() parses arguments from sys.argv using HfArgumentParser, making it suitable for command-line execution. S2SPipeline.from_dict() accepts a Python dictionary directly, enabling programmatic configuration within applications or web servers where CLI arguments are unavailable.

Where are session instructions stored in the argument architecture?

Session instructions reside in LanguageModelHandlerArguments, which configures the ChatCompletionsLanguageModelHandler. These arguments define the system prompt, available tools, and generation parameters that govern how the LLM responds to transcriptions during the speech-to-speech session.

How does the pipeline handle configuration validation?

The HfArgumentParser validates all configuration against the type hints defined in the dataclass argument classes (e.g., WhisperSTTHandlerArguments) before S2SPipeline constructs the handler graph. Invalid types or missing required fields raise parsing errors immediately, preventing misconfigured sessions from starting.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}