How to Use Voice Prompts to Customize LLM Behavior in the Hugging Face Speech-to-Speech Repository

Voice prompts in the Speech-to-Speech repository combine a fixed lead, a customizable session prompt, an optional tool section, and a constraint-enforcing tail to shape spoken LLM output.

The huggingface/speech-to-speech library provides a structured approach to controlling how large language models behave during audio conversations. Rather than sending raw text prompts, the system uses a layered voice system prompt architecture that separates conversational framing from behavioral constraints. This design lets developers inject personality, goals, and tool capabilities while maintaining consistent speech-optimized output rules.

How the Voice Prompt Architecture Works

The voice prompt system is implemented in src/speech_to_speech/LLM/voice_prompt.py. It decomposes the system message into four ordered components:

1. Lead (VOICE_SYSTEM_PROMPT_LEAD) — establishes that the interaction is spoken and audio-oriented
2. Session prompt — user-supplied persona, scenario, or task description
3. Tool section — optional function-calling instructions passed via tool_section parameter
4. Tail (VOICE_SYSTEM_PROMPT_TAIL) — hard constraints on brevity, naturalness, and tool-use policy

The build_voice_system_prompt(session_prompt, *, tool_section="") function assembles these pieces into the final string sent to the LLM.

When Voice Prompts Are Selected

The pipeline automatically chooses between voice and text prompt builders based on the wants_audio flag. In src/speech_to_speech/LLM/language_model.py, the selection logic is:

builder = build_voice_system_prompt if wants_audio else build_text_system_prompt

This ensures that audio-targeted requests receive the spoken-conversation framing while text-only interactions use a simpler format.

Customizing the Core Prompt Components

Modifying the Lead or Tail Constants

The lead and tail are defined as module-level strings in voice_prompt.py. To permanently change the conversational premise or output constraints, edit these constants directly:

Constant	Purpose	Typical Content
VOICE_SYSTEM_PROMPT_LEAD	Sets spoken-interaction context	"You are in a spoken conversation with a user..."
VOICE_SYSTEM_PROMPT_TAIL	Enforces speech-optimized rules	Brevity commands, hesitation handling, tool-use syntax

Changes to these constants affect all voice prompts system-wide.

Appending High-Priority Constraints

The tail appears last in the prompt, giving it the highest positional influence over model behavior. Append additional rules to override default tendencies:

from speech_to_speech.LLM.voice_prompt import VOICE_SYSTEM_PROMPT_TAIL

CUSTOM_TAIL = VOICE_SYSTEM_PROMPT_TAIL + "\n- Always confirm understanding before taking any action.\n"

Injecting Tool Capabilities

Function tools are passed through the optional tool_section parameter. The build_tool_system_prompt helper from src/speech_to_speech/LLM/tool_call/tool_prompt.py formats tool schemas into the required block:

from speech_to_speech.LLM.tool_call.tool_prompt import build_tool_system_prompt
from speech_to_speech.LLM.tool_call.function_tool import FunctionTool

tools = [
    FunctionTool(
        type="function",
        name="weather",
        description="Fetch the current weather for a city.",
        parameters={"type": "object", "properties": {"city": {"type": "string"}}},
    )
]
tool_section = build_tool_system_prompt(tools)

Practical Code Examples

Basic Voice Prompt with Session Context

from speech_to_speech.LLM.voice_prompt import build_voice_system_prompt

session_prompt = "You are a friendly travel guide who loves giving short, vivid directions."
voice_prompt = build_voice_system_prompt(session_prompt)
print(voice_prompt)

This produces a complete system message with the spoken-conversation lead, your session context, and the standard constraint tail.

Voice Prompt with Function Tools

from speech_to_speech.LLM.tool_call.tool_prompt import build_tool_system_prompt
from speech_to_speech.LLM.tool_call.function_tool import FunctionTool
from speech_to_speech.LLM.voice_prompt import build_voice_system_prompt

session_prompt = "You are a helpful assistant that can check the weather."

tools = [
    FunctionTool(
        type="function",
        name="weather",
        description="Fetch the current weather for a city.",
        parameters={"type": "object", "properties": {"city": {"type": "string"}}},
    )
]
tool_section = build_tool_system_prompt(tools)

voice_prompt = build_voice_system_prompt(session_prompt, tool_section=tool_section)
print(voice_prompt)

Custom Builder with Modified Tail

from speech_to_speech.LLM.voice_prompt import (
    VOICE_SYSTEM_PROMPT_LEAD,
    VOICE_SYSTEM_PROMPT_TAIL,
)

# Extend tail with an additional behavioral rule

CUSTOM_TAIL = VOICE_SYSTEM_PROMPT_TAIL + "\n- Greet the user before providing any answer.\n"

def custom_build_voice_system_prompt(session_prompt, *, tool_section=""):
    return (
        f"{VOICE_SYSTEM_PROMPT_LEAD.rstrip()}\n\n"
        f"Session Prompt:\n{session_prompt.strip()}"
        f"{('\\n\\n' + tool_section.strip()) if tool_section else ''}\n\n"
        f"{CUSTOM_TAIL.rstrip()}"
    )

session_prompt = "You are a formal concierge at a luxury hotel."
voice_prompt = custom_build_voice_system_prompt(session_prompt)

Key Source Files for Voice Prompt Customization

File	Role
src/speech_to_speech/LLM/voice_prompt.py	Defines VOICE_SYSTEM_PROMPT_LEAD, VOICE_SYSTEM_PROMPT_TAIL, and build_voice_system_prompt()
src/speech_to_speech/LLM/language_model.py	Selects voice vs. text builder based on wants_audio
src/speech_to_speech/LLM/base_openai_compatible_language_model.py	Implements OpenAI-compatible interface with builder switching
tests/test_voice_prompt.py	Validates prompt structure, length limits, and tool handling

Summary

• Voice prompts use a four-part architecture: lead, session prompt, optional tool section, and tail
• build_voice_system_prompt() in voice_prompt.py is the central assembly function
• The tail has highest priority for behavioral control due to its final position
• Tool capabilities are injected via the tool_section parameter using build_tool_system_prompt()
• Audio vs. text routing is automatic based on the wants_audio flag in language_model.py

Frequently Asked Questions

Can I completely replace the default voice prompt instead of modifying components?

Yes. The build_voice_system_prompt function is a convenience builder; nothing prevents you from constructing your own system message string and passing it directly to the LLM handler. However, you will lose the automatic formatting guarantees (proper lead/tail placement, tool section injection) that the builder provides.

Why does the tool section go between the session prompt and the tail?

The positional ordering follows prompt engineering best practices: context first, capabilities second, constraints last. Placing tool descriptions after the session prompt lets the model understand what it can do within the established persona, while the final tail ensures output formatting rules override any conflicting tendencies introduced by the tool descriptions.

How do I verify my custom voice prompt produces valid output?

The repository includes tests/test_voice_prompt.py with assertions for prompt length, required section presence, and tool formatting. Run these tests against your modifications to ensure compatibility with the pipeline's expectations.

What happens if I set wants_audio=False in the pipeline?

The system switches to build_text_system_prompt, which omits the spoken-conversation lead and speech-optimized tail. The resulting prompt is optimized for text-only interaction without audio-specific constraints like brevity or hesitation handling.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}