# How to Use Voice Prompts to Customize LLM Behavior in the Hugging Face Speech-to-Speech Repository

> Learn how to use voice prompts to customize LLM behavior in Hugging Face's Speech-to-Speech repository. Control spoken output with lead, session prompt, tools, and tail.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-06

---

**Voice prompts in the Speech-to-Speech repository combine a fixed lead, a customizable session prompt, an optional tool section, and a constraint-enforcing tail to shape spoken LLM output.**

The `huggingface/speech-to-speech` library provides a structured approach to controlling how large language models behave during audio conversations. Rather than sending raw text prompts, the system uses a layered **voice system prompt architecture** that separates conversational framing from behavioral constraints. This design lets developers inject personality, goals, and tool capabilities while maintaining consistent speech-optimized output rules.

## How the Voice Prompt Architecture Works

The voice prompt system is implemented in [`src/speech_to_speech/LLM/voice_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/voice_prompt.py). It decomposes the system message into four ordered components:

1. **Lead** (`VOICE_SYSTEM_PROMPT_LEAD`) — establishes that the interaction is spoken and audio-oriented
2. **Session prompt** — user-supplied persona, scenario, or task description
3. **Tool section** — optional function-calling instructions passed via `tool_section` parameter
4. **Tail** (`VOICE_SYSTEM_PROMPT_TAIL`) — hard constraints on brevity, naturalness, and tool-use policy

The `build_voice_system_prompt(session_prompt, *, tool_section="")` function assembles these pieces into the final string sent to the LLM.

### When Voice Prompts Are Selected

The pipeline automatically chooses between voice and text prompt builders based on the `wants_audio` flag. In [`src/speech_to_speech/LLM/language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/language_model.py), the selection logic is:

```python
builder = build_voice_system_prompt if wants_audio else build_text_system_prompt

```

This ensures that audio-targeted requests receive the spoken-conversation framing while text-only interactions use a simpler format.

## Customizing the Core Prompt Components

### Modifying the Lead or Tail Constants

The lead and tail are defined as module-level strings in [`voice_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/voice_prompt.py). To permanently change the conversational premise or output constraints, edit these constants directly:

| Constant | Purpose | Typical Content |
|----------|---------|---------------|
| `VOICE_SYSTEM_PROMPT_LEAD` | Sets spoken-interaction context | "You are in a spoken conversation with a user..." |
| `VOICE_SYSTEM_PROMPT_TAIL` | Enforces speech-optimized rules | Brevity commands, hesitation handling, tool-use syntax |

Changes to these constants affect all voice prompts system-wide.

### Appending High-Priority Constraints

The tail appears last in the prompt, giving it the highest positional influence over model behavior. Append additional rules to override default tendencies:

```python
from speech_to_speech.LLM.voice_prompt import VOICE_SYSTEM_PROMPT_TAIL

CUSTOM_TAIL = VOICE_SYSTEM_PROMPT_TAIL + "\n- Always confirm understanding before taking any action.\n"

```

### Injecting Tool Capabilities

Function tools are passed through the optional `tool_section` parameter. The `build_tool_system_prompt` helper from [`src/speech_to_speech/LLM/tool_call/tool_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/tool_call/tool_prompt.py) formats tool schemas into the required block:

```python
from speech_to_speech.LLM.tool_call.tool_prompt import build_tool_system_prompt
from speech_to_speech.LLM.tool_call.function_tool import FunctionTool

tools = [
    FunctionTool(
        type="function",
        name="weather",
        description="Fetch the current weather for a city.",
        parameters={"type": "object", "properties": {"city": {"type": "string"}}},
    )
]
tool_section = build_tool_system_prompt(tools)

```

## Practical Code Examples

### Basic Voice Prompt with Session Context

```python
from speech_to_speech.LLM.voice_prompt import build_voice_system_prompt

session_prompt = "You are a friendly travel guide who loves giving short, vivid directions."
voice_prompt = build_voice_system_prompt(session_prompt)
print(voice_prompt)

```

This produces a complete system message with the spoken-conversation lead, your session context, and the standard constraint tail.

### Voice Prompt with Function Tools

```python
from speech_to_speech.LLM.tool_call.tool_prompt import build_tool_system_prompt
from speech_to_speech.LLM.tool_call.function_tool import FunctionTool
from speech_to_speech.LLM.voice_prompt import build_voice_system_prompt

session_prompt = "You are a helpful assistant that can check the weather."

tools = [
    FunctionTool(
        type="function",
        name="weather",
        description="Fetch the current weather for a city.",
        parameters={"type": "object", "properties": {"city": {"type": "string"}}},
    )
]
tool_section = build_tool_system_prompt(tools)

voice_prompt = build_voice_system_prompt(session_prompt, tool_section=tool_section)
print(voice_prompt)

```

### Custom Builder with Modified Tail

```python
from speech_to_speech.LLM.voice_prompt import (
    VOICE_SYSTEM_PROMPT_LEAD,
    VOICE_SYSTEM_PROMPT_TAIL,
)

# Extend tail with an additional behavioral rule

CUSTOM_TAIL = VOICE_SYSTEM_PROMPT_TAIL + "\n- Greet the user before providing any answer.\n"

def custom_build_voice_system_prompt(session_prompt, *, tool_section=""):
    return (
        f"{VOICE_SYSTEM_PROMPT_LEAD.rstrip()}\n\n"
        f"Session Prompt:\n{session_prompt.strip()}"
        f"{('\\n\\n' + tool_section.strip()) if tool_section else ''}\n\n"
        f"{CUSTOM_TAIL.rstrip()}"
    )

session_prompt = "You are a formal concierge at a luxury hotel."
voice_prompt = custom_build_voice_system_prompt(session_prompt)

```

## Key Source Files for Voice Prompt Customization

| File | Role |
|------|------|
| [`src/speech_to_speech/LLM/voice_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/voice_prompt.py) | Defines `VOICE_SYSTEM_PROMPT_LEAD`, `VOICE_SYSTEM_PROMPT_TAIL`, and `build_voice_system_prompt()` |
| [`src/speech_to_speech/LLM/language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/language_model.py) | Selects voice vs. text builder based on `wants_audio` |
| [`src/speech_to_speech/LLM/base_openai_compatible_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/base_openai_compatible_language_model.py) | Implements OpenAI-compatible interface with builder switching |
| [`tests/test_voice_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/tests/test_voice_prompt.py) | Validates prompt structure, length limits, and tool handling |

## Summary

- **Voice prompts use a four-part architecture**: lead, session prompt, optional tool section, and tail
- **`build_voice_system_prompt()`** in [`voice_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/voice_prompt.py) is the central assembly function
- **The tail has highest priority** for behavioral control due to its final position
- **Tool capabilities are injected** via the `tool_section` parameter using `build_tool_system_prompt()`
- **Audio vs. text routing** is automatic based on the `wants_audio` flag in [`language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/language_model.py)

## Frequently Asked Questions

### Can I completely replace the default voice prompt instead of modifying components?

Yes. The `build_voice_system_prompt` function is a convenience builder; nothing prevents you from constructing your own system message string and passing it directly to the LLM handler. However, you will lose the automatic formatting guarantees (proper lead/tail placement, tool section injection) that the builder provides.

### Why does the tool section go between the session prompt and the tail?

The positional ordering follows prompt engineering best practices: context first, capabilities second, constraints last. Placing tool descriptions after the session prompt lets the model understand what it can do within the established persona, while the final tail ensures output formatting rules override any conflicting tendencies introduced by the tool descriptions.

### How do I verify my custom voice prompt produces valid output?

The repository includes [`tests/test_voice_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/tests/test_voice_prompt.py) with assertions for prompt length, required section presence, and tool formatting. Run these tests against your modifications to ensure compatibility with the pipeline's expectations.

### What happens if I set `wants_audio=False` in the pipeline?

The system switches to `build_text_system_prompt`, which omits the spoken-conversation lead and speech-optimized tail. The resulting prompt is optimized for text-only interaction without audio-specific constraints like brevity or hesitation handling.