# How to Define and Use Custom Prompts for the LLM in the Hugging Face Speech‑to‑Speech Pipeline

> Learn to define and use custom prompts for the LLM in the Hugging Face speech-to-speech pipeline. Build system prompts using build_text_system_prompt or build_voice_system_prompt and pass them to your handler.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-05

---

**Custom LLM prompts in the speech‑to‑speech pipeline are built from three parts (lead, session prompt, tail) using `build_text_system_prompt()` or `build_voice_system_prompt()`, then passed to your language model handler via the `system_prompt` parameter.**

The Hugging Face `speech-to-speech` repository lets you fully customize how the underlying Large Language Model behaves during conversations. By defining custom prompts, you control the persona, contextual knowledge, and operational rules that guide the LLM's responses—whether you're running a text‑based chat or a real‑time voice interaction.

## Understanding the Three‑Part Prompt Structure

The library constructs every system prompt from three layered components. This design keeps channel‑specific rules fixed while giving you full control over the dynamic context.

### 1. Lead: Channel Definition

The **lead** is a fixed instruction that tells the LLM which communication channel it's operating in. In [`src/speech_to_speech/LLM/text_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/text_prompt.py), the `TEXT_SYSTEM_PROMPT_LEAD` establishes text‑mode expectations, while [`src/speech_to_speech/LLM/voice_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/voice_prompt.py) defines `VOICE_SYSTEM_PROMPT_LEAD` for spoken interactions.

### 2. Session Prompt: Your Custom Context

The **session prompt** is where you inject your own content—persona descriptions, background knowledge, task definitions, or any instructions you want the model to retain throughout the conversation.

### 3. Tail: Hard Constraints

The **tail** contains the strongest formatting and behavioral rules. These are always appended last, ensuring they override earlier instructions. Text mode typically carries formatting constraints, while voice mode enforces brevity rules for natural spoken delivery.

## Building Custom Prompts with the Helper Functions

The library provides two builder functions that assemble these three parts automatically.

### `build_text_system_prompt(session_prompt, tool_section="")`

Located in [`src/speech_to_speech/LLM/text_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/text_prompt.py), this function constructs prompts for text‑only LLM interactions.

```python
from speech_to_speech.LLM.text_prompt import build_text_system_prompt

session = "You are a technical documentation expert specializing in Python async patterns."
system_prompt = build_text_system_prompt(session)

```

### `build_voice_system_prompt(session_prompt, tool_section="")`

Defined in [`src/speech_to_speech/LLM/voice_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/voice_prompt.py), this variant optimizes the prompt for spoken conversation flow.

```python
from speech_to_speech.LLM.voice_prompt import build_voice_system_prompt

session = "You are a concise travel assistant helping users plan trips."
tools = "Tools:\n- weather_lookup(city, date)\n- hotel_search(destination, budget)"
system_prompt = build_voice_system_prompt(session, tool_section=tools)

```

## Injecting Custom Prompts into the Pipeline

Once you've constructed your system prompt, pass it to your LLM handler. The `ChatCompletionsLanguageModel` class in [`src/speech_to_speech/LLM/chat_completions_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/chat_completions_language_model.py) accepts this through its `system_prompt` parameter.

```python
from speech_to_speech.LLM.chat_completions_language_model import ChatCompletionsLanguageModel

llm = ChatCompletionsLanguageModel(
    model="gpt-4o-mini",
    system_prompt=system_prompt,  # Your custom-built prompt

    temperature=0.7,
    max_tokens=512,
)

```

The pipeline orchestrator in [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) wires this LLM instance into the full speech‑to‑speech flow, ensuring your custom prompt becomes the first message in every backend request.

## CLI Configuration for Custom Prompts

When launching the pipeline from the command line, use the `--system_prompt` flag to pass your custom prompt directly:

```bash
python -m speech_to_speech \
    --system_prompt "You are a medical assistant providing general health information. Always include a disclaimer."

```

This bypasses the builder functions and uses your raw string as the complete system prompt.

## Complete Working Example

Here's a full implementation combining persona definition, tool specification, and pipeline initialization:

```python
from speech_to_speech.LLM.voice_prompt import build_voice_system_prompt
from speech_to_speech.LLM.chat_completions_language_model import ChatCompletionsLanguageModel
from speech_to_speech.s2s_pipeline import S2SPipeline

# 1. Define session context and available tools

session = """You are a coding interview coach. 
Your tone is encouraging but rigorous. 
Focus on algorithmic thinking rather than syntax details."""

tools = """Tools:
- run_tests(code_snippet): Executes the candidate's solution against hidden test cases
- hint_generator(problem_type, difficulty): Provides scaffolded hints without full solutions"""

# 2. Build the voice-optimized system prompt

system_prompt = build_voice_system_prompt(session, tool_section=tools)

# 3. Initialize the LLM handler with custom prompt

llm = ChatCompletionsLanguageModel(
    model="gpt-4o",
    system_prompt=system_prompt,
    temperature=0.4,
)

# 4. The pipeline receives this handler and uses your prompt throughout

# pipeline = S2SPipeline(llm_handler=llm, ...)

```

## How the Prompt Flows Through the System

The assembled prompt follows this path through the codebase:

1. **Builder functions** ([`text_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/text_prompt.py)/[`voice_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/voice_prompt.py)) concatenate lead + session prompt + tool section + tail
2. **Language model base class** ([`language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/language_model.py)) formats messages and injects the system prompt as the initial message
3. **Concrete implementations** ([`chat_completions_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/chat_completions_language_model.py)) send the formatted request to OpenAI‑compatible endpoints
4. **Pipeline orchestrator** ([`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py)) manages the full speech‑to‑speech loop with your custom behavior locked in

## Key Files for Custom Prompt Development

| File | Purpose |
|------|---------|
| [`src/speech_to_speech/LLM/text_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/text_prompt.py) | Text channel lead, tail, and `build_text_system_prompt` |
| [`src/speech_to_speech/LLM/voice_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/voice_prompt.py) | Voice channel lead, tail, and `build_voice_system_prompt` |
| [`src/speech_to_speech/LLM/language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/language_model.py) | Abstract base handling message formatting |
| [`src/speech_to_speech/LLM/chat_completions_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/chat_completions_language_model.py) | OpenAI‑compatible implementation receiving `system_prompt` |
| [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) | Pipeline orchestration wiring everything together |

## Summary

- **Custom prompts use three parts**: invariant lead (channel), your session prompt (context), invariant tail (constraints)
- **Use `build_text_system_prompt()`** for text chats and **`build_voice_system_prompt()`** for voice interactions, both accepting optional `tool_section` strings
- **Pass prompts to LLM handlers** via the `system_prompt` parameter in `ChatCompletionsLanguageModel` or similar classes
- **Tail constraints always apply last**, guaranteeing your formatting and brevity rules are respected
- **Core files**: [`text_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/text_prompt.py), [`voice_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/voice_prompt.py), [`chat_completions_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/chat_completions_language_model.py), and [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py)

## Frequently Asked Questions

### Can I modify the lead and tail sections directly?

The lead and tail are module‑level constants (`TEXT_SYSTEM_PROMPT_LEAD`, `VOICE_SYSTEM_PROMPT_TAIL`, etc.) defined in [`text_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/text_prompt.py) and [`voice_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/voice_prompt.py). While you can edit these source files, the recommended approach is to pass your full desired prompt via the `system_prompt` parameter or CLI flag, which overrides the builder entirely.

### How do tool calls work with custom prompts?

The optional `tool_section` parameter in both builder functions appends your tool definitions after the session prompt but before the tail. According to the `speech-to-speech` source code, this placement ensures the LLM knows available tools while still prioritizing the hard constraints in the tail.

### What's the difference between text and voice system prompts?

The voice prompt (`build_voice_system_prompt`) includes constraints optimized for spoken output—typically brevity rules, avoidance of markdown formatting, and natural turn-taking cues. The text prompt (`build_text_system_prompt`) preserves richer formatting capabilities. Both use identical builder signatures for consistent API design.