How to Define and Use Custom Prompts for the LLM in the Hugging Face Speech‑to‑Speech Pipeline

Custom LLM prompts in the speech‑to‑speech pipeline are built from three parts (lead, session prompt, tail) using build_text_system_prompt() or build_voice_system_prompt(), then passed to your language model handler via the system_prompt parameter.

The Hugging Face speech-to-speech repository lets you fully customize how the underlying Large Language Model behaves during conversations. By defining custom prompts, you control the persona, contextual knowledge, and operational rules that guide the LLM's responses—whether you're running a text‑based chat or a real‑time voice interaction.

Understanding the Three‑Part Prompt Structure

The library constructs every system prompt from three layered components. This design keeps channel‑specific rules fixed while giving you full control over the dynamic context.

1. Lead: Channel Definition

The lead is a fixed instruction that tells the LLM which communication channel it's operating in. In src/speech_to_speech/LLM/text_prompt.py, the TEXT_SYSTEM_PROMPT_LEAD establishes text‑mode expectations, while src/speech_to_speech/LLM/voice_prompt.py defines VOICE_SYSTEM_PROMPT_LEAD for spoken interactions.

2. Session Prompt: Your Custom Context

The session prompt is where you inject your own content—persona descriptions, background knowledge, task definitions, or any instructions you want the model to retain throughout the conversation.

3. Tail: Hard Constraints

The tail contains the strongest formatting and behavioral rules. These are always appended last, ensuring they override earlier instructions. Text mode typically carries formatting constraints, while voice mode enforces brevity rules for natural spoken delivery.

Building Custom Prompts with the Helper Functions

The library provides two builder functions that assemble these three parts automatically.

build_text_system_prompt(session_prompt, tool_section="")

Located in src/speech_to_speech/LLM/text_prompt.py, this function constructs prompts for text‑only LLM interactions.

from speech_to_speech.LLM.text_prompt import build_text_system_prompt

session = "You are a technical documentation expert specializing in Python async patterns."
system_prompt = build_text_system_prompt(session)

build_voice_system_prompt(session_prompt, tool_section="")

Defined in src/speech_to_speech/LLM/voice_prompt.py, this variant optimizes the prompt for spoken conversation flow.

from speech_to_speech.LLM.voice_prompt import build_voice_system_prompt

session = "You are a concise travel assistant helping users plan trips."
tools = "Tools:\n- weather_lookup(city, date)\n- hotel_search(destination, budget)"
system_prompt = build_voice_system_prompt(session, tool_section=tools)

Injecting Custom Prompts into the Pipeline

Once you've constructed your system prompt, pass it to your LLM handler. The ChatCompletionsLanguageModel class in src/speech_to_speech/LLM/chat_completions_language_model.py accepts this through its system_prompt parameter.

from speech_to_speech.LLM.chat_completions_language_model import ChatCompletionsLanguageModel

llm = ChatCompletionsLanguageModel(
    model="gpt-4o-mini",
    system_prompt=system_prompt,  # Your custom-built prompt

    temperature=0.7,
    max_tokens=512,
)

The pipeline orchestrator in src/speech_to_speech/s2s_pipeline.py wires this LLM instance into the full speech‑to‑speech flow, ensuring your custom prompt becomes the first message in every backend request.

CLI Configuration for Custom Prompts

When launching the pipeline from the command line, use the --system_prompt flag to pass your custom prompt directly:

python -m speech_to_speech \
    --system_prompt "You are a medical assistant providing general health information. Always include a disclaimer."

This bypasses the builder functions and uses your raw string as the complete system prompt.

Complete Working Example

Here's a full implementation combining persona definition, tool specification, and pipeline initialization:

from speech_to_speech.LLM.voice_prompt import build_voice_system_prompt
from speech_to_speech.LLM.chat_completions_language_model import ChatCompletionsLanguageModel
from speech_to_speech.s2s_pipeline import S2SPipeline

# 1. Define session context and available tools

session = """You are a coding interview coach. 
Your tone is encouraging but rigorous. 
Focus on algorithmic thinking rather than syntax details."""

tools = """Tools:
- run_tests(code_snippet): Executes the candidate's solution against hidden test cases
- hint_generator(problem_type, difficulty): Provides scaffolded hints without full solutions"""

# 2. Build the voice-optimized system prompt

system_prompt = build_voice_system_prompt(session, tool_section=tools)

# 3. Initialize the LLM handler with custom prompt

llm = ChatCompletionsLanguageModel(
    model="gpt-4o",
    system_prompt=system_prompt,
    temperature=0.4,
)

# 4. The pipeline receives this handler and uses your prompt throughout

# pipeline = S2SPipeline(llm_handler=llm, ...)

How the Prompt Flows Through the System

The assembled prompt follows this path through the codebase:

1. Builder functions (text_prompt.py/voice_prompt.py) concatenate lead + session prompt + tool section + tail
2. Language model base class (language_model.py) formats messages and injects the system prompt as the initial message
3. Concrete implementations (chat_completions_language_model.py) send the formatted request to OpenAI‑compatible endpoints
4. Pipeline orchestrator (s2s_pipeline.py) manages the full speech‑to‑speech loop with your custom behavior locked in

Key Files for Custom Prompt Development

File	Purpose
src/speech_to_speech/LLM/text_prompt.py	Text channel lead, tail, and build_text_system_prompt
src/speech_to_speech/LLM/voice_prompt.py	Voice channel lead, tail, and build_voice_system_prompt
src/speech_to_speech/LLM/language_model.py	Abstract base handling message formatting
src/speech_to_speech/LLM/chat_completions_language_model.py	OpenAI‑compatible implementation receiving system_prompt
src/speech_to_speech/s2s_pipeline.py	Pipeline orchestration wiring everything together

Summary

• Custom prompts use three parts: invariant lead (channel), your session prompt (context), invariant tail (constraints)
• Use build_text_system_prompt() for text chats and build_voice_system_prompt() for voice interactions, both accepting optional tool_section strings
• Pass prompts to LLM handlers via the system_prompt parameter in ChatCompletionsLanguageModel or similar classes
• Tail constraints always apply last, guaranteeing your formatting and brevity rules are respected
• Core files: text_prompt.py, voice_prompt.py, chat_completions_language_model.py, and s2s_pipeline.py

Frequently Asked Questions

Can I modify the lead and tail sections directly?

The lead and tail are module‑level constants (TEXT_SYSTEM_PROMPT_LEAD, VOICE_SYSTEM_PROMPT_TAIL, etc.) defined in text_prompt.py and voice_prompt.py. While you can edit these source files, the recommended approach is to pass your full desired prompt via the system_prompt parameter or CLI flag, which overrides the builder entirely.

How do tool calls work with custom prompts?

The optional tool_section parameter in both builder functions appends your tool definitions after the session prompt but before the tail. According to the speech-to-speech source code, this placement ensures the LLM knows available tools while still prioritizing the hard constraints in the tail.

What's the difference between text and voice system prompts?

The voice prompt (build_voice_system_prompt) includes constraints optimized for spoken output—typically brevity rules, avoidance of markdown formatting, and natural turn-taking cues. The text prompt (build_text_system_prompt) preserves richer formatting capabilities. Both use identical builder signatures for consistent API design.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}