How to Implement Tool Calling in the Hugging Face Speech-to-Speech Voice Pipeline

Enable your voice AI to trigger external actions by adding FunctionTool definitions, injecting tool prompts into the system message, and letting the pipeline parse <code> blocks into validated ResponseFunctionToolCall objects.

The Hugging Face speech-to-speech repository provides a complete OpenAI Realtime-compatible stack for building voice-enabled AI agents. Tool calling allows these agents to execute real-world actions—controlling smart devices, querying APIs, or modifying databases—while maintaining natural conversation flow. This guide walks through the exact implementation based on the source code in huggingface/speech-to-speech.

---

Architecture Overview

The tool-calling stack spans three layers that mirror the OpenAI Realtime protocol:

Layer	Purpose	Source File
Tool Definition	Create FunctionTool objects with JSON schemas	src/speech_to_speech/LLM/tool_call/function_tool.py
Prompt Construction	Inject tool signatures and <code> delimiters into system prompts	src/speech_to_speech/LLM/tool_call/tool_prompt.py
Runtime Parsing	Extract, validate, and emit ResponseFunctionToolCall objects during streaming	src/speech_to_speech/LLM/tool_call/function_call.py, src/speech_to_speech/LLM/language_model.py

---

Step 1: Declare Your Tools with FunctionTool

Tools are represented as FunctionTool instances that subclass OpenAI's RealtimeFunctionTool. Each tool needs a name, description, and JSON Schema parameters.

from speech_to_speech.LLM.tool_call.function_tool import FunctionTool

light_tool = FunctionTool()
light_tool.name = "set_lights"
light_tool.description = "Set the smart-home lights to a colour or brightness."
light_tool.type = "function"
light_tool.parameters = {
    "type": "object",
    "properties": {
        "room": {
            "type": "string",
            "description": "Room identifier, e.g. 'kitchen'"
        },
        "color": {
            "type": "string",
            "enum": ["red", "green", "blue"],
            "description": "Desired colour"
        },
        "brightness": {
            "type": "integer",
            "minimum": 0,
            "maximum": 100
        }
    },
    "required": ["room"]
}

The FunctionTool.render() method automatically converts this schema into a Python-style function signature for inclusion in the LLM's system prompt.

---

Step 2: Attach Tools to the Session

Register your tools in the RuntimeConfig before starting a conversation turn. The tool_choice field controls whether the model can auto-select tools or must use a specific one.

from speech_to_speech.api.openai_realtime.runtime_config import RuntimeConfig

runtime_cfg = RuntimeConfig()
runtime_cfg.session.tools = [light_tool]
runtime_cfg.session.tool_choice = "auto"  # "auto", "none", or a specific tool name

Behind the scenes, BaseLanguageModelHandler._apply_instructions extracts this list and stores it on the per-turn StreamContext for validation during parsing.

---

Step 3: Build the Tool-Aware System Prompt

Use build_tool_system_prompt to inject tool definitions and calling conventions into the LLM's instructions.

from speech_to_speech.LLM.tool_call.tool_prompt import (
    build_tool_system_prompt,
    ENTER_CODE,  # "<code>" delimiter

    END_CODE,    # "</code>" delimiter

)

tool_section = build_tool_system_prompt(
    tools=[light_tool],
    text_only=False,  # False = voice mode with "speak first" guidance

)

The resulting prompt section includes:

• Python-style function signatures for all available tools
• The exact delimiter format: <code>function_name(arg='value')</code>
• Voice-mode instructions requiring natural speech before the code block

---

Step 4: Process LLM Output and Extract Tool Calls

During streaming generation, BaseLanguageModelHandler._process_printable_text detects when the LLM emits a <code> block. It splits the preceding text into spoken sentences and processes the code block through extract_function_calls_from_text.

from speech_to_speech.LLM.tool_call.function_call import extract_function_calls_from_text

text_with_code = "I'll dim the lights for you. <code>set_lights(room='living room', brightness=30)</code>"

outside_text, function_calls = extract_function_calls_from_text(
    text_with_code,
    block_regex=r"<code>(.*?)</code>"
)

The parse_function_call function (tokenizer-aware) handles nested parentheses, quoted strings, tuples, and dictionaries to extract the function name and keyword arguments.

---

Step 5: Validate and Convert to Realtime Protocol

Each extracted FunctionToolCall is validated and converted via to_realtime_function_tool_call:

# Inside the handler during processing

for fc in function_calls:
    try:
        realtime_call = fc.to_realtime_function_tool_call(
            available_tools=ctx.function_tools  # dict[str, FunctionTool]

        )
        ctx.tools.append(realtime_call)
    except ValueError as e:
        logger.warning(f"Invalid tool call discarded: {e}")

Validation enforces:

• The called tool must exist in ctx.function_tools
• Only keyword arguments are permitted (no positional args)
• All required parameters must be present
• No undeclared parameters are allowed

---

Step 6: Emit Tool Calls to Downstream Services

Validated calls are packaged into LLMResponseChunk objects and yielded to the Realtime server, which serializes them as RealtimeConversationItemFunctionCall events.

from speech_to_speech.pipeline.messages import LLMResponseChunk

yield LLMResponseChunk(
    text=None,  # tool calls don't include spoken text

    tools=ctx.tools,  # list[ResponseFunctionToolCall]

    turn_id=ctx.turn_id,
    is_finished=True,
)

The client receives these events and can execute the actual API calls, then optionally return results to continue the conversation.

---

Complete Working Example

from speech_to_speech.LLM.tool_call.function_tool import FunctionTool
from speech_to_speech.LLM.tool_call.tool_prompt import build_tool_system_prompt
from speech_to_speech.LLM.language_model import LanguageModelHandler
from speech_to_speech.api.openai_realtime.runtime_config import RuntimeConfig
from speech_to_speech.pipeline.handler_types import GenerateResponseRequest

# 1. Define the tool

weather_tool = FunctionTool()
weather_tool.name = "get_weather"
weather_tool.description = "Get current weather for a location."
weather_tool.parameters = {
    "type": "object",
    "properties": {
        "city": {"type": "string", "description": "City name"},
        "unit": {"type": "string", "enum": ["celsius", "fahrenheit"], "default": "celsius"}
    },
    "required": ["city"]
}

# 2. Configure session

runtime_cfg = RuntimeConfig()
runtime_cfg.session.tools = [weather_tool]
runtime_cfg.session.tool_choice = "auto"

# 3. Build prompt (normally done automatically by handler)

tool_prompt = build_tool_system_prompt([weather_tool], text_only=False)

# 4. Initialize handler

handler = LanguageModelHandler()
handler.setup(
    model_name="Qwen/Qwen3-4B-Instruct-2507",
    device="cuda",
    torch_dtype="float16",
    stream_batch_sentences=2,
)

# 5. Process request

request = GenerateResponseRequest(
    turn_id="turn-456",
    turn_revision=0,
    runtime_config=runtime_cfg,
)

for chunk in handler.process(request):
    if chunk.tools:
        for tool in chunk.tools:
            print(f"EXECUTE: {tool.name}({tool.arguments})")
            # Client would call actual weather API here

    elif chunk.text:
        print(f"ASSISTANT: {chunk.text}")

---

Key Implementation Files

File	Role
src/speech_to_speech/LLM/tool_call/function_tool.py	FunctionTool class for tool definition and schema rendering
src/speech_to_speech/LLM/tool_call/tool_prompt.py	build_tool_system_prompt() and build_tool_system_prompt_text_only() templates
src/speech_to_speech/LLM/tool_call/function_call.py	extract_function_calls_from_text(), parse_function_call(), FunctionToolCall
src/speech_to_speech/LLM/language_model.py	BaseLanguageModelHandler._process_printable_text() for streaming integration
src/speech_to_speech/pipeline/messages.py	LLMResponseChunk dataclass carrying parsed tools

---

Voice-Specific Considerations

The pipeline enforces two critical behaviors for voice interaction:

• Speak-first requirement: The voice prompt template (via build_tool_system_prompt with text_only=False) instructs the model to provide natural speech before emitting any <code> block, preventing silent tool calls that confuse users
• Single call per response: BaseLanguageModelHandler._process_printable_text explicitly filters to only the first valid tool call, logging warnings for additional calls

Speculative turn handling allows the stream to abort if new user speech arrives, keeping conversations responsive even during tool-elaboration phases.

---

Summary

• Define tools with FunctionTool including JSON Schema parameters
• Register tools in RuntimeConfig.session.tools with tool_choice control
• Inject prompts via build_tool_system_prompt to teach the LLM the <code> calling convention
• Parse automatically through extract_function_calls_from_text during streaming
• Validate strictly against ctx.function_tools with to_realtime_function_tool_call
• Emit downstream as ResponseFunctionToolCall objects in LLMResponseChunk.tools

---

Frequently Asked Questions

How does the LLM know when to call a tool versus responding normally?

The system prompt explicitly lists available tools with their descriptions and schemas, plus instructions that the model "can use tools if needed." The tool_choice setting ("auto", "none", or a specific name) further guides this decision. According to the tool_prompt.py templates, voice mode adds guidance to first speak naturally, then optionally include a code block.

What happens if the LLM generates an invalid tool call?

The FunctionToolCall.to_realtime_function_tool_call method validates against the registered tool schema. If the tool doesn't exist, required arguments are missing, or undeclared parameters are present, it raises ValueError and logs a warning. The pipeline discards the invalid call and continues processing any valid text or subsequent calls.

Can I force the model to always use a specific tool?

Yes. Set runtime_cfg.session.tool_choice = "tool_name" instead of "auto". Per the implementation in language_model.py, this constrains the model to only that tool, though it may still choose not to call it depending on the conversation context.

Is streaming interrupted when a tool call is detected?

No—tool calls are extracted seamlessly during streaming. The BaseLanguageModelHandler._process_printable_text method buffers text until it detects the ENTER_CODE delimiter, yields preceding sentences as normal speech chunks, then parses and validates the code block without breaking the stream flow.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}