# How to Implement Tool Calling in the Hugging Face Speech-to-Speech Voice Pipeline > Implement tool calling in Hugging Face speech-to-speech. Trigger external actions by defining tools, injecting prompts, and parsing code blocks into function calls. - Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech) - Tags: how-to-guide - Published: 2026-08-05 --- **Enable your voice AI to trigger external actions by adding FunctionTool definitions, injecting tool prompts into the system message, and letting the pipeline parse `` blocks into validated ResponseFunctionToolCall objects.** The **Hugging Face speech-to-speech** repository provides a complete OpenAI Realtime-compatible stack for building voice-enabled AI agents. Tool calling allows these agents to execute real-world actions—controlling smart devices, querying APIs, or modifying databases—while maintaining natural conversation flow. This guide walks through the exact implementation based on the source code in `huggingface/speech-to-speech`. --- ## Architecture Overview The tool-calling stack spans three layers that mirror the OpenAI Realtime protocol: | Layer | Purpose | Source File | |-------|---------|-------------| | **Tool Definition** | Create `FunctionTool` objects with JSON schemas | [`src/speech_to_speech/LLM/tool_call/function_tool.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/tool_call/function_tool.py) | | **Prompt Construction** | Inject tool signatures and `` delimiters into system prompts | [`src/speech_to_speech/LLM/tool_call/tool_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/tool_call/tool_prompt.py) | | **Runtime Parsing** | Extract, validate, and emit `ResponseFunctionToolCall` objects during streaming | [`src/speech_to_speech/LLM/tool_call/function_call.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/tool_call/function_call.py), [`src/speech_to_speech/LLM/language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/language_model.py) | --- ## Step 1: Declare Your Tools with FunctionTool Tools are represented as `FunctionTool` instances that subclass OpenAI's `RealtimeFunctionTool`. Each tool needs a name, description, and JSON Schema parameters. ```python from speech_to_speech.LLM.tool_call.function_tool import FunctionTool light_tool = FunctionTool() light_tool.name = "set_lights" light_tool.description = "Set the smart-home lights to a colour or brightness." light_tool.type = "function" light_tool.parameters = { "type": "object", "properties": { "room": { "type": "string", "description": "Room identifier, e.g. 'kitchen'" }, "color": { "type": "string", "enum": ["red", "green", "blue"], "description": "Desired colour" }, "brightness": { "type": "integer", "minimum": 0, "maximum": 100 } }, "required": ["room"] } ``` The `FunctionTool.render()` method automatically converts this schema into a Python-style function signature for inclusion in the LLM's system prompt. --- ## Step 2: Attach Tools to the Session Register your tools in the `RuntimeConfig` before starting a conversation turn. The `tool_choice` field controls whether the model can auto-select tools or must use a specific one. ```python from speech_to_speech.api.openai_realtime.runtime_config import RuntimeConfig runtime_cfg = RuntimeConfig() runtime_cfg.session.tools = [light_tool] runtime_cfg.session.tool_choice = "auto" # "auto", "none", or a specific tool name ``` Behind the scenes, `BaseLanguageModelHandler._apply_instructions` extracts this list and stores it on the per-turn `StreamContext` for validation during parsing. --- ## Step 3: Build the Tool-Aware System Prompt Use `build_tool_system_prompt` to inject tool definitions and calling conventions into the LLM's instructions. ```python from speech_to_speech.LLM.tool_call.tool_prompt import ( build_tool_system_prompt, ENTER_CODE, # "" delimiter END_CODE, # "" delimiter ) tool_section = build_tool_system_prompt( tools=[light_tool], text_only=False, # False = voice mode with "speak first" guidance ) ``` The resulting prompt section includes: - Python-style function signatures for all available tools - The exact delimiter format: `function_name(arg='value')` - Voice-mode instructions requiring natural speech before the code block --- ## Step 4: Process LLM Output and Extract Tool Calls During streaming generation, `BaseLanguageModelHandler._process_printable_text` detects when the LLM emits a `` block. It splits the preceding text into spoken sentences and processes the code block through `extract_function_calls_from_text`. ```python from speech_to_speech.LLM.tool_call.function_call import extract_function_calls_from_text text_with_code = "I'll dim the lights for you. set_lights(room='living room', brightness=30)" outside_text, function_calls = extract_function_calls_from_text( text_with_code, block_regex=r"(.*?)" ) ``` The `parse_function_call` function (tokenizer-aware) handles nested parentheses, quoted strings, tuples, and dictionaries to extract the function name and keyword arguments. --- ## Step 5: Validate and Convert to Realtime Protocol Each extracted `FunctionToolCall` is validated and converted via `to_realtime_function_tool_call`: ```python # Inside the handler during processing for fc in function_calls: try: realtime_call = fc.to_realtime_function_tool_call( available_tools=ctx.function_tools # dict[str, FunctionTool] ) ctx.tools.append(realtime_call) except ValueError as e: logger.warning(f"Invalid tool call discarded: {e}") ``` Validation enforces: - The called tool must exist in `ctx.function_tools` - Only keyword arguments are permitted (no positional args) - All `required` parameters must be present - No undeclared parameters are allowed --- ## Step 6: Emit Tool Calls to Downstream Services Validated calls are packaged into `LLMResponseChunk` objects and yielded to the Realtime server, which serializes them as `RealtimeConversationItemFunctionCall` events. ```python from speech_to_speech.pipeline.messages import LLMResponseChunk yield LLMResponseChunk( text=None, # tool calls don't include spoken text tools=ctx.tools, # list[ResponseFunctionToolCall] turn_id=ctx.turn_id, is_finished=True, ) ``` The client receives these events and can execute the actual API calls, then optionally return results to continue the conversation. --- ## Complete Working Example ```python from speech_to_speech.LLM.tool_call.function_tool import FunctionTool from speech_to_speech.LLM.tool_call.tool_prompt import build_tool_system_prompt from speech_to_speech.LLM.language_model import LanguageModelHandler from speech_to_speech.api.openai_realtime.runtime_config import RuntimeConfig from speech_to_speech.pipeline.handler_types import GenerateResponseRequest # 1. Define the tool weather_tool = FunctionTool() weather_tool.name = "get_weather" weather_tool.description = "Get current weather for a location." weather_tool.parameters = { "type": "object", "properties": { "city": {"type": "string", "description": "City name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"], "default": "celsius"} }, "required": ["city"] } # 2. Configure session runtime_cfg = RuntimeConfig() runtime_cfg.session.tools = [weather_tool] runtime_cfg.session.tool_choice = "auto" # 3. Build prompt (normally done automatically by handler) tool_prompt = build_tool_system_prompt([weather_tool], text_only=False) # 4. Initialize handler handler = LanguageModelHandler() handler.setup( model_name="Qwen/Qwen3-4B-Instruct-2507", device="cuda", torch_dtype="float16", stream_batch_sentences=2, ) # 5. Process request request = GenerateResponseRequest( turn_id="turn-456", turn_revision=0, runtime_config=runtime_cfg, ) for chunk in handler.process(request): if chunk.tools: for tool in chunk.tools: print(f"EXECUTE: {tool.name}({tool.arguments})") # Client would call actual weather API here elif chunk.text: print(f"ASSISTANT: {chunk.text}") ``` --- ## Key Implementation Files | File | Role | |------|------| | [`src/speech_to_speech/LLM/tool_call/function_tool.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/tool_call/function_tool.py) | `FunctionTool` class for tool definition and schema rendering | | [`src/speech_to_speech/LLM/tool_call/tool_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/tool_call/tool_prompt.py) | `build_tool_system_prompt()` and `build_tool_system_prompt_text_only()` templates | | [`src/speech_to_speech/LLM/tool_call/function_call.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/tool_call/function_call.py) | `extract_function_calls_from_text()`, `parse_function_call()`, `FunctionToolCall` | | [`src/speech_to_speech/LLM/language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/language_model.py) | `BaseLanguageModelHandler._process_printable_text()` for streaming integration | | [`src/speech_to_speech/pipeline/messages.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/messages.py) | `LLMResponseChunk` dataclass carrying parsed tools | --- ## Voice-Specific Considerations The pipeline enforces two critical behaviors for voice interaction: - **Speak-first requirement**: The voice prompt template (via `build_tool_system_prompt` with `text_only=False`) instructs the model to provide natural speech before emitting any `` block, preventing silent tool calls that confuse users - **Single call per response**: `BaseLanguageModelHandler._process_printable_text` explicitly filters to only the first valid tool call, logging warnings for additional calls Speculative turn handling allows the stream to abort if new user speech arrives, keeping conversations responsive even during tool-elaboration phases. --- ## Summary - **Define tools** with `FunctionTool` including JSON Schema parameters - **Register tools** in `RuntimeConfig.session.tools` with `tool_choice` control - **Inject prompts** via `build_tool_system_prompt` to teach the LLM the `` calling convention - **Parse automatically** through `extract_function_calls_from_text` during streaming - **Validate strictly** against `ctx.function_tools` with `to_realtime_function_tool_call` - **Emit downstream** as `ResponseFunctionToolCall` objects in `LLMResponseChunk.tools` --- ## Frequently Asked Questions ### How does the LLM know when to call a tool versus responding normally? The system prompt explicitly lists available tools with their descriptions and schemas, plus instructions that the model "can use tools if needed." The `tool_choice` setting (`"auto"`, `"none"`, or a specific name) further guides this decision. According to the [`tool_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/tool_prompt.py) templates, voice mode adds guidance to first speak naturally, then optionally include a code block. ### What happens if the LLM generates an invalid tool call? The `FunctionToolCall.to_realtime_function_tool_call` method validates against the registered tool schema. If the tool doesn't exist, required arguments are missing, or undeclared parameters are present, it raises `ValueError` and logs a warning. The pipeline discards the invalid call and continues processing any valid text or subsequent calls. ### Can I force the model to always use a specific tool? Yes. Set `runtime_cfg.session.tool_choice = "tool_name"` instead of `"auto"`. Per the implementation in [`language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/language_model.py), this constrains the model to only that tool, though it may still choose not to call it depending on the conversation context. ### Is streaming interrupted when a tool call is detected? No—tool calls are extracted seamlessly during streaming. The `BaseLanguageModelHandler._process_printable_text` method buffers text until it detects the `ENTER_CODE` delimiter, yields preceding sentences as normal speech chunks, then parses and validates the code block without breaking the stream flow.