How to Use Tool Calls With the OpenAI Realtime Protocol: A Complete Implementation Guide

Tool calls in the OpenAI Realtime protocol are handled through a three-layer pipeline—tool definition via FunctionTool, system prompt construction with build_tool_system_prompt(), and output parsing through extract_function_calls_from_text()—that converts LLM-generated function calls into ResponseFunctionCallArgumentsDoneEvent events.

The speech-to-speech repository implements full OpenAI Realtime tool (function) call support for streaming voice assistants. This guide walks through the complete architecture using actual source paths and runnable code from the huggingface/speech-to-speech codebase.

Architecture Overview: The Three Layers of Realtime Tool Calls

The implementation divides tool call handling into tightly-coupled layers:

• Tool definition — FunctionTool class declares tools with JSON schemas
• Prompt construction — build_tool_system_prompt() instructs the model when and how to emit calls
• Output parsing and conversion — extract_function_calls_from_text() validates and converts to Realtime events

┌─────────────────┐     ┌──────────────────┐     ┌─────────────────────┐
│  FunctionTool   │────▶│ build_tool_system│────▶│ extract_function_   │
│  (definition)   │     │ _prompt()        │     │ calls_from_text()   │
└─────────────────┘     └──────────────────┘     └─────────────────────┘
                                                         │
                              ┌────────────────────────┘
                              ▼
                    ┌─────────────────┐
                    │ ResponseFunction│
                    │ CallArguments   │
                    │ DoneEvent       │
                    └─────────────────┘

Layer 1: Defining Tools With FunctionTool

Tools are declared as instances of FunctionTool, a subclass of openai.types.realtime.RealtimeFunctionTool located in src/speech_to_speech/LLM/tool_call/function_tool.py.

The class adds to_code_prompt() method that builds Python-style signatures from JSON schemas via signature_from_schema():

from speech_to_speech.LLM.tool_call.function_tool import FunctionTool

weather_tool = FunctionTool()
weather_tool.name = "get_weather"
weather_tool.description = "Retrieve current weather for a location."
weather_tool.type = "function"
weather_tool.parameters = {
    "type": "object",
    "properties": {
        "location": {
            "type": "string",
            "description": "City and country, e.g. 'Paris, France'"
        },
        "unit": {
            "type": "string",
            "enum": ["celsius", "fahrenheit"],
            "default": "celsius"
        }
    },
    "required": ["location"]
}

Key methods on FunctionTool:

• to_code_prompt() — Returns Python-style function signature for the system prompt
• signature_from_schema() — Utility in signature_from_schema.py that converts JSON Schema to Python parameters

Layer 2: Building the Tool System Prompt

The build_tool_system_prompt() function in src/speech_to_speech/LLM/tool_call/tool_prompt.py constructs the critical instruction that teaches the LLM to emit tool calls:

from speech_to_speech.LLM.tool_call.tool_prompt import build_tool_system_prompt

tools = [weather_tool]
system_prompt = build_tool_system_prompt(tools)

print(system_prompt)

# Output includes:

# - Instruction to use brief natural utterance before tool call

# - Python-style signatures for all registered tools

# - <code> block formatting requirement

The prompt instructs the model to:

1. Use a brief natural sentence before the tool call (e.g., "Let me check that for you")
2. Wrap the actual call in <code> tags with valid Python syntax
3. Include only the function call, no explanations inside the code block

Layer 3: Parsing Model Output and Converting to Events

When the LLM generates a response containing a tool call, extract_function_calls_from_text() in src/speech_to_speech/LLM/tool_call/function_call.py handles extraction:

Step 3a: Extract Function Calls From Text

from speech_to_speech.LLM.tool_call.function_call import (
    extract_function_calls_from_text
)

model_output = """
I'll check the weather for you.

<code>get_weather(location="London, UK", unit="celsius")</code>
"""

plain_text, calls = extract_function_calls_from_text(model_output)

print(plain_text)  # "I'll check the weather for you."

print(len(calls))  # 1

The function returns:

• Plain text — Content outside <code> blocks (spoken to user)
• FunctionToolCall objects — Structured representations of each call

Step 3b: Validate and Convert to Realtime Event

Each FunctionToolCall validates against its tool's schema and converts to a ResponseFunctionCallArgumentsDoneEvent:

from speech_to_speech.LLM.tool_call.function_call import FunctionToolCall

tool_call: FunctionToolCall = calls[0]

# Validate against declared tools and create Realtime event

realtime_event = tool_call.to_realtime_function_tool_call(
    function_tools=[weather_tool]
)

# realtime_event is ResponseFunctionCallArgumentsDoneEvent

# Contains: call_id, name, arguments (JSON string)

The to_realtime_function_tool_call() method (lines 91-136 in function_call.py):

• Matches the call name against available FunctionTool instances
• Validates arguments against the tool's JSON schema
• Drops unknown arguments
• Ensures required parameters are present
• Returns properly formatted ResponseFunctionCallArgumentsDoneEvent

Integration: Emitting Events Through ResponseHandler

The ResponseHandler class in src/speech_to_speech/api/openai_realtime/handlers/response.py orchestrates the final event emission:

# From src/speech_to_speech/api/openai_realtime/handlers/response.py

# Simplified excerpt showing the dispatch flow

def handle_function_call(
    self,
    conn_id: str,
    tool_call: FunctionToolCall
) -> list[ServerEvent]:
    """
    Convert parsed tool call to Realtime event and queue for client.
    """
    # Convert to OpenAI Realtime protocol event

    event = tool_call.to_realtime_function_tool_call(
        self.available_tools
    )
    
    # Return as server-side event to be streamed via WebSocket

    return [event]

The RealtimeService in src/speech_to_speech/api/openai_realtime/service.py manages the full lifecycle:

• Tracks usage and token accounting
• Handles speculative turn processing
• Routes events through _dispatch_pipeline_event()
• Maintains WebSocket connection state

Complete End-to-End Example

Here's a runnable pattern integrating all three layers:

from speech_to_speech.LLM.tool_call.function_tool import FunctionTool
from speech_to_speech.LLM.tool_call.tool_prompt import build_tool_system_prompt
from speech_to_speech.LLM.tool_call.function_call import (
    extract_function_calls_from_text,
    FunctionToolCall
)

# 1. Define tools

def create_timer_tool() -> FunctionTool:
    tool = FunctionTool()
    tool.name = "set_timer"
    tool.description = "Set a countdown timer."
    tool.type = "function"
    tool.parameters = {
        "type": "object",
        "properties": {
            "seconds": {
                "type": "integer",
                "description": "Duration in seconds",
                "minimum": 1,
                "maximum": 3600
            },
            "label": {
                "type": "string",
                "description": "Optional label for the timer"
            }
        },
        "required": ["seconds"]
    }
    return tool

# 2. Build system prompt

timer_tool = create_timer_tool()
system_prompt = build_tool_system_prompt([timer_tool])

# 3. Simulate LLM response with tool call

simulated_response = """
Okay, I'll start a five-minute timer.

<code>set_timer(seconds=300, label="Pasta cooking")</code>
"""

# 4. Parse and convert

plain_text, calls = extract_function_calls_from_text(simulated_response)

for call in calls:
    print(f"Function: {call.name}")
    print(f"Arguments: {call.arguments}")
    
    # 5. Convert to Realtime event

    event = call.to_realtime_function_tool_call([timer_tool])
    print(f"Event type: {type(event).__name__}")
    print(f"Call ID: {event.call_id}")

Key Source Files and Their Roles

File Path	Purpose
src/speech_to_speech/LLM/tool_call/function_tool.py	FunctionTool class definition and to_code_prompt() method
src/speech_to_speech/LLM/tool_call/signature_from_schema.py	JSON Schema to Python signature conversion
src/speech_to_speech/LLM/tool_call/tool_prompt.py	build_tool_system_prompt() implementation
src/speech_to_speech/LLM/tool_call/function_call.py	Parsing (extract_function_calls_from_text), validation, and to_realtime_function_tool_call()
src/speech_to_speech/api/openai_realtime/service.py	RealtimeService orchestrating protocol translation
src/speech_to_speech/api/openai_realtime/handlers/response.py	ResponseHandler emitting ResponseFunctionCallArgumentsDoneEvent

Summary

• FunctionTool in function_tool.py wraps tool definitions with schema-to-signature conversion
• build_tool_system_prompt() in tool_prompt.py constructs LLM instructions for proper tool call formatting
• extract_function_calls_from_text() in function_call.py separates spoken text from executable calls
• FunctionToolCall.to_realtime_function_tool_call() validates arguments and produces protocol-compliant events
• ResponseHandler in handlers/response.py streams ResponseFunctionCallArgumentsDoneEvent to clients via WebSocket

Frequently Asked Questions

How does the OpenAI Realtime protocol represent tool call arguments?

Arguments are transmitted as a JSON string in the arguments field of ResponseFunctionCallArgumentsDoneEvent. The to_realtime_function_tool_call() method serializes validated Python arguments to this JSON format, matching OpenAI's specification exactly.

What happens if the LLM generates an invalid tool call?

The to_realtime_function_tool_call() method validates arguments against the tool's JSON schema. Unknown arguments are dropped, missing required parameters raise validation errors, and type mismatches are caught before event creation—preventing malformed events from reaching the client.

Can I use multiple tools in a single LLM response?

Yes. The extract_function_calls_from_text() function returns a list of FunctionToolCall objects. Multiple <code> blocks in one response are parsed independently, and each produces its own ResponseFunctionCallArgumentsDoneEvent for sequential client handling.

Where is the tool call event actually sent to the client?

The ResponseHandler.handle_function_call() method in handlers/response.py returns the event list, which RealtimeService._dispatch_pipeline_event() queues for WebSocket transmission. The client receives it as a standard OpenAI Realtime server event through the established WebSocket connection.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}