# How to Use Tool Calls With the OpenAI Realtime Protocol: A Complete Implementation Guide

> Master tool calls with the OpenAI Realtime protocol. Learn the three-layer pipeline for function calls using FunctionTool, build system prompts, and parse output for seamless integration.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-06

---

**Tool calls in the OpenAI Realtime protocol are handled through a three-layer pipeline—tool definition via `FunctionTool`, system prompt construction with `build_tool_system_prompt()`, and output parsing through `extract_function_calls_from_text()`—that converts LLM-generated function calls into `ResponseFunctionCallArgumentsDoneEvent` events.**

The speech-to-speech repository implements full OpenAI Realtime tool (function) call support for streaming voice assistants. This guide walks through the complete architecture using actual source paths and runnable code from the huggingface/speech-to-speech codebase.

## Architecture Overview: The Three Layers of Realtime Tool Calls

The implementation divides tool call handling into tightly-coupled layers:

- **Tool definition** — `FunctionTool` class declares tools with JSON schemas
- **Prompt construction** — `build_tool_system_prompt()` instructs the model when and how to emit calls
- **Output parsing and conversion** — `extract_function_calls_from_text()` validates and converts to Realtime events

```

┌─────────────────┐     ┌──────────────────┐     ┌─────────────────────┐
│  FunctionTool   │────▶│ build_tool_system│────▶│ extract_function_   │
│  (definition)   │     │ _prompt()        │     │ calls_from_text()   │
└─────────────────┘     └──────────────────┘     └─────────────────────┘
                                                         │
                              ┌────────────────────────┘
                              ▼
                    ┌─────────────────┐
                    │ ResponseFunction│
                    │ CallArguments   │
                    │ DoneEvent       │
                    └─────────────────┘

```

## Layer 1: Defining Tools With FunctionTool

Tools are declared as instances of `FunctionTool`, a subclass of `openai.types.realtime.RealtimeFunctionTool` located in [`src/speech_to_speech/LLM/tool_call/function_tool.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/tool_call/function_tool.py).

The class adds `to_code_prompt()` method that builds Python-style signatures from JSON schemas via `signature_from_schema()`:

```python
from speech_to_speech.LLM.tool_call.function_tool import FunctionTool

weather_tool = FunctionTool()
weather_tool.name = "get_weather"
weather_tool.description = "Retrieve current weather for a location."
weather_tool.type = "function"
weather_tool.parameters = {
    "type": "object",
    "properties": {
        "location": {
            "type": "string",
            "description": "City and country, e.g. 'Paris, France'"
        },
        "unit": {
            "type": "string",
            "enum": ["celsius", "fahrenheit"],
            "default": "celsius"
        }
    },
    "required": ["location"]
}

```

Key methods on `FunctionTool`:

- **`to_code_prompt()`** — Returns Python-style function signature for the system prompt
- **`signature_from_schema()`** — Utility in [`signature_from_schema.py`](https://github.com/huggingface/speech-to-speech/blob/main/signature_from_schema.py) that converts JSON Schema to Python parameters

## Layer 2: Building the Tool System Prompt

The `build_tool_system_prompt()` function in [`src/speech_to_speech/LLM/tool_call/tool_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/tool_call/tool_prompt.py) constructs the critical instruction that teaches the LLM to emit tool calls:

```python
from speech_to_speech.LLM.tool_call.tool_prompt import build_tool_system_prompt

tools = [weather_tool]
system_prompt = build_tool_system_prompt(tools)

print(system_prompt)

# Output includes:

# - Instruction to use brief natural utterance before tool call

# - Python-style signatures for all registered tools

# - <code> block formatting requirement

```

The prompt instructs the model to:

1. Use a **brief natural sentence** before the tool call (e.g., "Let me check that for you")
2. Wrap the actual call in `<code>` tags with valid Python syntax
3. Include only the function call, no explanations inside the code block

## Layer 3: Parsing Model Output and Converting to Events

When the LLM generates a response containing a tool call, `extract_function_calls_from_text()` in [`src/speech_to_speech/LLM/tool_call/function_call.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/tool_call/function_call.py) handles extraction:

### Step 3a: Extract Function Calls From Text

```python
from speech_to_speech.LLM.tool_call.function_call import (
    extract_function_calls_from_text
)

model_output = """
I'll check the weather for you.

<code>get_weather(location="London, UK", unit="celsius")</code>
"""

plain_text, calls = extract_function_calls_from_text(model_output)

print(plain_text)  # "I'll check the weather for you."

print(len(calls))  # 1

```

The function returns:

- **Plain text** — Content outside `<code>` blocks (spoken to user)
- **`FunctionToolCall` objects** — Structured representations of each call

### Step 3b: Validate and Convert to Realtime Event

Each `FunctionToolCall` validates against its tool's schema and converts to a `ResponseFunctionCallArgumentsDoneEvent`:

```python
from speech_to_speech.LLM.tool_call.function_call import FunctionToolCall

tool_call: FunctionToolCall = calls[0]

# Validate against declared tools and create Realtime event

realtime_event = tool_call.to_realtime_function_tool_call(
    function_tools=[weather_tool]
)

# realtime_event is ResponseFunctionCallArgumentsDoneEvent

# Contains: call_id, name, arguments (JSON string)

```

The `to_realtime_function_tool_call()` method (lines 91-136 in [`function_call.py`](https://github.com/huggingface/speech-to-speech/blob/main/function_call.py)):

- Matches the call name against available `FunctionTool` instances
- Validates arguments against the tool's JSON schema
- Drops unknown arguments
- Ensures required parameters are present
- Returns properly formatted `ResponseFunctionCallArgumentsDoneEvent`

## Integration: Emitting Events Through ResponseHandler

The `ResponseHandler` class in [`src/speech_to_speech/api/openai_realtime/handlers/response.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/handlers/response.py) orchestrates the final event emission:

```python

# From src/speech_to_speech/api/openai_realtime/handlers/response.py

# Simplified excerpt showing the dispatch flow

def handle_function_call(
    self,
    conn_id: str,
    tool_call: FunctionToolCall
) -> list[ServerEvent]:
    """
    Convert parsed tool call to Realtime event and queue for client.
    """
    # Convert to OpenAI Realtime protocol event

    event = tool_call.to_realtime_function_tool_call(
        self.available_tools
    )
    
    # Return as server-side event to be streamed via WebSocket

    return [event]

```

The `RealtimeService` in [`src/speech_to_speech/api/openai_realtime/service.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/service.py) manages the full lifecycle:

- Tracks usage and token accounting
- Handles speculative turn processing
- Routes events through `_dispatch_pipeline_event()`
- Maintains WebSocket connection state

## Complete End-to-End Example

Here's a runnable pattern integrating all three layers:

```python
from speech_to_speech.LLM.tool_call.function_tool import FunctionTool
from speech_to_speech.LLM.tool_call.tool_prompt import build_tool_system_prompt
from speech_to_speech.LLM.tool_call.function_call import (
    extract_function_calls_from_text,
    FunctionToolCall
)

# 1. Define tools

def create_timer_tool() -> FunctionTool:
    tool = FunctionTool()
    tool.name = "set_timer"
    tool.description = "Set a countdown timer."
    tool.type = "function"
    tool.parameters = {
        "type": "object",
        "properties": {
            "seconds": {
                "type": "integer",
                "description": "Duration in seconds",
                "minimum": 1,
                "maximum": 3600
            },
            "label": {
                "type": "string",
                "description": "Optional label for the timer"
            }
        },
        "required": ["seconds"]
    }
    return tool

# 2. Build system prompt

timer_tool = create_timer_tool()
system_prompt = build_tool_system_prompt([timer_tool])

# 3. Simulate LLM response with tool call

simulated_response = """
Okay, I'll start a five-minute timer.

<code>set_timer(seconds=300, label="Pasta cooking")</code>
"""

# 4. Parse and convert

plain_text, calls = extract_function_calls_from_text(simulated_response)

for call in calls:
    print(f"Function: {call.name}")
    print(f"Arguments: {call.arguments}")
    
    # 5. Convert to Realtime event

    event = call.to_realtime_function_tool_call([timer_tool])
    print(f"Event type: {type(event).__name__}")
    print(f"Call ID: {event.call_id}")

```

## Key Source Files and Their Roles

| File Path | Purpose |
|-----------|---------|
| [`src/speech_to_speech/LLM/tool_call/function_tool.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/tool_call/function_tool.py) | `FunctionTool` class definition and `to_code_prompt()` method |
| [`src/speech_to_speech/LLM/tool_call/signature_from_schema.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/tool_call/signature_from_schema.py) | JSON Schema to Python signature conversion |
| [`src/speech_to_speech/LLM/tool_call/tool_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/tool_call/tool_prompt.py) | `build_tool_system_prompt()` implementation |
| [`src/speech_to_speech/LLM/tool_call/function_call.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/tool_call/function_call.py) | Parsing (`extract_function_calls_from_text`), validation, and `to_realtime_function_tool_call()` |
| [`src/speech_to_speech/api/openai_realtime/service.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/service.py) | `RealtimeService` orchestrating protocol translation |
| [`src/speech_to_speech/api/openai_realtime/handlers/response.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/handlers/response.py) | `ResponseHandler` emitting `ResponseFunctionCallArgumentsDoneEvent` |

## Summary

- **`FunctionTool`** in [`function_tool.py`](https://github.com/huggingface/speech-to-speech/blob/main/function_tool.py) wraps tool definitions with schema-to-signature conversion
- **`build_tool_system_prompt()`** in [`tool_prompt.py`](https://github.com/huggingface/speech-to-speech/blob/main/tool_prompt.py) constructs LLM instructions for proper tool call formatting
- **`extract_function_calls_from_text()`** in [`function_call.py`](https://github.com/huggingface/speech-to-speech/blob/main/function_call.py) separates spoken text from executable calls
- **`FunctionToolCall.to_realtime_function_tool_call()`** validates arguments and produces protocol-compliant events
- **`ResponseHandler`** in [`handlers/response.py`](https://github.com/huggingface/speech-to-speech/blob/main/handlers/response.py) streams `ResponseFunctionCallArgumentsDoneEvent` to clients via WebSocket

## Frequently Asked Questions

### How does the OpenAI Realtime protocol represent tool call arguments?

Arguments are transmitted as a JSON string in the `arguments` field of `ResponseFunctionCallArgumentsDoneEvent`. The `to_realtime_function_tool_call()` method serializes validated Python arguments to this JSON format, matching OpenAI's specification exactly.

### What happens if the LLM generates an invalid tool call?

The `to_realtime_function_tool_call()` method validates arguments against the tool's JSON schema. Unknown arguments are dropped, missing required parameters raise validation errors, and type mismatches are caught before event creation—preventing malformed events from reaching the client.

### Can I use multiple tools in a single LLM response?

Yes. The `extract_function_calls_from_text()` function returns a **list** of `FunctionToolCall` objects. Multiple `<code>` blocks in one response are parsed independently, and each produces its own `ResponseFunctionCallArgumentsDoneEvent` for sequential client handling.

### Where is the tool call event actually sent to the client?

The `ResponseHandler.handle_function_call()` method in [`handlers/response.py`](https://github.com/huggingface/speech-to-speech/blob/main/handlers/response.py) returns the event list, which `RealtimeService._dispatch_pipeline_event()` queues for WebSocket transmission. The client receives it as a standard OpenAI Realtime server event through the established WebSocket connection.