# Implementing Tool Calls and Function Calling in Voice Conversations with Speech-to-Speech

> Learn how to implement tool calls and function calling in voice conversations using the Speech-to-Speech pipeline. Stream function invocations for structured tool execution.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-09

---

**The Speech-to-Speech pipeline supports OpenAI-compatible tool calls by streaming function invocations through a four-stage concurrent architecture that converts spoken input into structured tool executions.**

This article explores how the `huggingface/speech-to-speech` repository implements real-time function calling in voice conversations. The system processes audio through modular handlers that run in separate threads, enabling low-latency tool execution while maintaining natural conversational flow.

## Architecture Overview

The pipeline consists of four concurrent stages connected by asynchronous queues. Each stage operates in its own thread and can be swapped via command-line arguments.

| Stage | Default Backend | Description |
|------|----------------|-------------|
| **Voice Activity Detection (VAD)** | [Silero VAD v5](https://github.com/snakers4/silero-vad) | Detects speech boundaries and generates turn-detection events. Implemented in [`src/speech_to_speech/handlers/vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/handlers/vad_handler.py). |
| **Speech-to-Text (STT)** | [Parakeet TDT](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3) | Streams transcriptions of the user's turn. The STT interface lives in [`src/speech_to_speech/handlers/stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/handlers/stt_handler.py). |
| **Language Model (LLM)** | OpenAI-compatible Responses-API or Chat-Completions | Generates text and emits tool-call objects. Core logic is in [`src/speech_to_speech/LLM/language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/language_model.py) and the streaming helper [`src/speech_to_speech/LLM/base_openai_compatible_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/base_openai_compatible_language_model.py). |
| **Text-to-Speech (TTS)** | [Qwen3-TTS](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice) | Synthesizes audio for the LLM reply. The TTS handler resides in [`src/speech_to_speech/handlers/tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/handlers/tts_handler.py). |

## Tool-Call Flow

When implementing tool calls and function calling in voice conversations, the pipeline follows a strict three-step validation and conversion process:

1. **LLM emits a function call** – When the LLM generates a tool call, it creates a `ResponseFunctionToolCall` object defined in [`src/speech_to_speech/pipeline/events.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/events.py).

2. **Conversion to Realtime format** – `FunctionToolCall.to_realtime_function_tool_call` (implemented in [`src/speech_to_speech/LLM/tool_call/function_call.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/tool_call/function_call.py)) strips positional arguments that are not required by the declared schema and validates required fields. The underlying schema handling lives in [`src/speech_to_speech/LLM/tool_call/function_tool.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/tool_call/function_tool.py) and [`src/speech_to_speech/LLM/tool_call/signature_from_schema.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/tool_call/signature_from_schema.py).

3. **Dispatch to the client** – The converted `TransformersToolCall` is attached to the outbound Realtime event (`tool_calls` field) and streamed to the client over the OpenAI Realtime WebSocket. The server's Realtime engine is described in [`src/speech_to_speech/api/openai_realtime/README.md`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/README.md).

The test suite (`tests/tool_call/`) verifies parsing of arbitrary Python-style calls in [`tests/tool_call/test_function_parser.py`](https://github.com/huggingface/speech-to-speech/blob/main/tests/tool_call/test_function_parser.py) and proper argument stripping in [`tests/tool_call/test_signature_from_schema.py`](https://github.com/huggingface/speech-to-speech/blob/main/tests/tool_call/test_signature_from_schema.py).

## Implementation Guide

### Running the Pipeline Locally

Start the complete server and client stack in a single process using the built-in local mode:

```bash
speech-to-speech local

```

This command initializes the VAD, STT, LLM, and TTS handlers concurrently, equivalent to running `serve` and `talk` components together.

### Connecting via Realtime WebSocket

Connect a custom client using the OpenAI SDK to handle tool calls in real time:

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8765/v1",
    websocket_base_url="ws://localhost:8765/v1",
    api_key="not-needed",
)

with client.realtime.connect(model="local") as conn:
    # Update session with VAD turn detection

    conn.send({
        "type": "session.update",
        "session": {
            "type": "realtime",
            "instructions": "You are a helpful assistant.",
            "audio": {"input": {"turn_detection": {"type": "server_vad", "interrupt_response": True}}}
        }
    })
    # Stream events (transcriptions, tool calls, audio deltas, …)

    for event in conn:
        print(event.type, event)

```

The client receives `tool_calls` events when the LLM invokes registered functions, allowing your application to execute logic and return results via the same WebSocket connection.

### Defining Custom Function Tools

Register new capabilities by extending `FunctionTool` with a JSON schema describing parameters. The LLM automatically enforces the schema during streaming:

```python
from speech_to_speech.LLM.tool_call.function_tool import FunctionTool

# Declare a tool that greets a user

greet_tool = FunctionTool(
    type="function",
    name="greet",
    parameters={"msg": {"type": "string"}},
    required=["msg"],
)

# Pass the tool to the language-model configuration

llm = LanguageModel(
    backend="responses-api",
    model_name="gpt-4o-mini",
    function_tools=[greet_tool],
)

```

When the LLM generates `greet(msg="hello")`, the pipeline strips any stray positional arguments, validates the required `msg` field, and sends a Realtime `tool_calls` event to the client.

### Configuring Alternative Backends

Swap components without code changes using CLI flags mapped to handler classes in the `arguments_classes` package:

```bash
speech-to-speech serve \
    --stt whisper-mlx \
    --stt_model_name large-v3 \
    --llm_backend responses-api \
    --tts qwen3

```

The [`src/speech_to_speech/arguments_classes/whisper_stt_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/whisper_stt_arguments.py) file contains the specific CLI options for the Whisper-MLX backend, while platform-specific builds (CUDA 12, CPU, or Apple Silicon) are handled via optional dependencies in [`pyproject.toml`](https://github.com/huggingface/speech-to-speech/blob/main/pyproject.toml).

## Key Implementation Files

Understanding these source files is essential for extending tool-call functionality:

- **[`src/speech_to_speech/pipeline/messages.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/messages.py)** – Defines the internal message representation exchanged between pipeline stages.
- **[`src/speech_to_speech/pipeline/events.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/events.py)** – Implements the event loop that forwards VAD, STT, LLM, and TTS events, including `ResponseFunctionToolCall` definitions.
- **[`src/speech_to_speech/LLM/language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/language_model.py)** – Core LLM wrapper that streams text and tool calls according to the OpenAI Responses API format.
- **[`src/speech_to_speech/LLM/tool_call/function_call.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/tool_call/function_call.py)** – Parses raw function-call strings and converts them to Realtime objects via `to_realtime_function_tool_call`.
- **[`src/speech_to_speech/LLM/tool_call/function_tool.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/tool_call/function_tool.py)** – Holds the JSON schema for a tool and provides validation utilities.
- **[`src/speech_to_speech/api/openai_realtime/README.md`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/README.md)** – Detailed description of the Realtime protocol implementation and supported event types.
- **`src/speech_to_speech/arguments_classes/*.py`** – CLI argument groups for each component (VAD, STT, LLM, TTS).

## Summary

- The **Speech-to-Speech pipeline** uses a four-stage concurrent architecture (VAD → STT → LLM → TTS) to process voice input with minimal latency.
- **Tool calls** flow from `ResponseFunctionToolCall` objects through [`function_call.py`](https://github.com/huggingface/speech-to-speech/blob/main/function_call.py) conversion logic before dispatch via the OpenAI Realtime WebSocket.
- **Custom tools** are implemented by extending `FunctionTool` with JSON schemas, automatically enforced by the LLM during streaming generation.
- **Backend flexibility** allows swapping VAD, STT, LLM, or TTS components via CLI flags defined in the `arguments_classes` package.
- **Platform-specific optimizations** are handled through conditional dependencies in [`pyproject.toml`](https://github.com/huggingface/speech-to-speech/blob/main/pyproject.toml) for CUDA, CPU, and Apple Silicon deployments.

## Frequently Asked Questions

### How does the pipeline handle interruptions during tool execution?

The VAD handler in [`src/speech_to_speech/handlers/vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/handlers/vad_handler.py) emits turn-detection events that can interrupt ongoing responses when `interrupt_response` is enabled in the session configuration. This allows users to abort tool-call sequences mid-execution by speaking again, with the LLM handler discarding pending tokens and resetting the generation state.

### Can I use local models instead of OpenAI's API for tool calling?

Yes. The `LanguageModel` class in [`src/speech_to_speech/LLM/language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/language_model.py) supports local backends including Transformers and `mlx-lm` through the `--llm_backend` CLI flag. When using local models, the same `FunctionTool` schema validation applies, though you must ensure the local model supports function-calling syntax.

### What validation occurs when the LLM generates a tool call?

The `FunctionToolCall.to_realtime_function_tool_call` method validates that all required parameters specified in [`function_tool.py`](https://github.com/huggingface/speech-to-speech/blob/main/function_tool.py) are present, strips any positional arguments not defined in the schema, and ensures type compliance before converting to the Realtime API format. Failed validations are caught in [`test_signature_from_schema.py`](https://github.com/huggingface/speech-to-speech/blob/main/test_signature_from_schema.py) patterns.

### How do I add support for a new speech-to-text backend?

Create a new handler class inheriting from the STT interface in [`src/speech_to_speech/handlers/stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/handlers/stt_handler.py), then add corresponding CLI arguments in a new file under `src/speech_to_speech/arguments_classes/`. Register the backend in the main entry point to enable selection via the `--stt` flag, following the pattern established in [`whisper_stt_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/whisper_stt_arguments.py).