Implementing Tool Calls and Function Calling in Voice Conversations with Speech-to-Speech

The Speech-to-Speech pipeline supports OpenAI-compatible tool calls by streaming function invocations through a four-stage concurrent architecture that converts spoken input into structured tool executions.

This article explores how the huggingface/speech-to-speech repository implements real-time function calling in voice conversations. The system processes audio through modular handlers that run in separate threads, enabling low-latency tool execution while maintaining natural conversational flow.

Architecture Overview

The pipeline consists of four concurrent stages connected by asynchronous queues. Each stage operates in its own thread and can be swapped via command-line arguments.

Stage	Default Backend	Description
Voice Activity Detection (VAD)	Silero VAD v5	Detects speech boundaries and generates turn-detection events. Implemented in src/speech_to_speech/handlers/vad_handler.py.
Speech-to-Text (STT)	Parakeet TDT	Streams transcriptions of the user's turn. The STT interface lives in src/speech_to_speech/handlers/stt_handler.py.
Language Model (LLM)	OpenAI-compatible Responses-API or Chat-Completions	Generates text and emits tool-call objects. Core logic is in src/speech_to_speech/LLM/language_model.py and the streaming helper src/speech_to_speech/LLM/base_openai_compatible_language_model.py.
Text-to-Speech (TTS)	Qwen3-TTS	Synthesizes audio for the LLM reply. The TTS handler resides in src/speech_to_speech/handlers/tts_handler.py.

Tool-Call Flow

When implementing tool calls and function calling in voice conversations, the pipeline follows a strict three-step validation and conversion process:

1. LLM emits a function call – When the LLM generates a tool call, it creates a ResponseFunctionToolCall object defined in src/speech_to_speech/pipeline/events.py.

2. Conversion to Realtime format – FunctionToolCall.to_realtime_function_tool_call (implemented in src/speech_to_speech/LLM/tool_call/function_call.py) strips positional arguments that are not required by the declared schema and validates required fields. The underlying schema handling lives in src/speech_to_speech/LLM/tool_call/function_tool.py and src/speech_to_speech/LLM/tool_call/signature_from_schema.py.

3. Dispatch to the client – The converted TransformersToolCall is attached to the outbound Realtime event (tool_calls field) and streamed to the client over the OpenAI Realtime WebSocket. The server's Realtime engine is described in src/speech_to_speech/api/openai_realtime/README.md.

The test suite (tests/tool_call/) verifies parsing of arbitrary Python-style calls in tests/tool_call/test_function_parser.py and proper argument stripping in tests/tool_call/test_signature_from_schema.py.

Implementation Guide

Running the Pipeline Locally

Start the complete server and client stack in a single process using the built-in local mode:

speech-to-speech local

This command initializes the VAD, STT, LLM, and TTS handlers concurrently, equivalent to running serve and talk components together.

Connecting via Realtime WebSocket

Connect a custom client using the OpenAI SDK to handle tool calls in real time:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8765/v1",
    websocket_base_url="ws://localhost:8765/v1",
    api_key="not-needed",
)

with client.realtime.connect(model="local") as conn:
    # Update session with VAD turn detection

    conn.send({
        "type": "session.update",
        "session": {
            "type": "realtime",
            "instructions": "You are a helpful assistant.",
            "audio": {"input": {"turn_detection": {"type": "server_vad", "interrupt_response": True}}}
        }
    })
    # Stream events (transcriptions, tool calls, audio deltas, …)

    for event in conn:
        print(event.type, event)

The client receives tool_calls events when the LLM invokes registered functions, allowing your application to execute logic and return results via the same WebSocket connection.

Defining Custom Function Tools

Register new capabilities by extending FunctionTool with a JSON schema describing parameters. The LLM automatically enforces the schema during streaming:

from speech_to_speech.LLM.tool_call.function_tool import FunctionTool

# Declare a tool that greets a user

greet_tool = FunctionTool(
    type="function",
    name="greet",
    parameters={"msg": {"type": "string"}},
    required=["msg"],
)

# Pass the tool to the language-model configuration

llm = LanguageModel(
    backend="responses-api",
    model_name="gpt-4o-mini",
    function_tools=[greet_tool],
)

When the LLM generates greet(msg="hello"), the pipeline strips any stray positional arguments, validates the required msg field, and sends a Realtime tool_calls event to the client.

Configuring Alternative Backends

Swap components without code changes using CLI flags mapped to handler classes in the arguments_classes package:

speech-to-speech serve \
    --stt whisper-mlx \
    --stt_model_name large-v3 \
    --llm_backend responses-api \
    --tts qwen3

The src/speech_to_speech/arguments_classes/whisper_stt_arguments.py file contains the specific CLI options for the Whisper-MLX backend, while platform-specific builds (CUDA 12, CPU, or Apple Silicon) are handled via optional dependencies in pyproject.toml.

Key Implementation Files

Understanding these source files is essential for extending tool-call functionality:

• src/speech_to_speech/pipeline/messages.py – Defines the internal message representation exchanged between pipeline stages.
• src/speech_to_speech/pipeline/events.py – Implements the event loop that forwards VAD, STT, LLM, and TTS events, including ResponseFunctionToolCall definitions.
• src/speech_to_speech/LLM/language_model.py – Core LLM wrapper that streams text and tool calls according to the OpenAI Responses API format.
• src/speech_to_speech/LLM/tool_call/function_call.py – Parses raw function-call strings and converts them to Realtime objects via to_realtime_function_tool_call.
• src/speech_to_speech/LLM/tool_call/function_tool.py – Holds the JSON schema for a tool and provides validation utilities.
• src/speech_to_speech/api/openai_realtime/README.md – Detailed description of the Realtime protocol implementation and supported event types.
• src/speech_to_speech/arguments_classes/*.py – CLI argument groups for each component (VAD, STT, LLM, TTS).

Summary

• The Speech-to-Speech pipeline uses a four-stage concurrent architecture (VAD → STT → LLM → TTS) to process voice input with minimal latency.
• Tool calls flow from ResponseFunctionToolCall objects through function_call.py conversion logic before dispatch via the OpenAI Realtime WebSocket.
• Custom tools are implemented by extending FunctionTool with JSON schemas, automatically enforced by the LLM during streaming generation.
• Backend flexibility allows swapping VAD, STT, LLM, or TTS components via CLI flags defined in the arguments_classes package.
• Platform-specific optimizations are handled through conditional dependencies in pyproject.toml for CUDA, CPU, and Apple Silicon deployments.

Frequently Asked Questions

How does the pipeline handle interruptions during tool execution?

The VAD handler in src/speech_to_speech/handlers/vad_handler.py emits turn-detection events that can interrupt ongoing responses when interrupt_response is enabled in the session configuration. This allows users to abort tool-call sequences mid-execution by speaking again, with the LLM handler discarding pending tokens and resetting the generation state.

Can I use local models instead of OpenAI's API for tool calling?

Yes. The LanguageModel class in src/speech_to_speech/LLM/language_model.py supports local backends including Transformers and mlx-lm through the --llm_backend CLI flag. When using local models, the same FunctionTool schema validation applies, though you must ensure the local model supports function-calling syntax.

What validation occurs when the LLM generates a tool call?

The FunctionToolCall.to_realtime_function_tool_call method validates that all required parameters specified in function_tool.py are present, strips any positional arguments not defined in the schema, and ensures type compliance before converting to the Realtime API format. Failed validations are caught in test_signature_from_schema.py patterns.

How do I add support for a new speech-to-text backend?

Create a new handler class inheriting from the STT interface in src/speech_to_speech/handlers/stt_handler.py, then add corresponding CLI arguments in a new file under src/speech_to_speech/arguments_classes/. Register the backend in the main entry point to enable selection via the --stt flag, following the pattern established in whisper_stt_arguments.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}