Creating Voice Agents with LiveKit and Real-Time Speech Capabilities: A Complete Guide

LiveKit combined with the Gemini Live API enables developers to build low-latency, real-time voice assistants using the LiveKit Agents framework and WebRTC-based media streaming.

The awesome-ai-apps repository provides production-ready reference implementations demonstrating how to create sophisticated voice agents. These examples leverage LiveKit's WebRTC media layer integrated with Google's Gemini Live API to process continuous audio streams with minimal latency.

Architecture Overview

Creating voice agents with LiveKit follows a consistent architectural pattern across all implementations. The system establishes a bidirectional audio pipeline where participants join a LiveKit room, and an AgentSession streams audio to a real-time LLM while returning synthesized responses.

The core components include:

• LiveKit Room: Handles WebRTC connections between clients and agents
• AgentSession: Manages the participant connection and audio routing
• Real-time LLM: Gemini's "flash-live-preview" model performs token-level inference on streaming audio
• Function Tools: Optional capabilities like web search that the LLM can invoke during conversations
• Voice Configuration: Pluggable voices such as "Zephyr" or "Puck" for audio synthesis

Setting Up Your Development Environment

All voice agent implementations in the repository follow a standardized setup process using environment-based configuration.

Clone the repository and navigate to your chosen example:

git clone https://github.com/Arindam200/awesome-ai-apps.git
cd awesome-ai-apps/voice_agents/livekit_gemini_agents

Install dependencies using the recommended package manager:

uv sync

Configure your credentials by copying the example environment file:

cp .env.example .env

Edit the .env file to include your LiveKit credentials (LIVEKIT_URL, LIVEKIT_API_KEY, LIVEKIT_API_SECRET) and Google API key (GOOGLE_API_KEY).

Building a Basic Gemini Voice Agent

The foundation of real-time voice processing resides in voice_agents/livekit_gemini_agents/main.py.

Core Agent Configuration

The agent initialization defines the voice characteristics and session parameters. In main.py, the VOICE constant configures the speech synthesis output:

# From voice_agents/livekit_gemini_agents/main.py

VOICE = "Zephyr"  # Alternative options include "Puck"

Real-Time Model Integration

The integration point for streaming inference uses the REALTIME_MODEL constant to bind Gemini's live capabilities:

# From voice_agents/livekit_gemini_agents/main.py

REALTIME_MODEL = "models/gemini-2.0-flash-live-preview"

Start the agent to begin listening for participants:

python main.py start

For development with auto-reload capabilities, use:

python main.py dev

Adding Function Tools for Web Search

The voice_agents/livekit_web_search_agent/main.py file demonstrates extending voice agents with external capabilities. This implementation integrates Olostep for real-time web search functionality.

The web_search tool definition appears in the agent configuration:

# From voice_agents/livekit_web_search_agent/main.py

def web_search(query: str) -> str:
    """Search the web for up-to-date information"""
    # Implementation details for Olostep integration

When Gemini requires current information, it automatically invokes this function during the conversation flow. Setup requires the additional OLOSTEP_API_KEY in your environment configuration.

Deploy this variant using:

cd ../livekit_web_search_agent
uv sync
cp .env.example .env
python main.py start

Deploying Telephony Agents with SIP

The RSVP agent example in voice_agents/livekit_rsvp_agent/ demonstrates telephony integration for outbound phone calls. This implementation combines multiple AI services: Nebius for LLM inference, Deepgram for Speech-to-Text, and Cartesia for Text-to-Speech.

The Agent Worker

The agent.py file defines the worker process that handles audio processing and conversation state:

cd voice_agents/livekit_rsvp_agent
uv sync
cp .env.example .env  # Include SIP_OUTBOUND_TRUNK_ID, Nebius, Deepgram, and Cartesia keys

Launch the worker in development mode:

python agent.py dev

Dispatching Calls

The dispatch.py script automates room creation and outbound SIP dialing. Located at voice_agents/livekit_rsvp_agent/dispatch.py, this utility places calls to phone numbers and connects them to your voice agent.

Place outbound calls to all pending attendees:

python dispatch.py

For testing single recipients:

python dispatch.py --id A1

The dispatcher creates the LiveKit room, dispatches the agent worker, and initiates the SIP call sequence.

Configuration and Environment Variables

Each implementation requires specific environment variables defined in .env.example files. Never commit actual API keys to version control.

Required variables for basic agents:

• LIVEKIT_URL
• LIVEKIT_API_KEY
• LIVEKIT_API_SECRET
• GOOGLE_API_KEY

Additional variables for web search:

• OLOSTEP_API_KEY

Additional variables for telephony:

• SIP_OUTBOUND_TRUNK_ID
• NEBIUS_API_KEY
• DEEPGRAM_API_KEY
• CARTESIA_API_KEY

Test your deployment using the LiveKit CLI to generate access tokens:

lk token create \
  --room gemini-agent \
  --identity me \
  --agent gemini-voice-agent \
  --join \
  --open console

This command creates a token that registers the agent and opens the LiveKit Agent Console for immediate voice interaction.

Summary

• LiveKit Agents framework provides the WebRTC infrastructure for real-time voice communication between participants and AI models.
• Gemini Live API integration enables token-level streaming inference with synchronized audio input/output through constants like REALTIME_MODEL.
• Function tools extend agent capabilities beyond conversation, allowing web searches and database operations during active calls.
• Telephony deployment requires coordinating agent workers (agent.py) with dispatcher scripts (dispatch.py) for outbound SIP calls.
• Environment isolation via .env files separates credentials from source code, supporting safe deployment across development and production environments.

Frequently Asked Questions

What is the LiveKit Agents framework?

The LiveKit Agents framework is a Python SDK that manages WebRTC media streams between participants and AI services. It handles audio streaming, room management, and connection lifecycle, allowing developers to focus on conversation logic rather than low-level media handling.

How does real-time speech processing work with Gemini Live?

Gemini Live accepts continuous audio streams through a persistent connection, processing speech token-by-token rather than waiting for complete utterances. The model returns synchronized audio responses with minimal latency, enabling natural turn-taking in conversations.

Can I customize the voice used by LiveKit agents?

Yes. Voice agents support configurable speech synthesis parameters. In voice_agents/livekit_gemini_agents/main.py, modify the VOICE constant to select alternatives like "Puck" or "Zephyr". For advanced customization, the RSVP agent demonstrates integration with Cartesia TTS for fine-grained voice control.

How do I deploy a voice agent for production phone calls?

Production telephony deployment requires three components: a running agent worker (python agent.py start), the LiveKit SIP integration configured with your trunk ID, and a dispatch mechanism to initiate calls. The dispatch.py script in the RSVP example provides a reference implementation for automating outbound calls and room management.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}