Building Healthcare Voice Agents for Appointment Booking: A Complete Pipecat Implementation

You can build a production-ready healthcare voice agent for appointment booking by orchestrating Cartesia STT/TTS, Nebius LLM with tool calling, and Pipecat pipelines to handle audio streaming, slot checking, and intelligent escalations.

The Healthcare Voice Contact Center example in the Arindam200/awesome-ai-apps repository demonstrates a complete end-to-end implementation of building healthcare voice agents for appointment booking. Built on the Pipecat framework, this system streams real-time audio through a pipeline that transcribes speech, processes intent via an LLM with function calling, and synthesizes responses while managing appointment data and escalation logic.

Architecture Overview

The voice agent follows a modular pipeline architecture that processes audio in real-time:

Caller audio → Cartesia STT → Nebius LLM (tools) → Cartesia TTS → Caller audio

According to the source code in voice_agents/healthcare_contact_center/main.py, the pipeline is assembled using Pipecat's Pipeline class and executed by a PipelineRunner. The system supports two transport options: WebRTC for local development (default) or Daily rooms for production deployment via the -t daily flag.

The core services include:

• STT Service: CartesiaSTTService using the ink-whisper model for transcription
• LLM Service: NebiusLLMService running openai/gpt-oss-120b with tool-calling capabilities
• TTS Service: CartesiaTTSService utilizing the FRONT_DESK_VOICE personality by default

Implementing the Voice Pipeline

Transport and Audio Streaming

The transport layer is initialized through create_transport() in main.py, which configures either a local WebRTC client or a Daily.co room connection. This transport handles bidirectional audio streaming between the caller and the AI agent.

# From main.py - pipeline construction

transport = create_transport(args.transport)
stt = CartesiaSTTService(api_key=os.getenv("CARTESIA_API_KEY"))
llm = NebiusLLMService(
    api_key=os.getenv("NEBIUS_API_KEY"),
    model="openai/gpt-oss-120b"
)
tts = CartesiaTTSService(
    api_key=os.getenv("CARTESIA_API_KEY"),
    voice=FRONT_DESK_VOICE
)

LLM Context and Tool Registration

The LLMContext object manages the conversation state, holding the system prompt (FRONT_DESK_PROMPT) and available tools. Tools are defined using FunctionSchema objects that specify the function name, description, and JSON schema for arguments.

The appointment booking system implements four primary tools:

• check_availability: Queries open slots filtered by date and doctor
• book_appointment: Persists a selected slot for the patient
• lookup_faq: Performs keyword searches against data/faq.json
• escalate_to_human: Triggers voice/personality switching for complex cases

# Tool schema definition from main.py

check_availability_tool = FunctionSchema(
    name="check_availability",
    description="List open appointment slots for a given date and doctor",
    properties={
        "date": {"type": "string"},
        "doctor": {"type": "string"}
    },
    required=[]
)

Appointment Booking Logic

Data Layer Implementation

The mock data layer consists of two JSON files in the data/ directory:

• slots.json: Stores available appointment slots
• faq.json: Contains question-answer pairs for common queries

The tool implementations in tools/appointments.py handle reading from and writing to these files. In production, you replace these with REST API calls to your scheduling system.

# Example slot checker (tools/appointments.py)

def check_availability(date: str | None, doctor: str | None):
    with open("data/slots.json") as f:
        slots = json.load(f)
    # Filtering logic here

    return available_slots

Booking Flow

When the LLM detects intent to book an appointment, it invokes the book_appointment function with parameters like date, time, doctor, and patient_name. The handler in tools/appointments.py marks the slot as occupied in slots.json and returns a confirmation ID to the LLM, which then verbally confirms the booking to the caller.

Handling Escalations and Personality Switching

A critical feature of production healthcare voice agents is seamless escalation to human supervisors. The implementation in main.py demonstrates dynamic personality switching through two mechanisms:

1. Voice Change: Sending a TTSUpdateSettingsFrame to switch from FRONT_DESK_VOICE to SUPERVISOR_VOICE
2. Prompt Injection: Appending SUPERVISOR_PROMPT to the conversation context to alter the LLM's persona

async def handle_escalate(params: FunctionCallParams):
    result = escalate_to_human(params.arguments)
    
    # Switch voice identity

    await tts.push_frame(
        TTSUpdateSettingsFrame(
            settings=CartesiaTTSService.Settings(voice=SUPERVISOR_VOICE)
        )
    )
    
    # Change system personality

    messages.append({
        "role": "system",
        "content": SUPERVISOR_PROMPT
    })
    await params.result_callback(result)

The personalities.py file contains the full text of FRONT_DESK_PROMPT and SUPERVISOR_PROMPT, allowing you to customize the agent's tone, empathy level, and escalation criteria.

Setup and Customization

Running the Agent Locally

To test the healthcare voice agent with WebRTC transport:

cd voice_agents/healthcare_contact_center
uv sync  # Install dependencies

uv run python main.py  # Defaults to WebRTC

Open the printed URL (typically http://localhost:7860/client) in your browser to access the test client with microphone input.

For Daily.co integration:

export DAILY_API_KEY=your_key
export DAILY_ROOM_URL=your_room_url
uv run python main.py -t daily

Production Customization Points

To adapt this template for live deployment:

• Backend Integration: Replace JSON file operations in tools/appointments.py with calls to your EMR or scheduling API
• Knowledge Base: Extend data/faq.json or implement vector search in tools/knowledge_base.py using embeddings
• Voice Configuration: Modify FRONT_DESK_VOICE and SUPERVISOR_VOICE in personalities.py to match your brand
• Escalation Bridge: Replace the stub logger in tools/escalation.py with Twilio bridges, Daily SIP interconnects, or webhook notifications to human agents

Summary

• Pipecat Framework: The main.py implementation demonstrates how to orchestrate STT, LLM, and TTS services into a real-time voice pipeline using Pipeline and PipelineRunner classes.
• Tool-Enabled LLM: The Nebius LLM service uses FunctionSchema definitions to handle appointment checking, booking, FAQ lookups, and escalations through structured function calling.
• Dynamic Personalization: The system supports runtime voice and personality switching via TTSUpdateSettingsFrame and context manipulation, enabling supervisor escalation flows.
• Extensible Data Layer: Mock JSON backends in tools/appointments.py and tools/knowledge_base.py can be swapped for production APIs without modifying the core pipeline logic.

Frequently Asked Questions

What hardware requirements are needed to run this healthcare voice agent?

The voice agent runs on standard consumer hardware with a stable internet connection. The heavy processing (STT, LLM inference, TTS) is handled by cloud APIs (Cartesia and Nebius), so local compute requirements are minimal—only sufficient to run the Python async event loop and audio streaming transport.

Can I integrate this with my existing scheduling software instead of the JSON files?

Yes. The tools/appointments.py file is designed as a pluggable layer. Replace the JSON read/write operations with HTTP requests to your scheduling API. Ensure the function signatures remain unchanged (accepting date and doctor parameters, returning lists of slots) to maintain compatibility with the LLM tool schema defined in main.py.

How does the system handle patient privacy and HIPAA compliance?

The open-source example uses mock data and cloud API providers. For HIPAA compliance, you must configure the transport to use encrypted connections (WSS for WebRTC), ensure your Cartesia and Nebius accounts have BAA agreements, replace the JSON data stores with secure databases, and implement proper authentication in the transport layer. The Pipecat framework itself supports these configurations but requires proper infrastructure setup.

Is it possible to add more complex appointment types or multi-step booking flows?

Yes. You can extend the FunctionSchema definitions in main.py to include additional parameters like appointment_type, insurance_provider, or preferred_location. For multi-step flows, modify the tool handlers in tools/appointments.py to return intermediate confirmations, allowing the LLM to ask clarifying questions before finalizing the booking. The conversational state is maintained in the LLMContext, supporting arbitrary complexity in the dialogue flow.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}