# Building Healthcare Voice Agents for Appointment Booking: A Complete Pipecat Implementation

> Build a healthcare voice agent for appointment booking using Pipecat, Cartesia, and Nebius LLM. Discover how to handle audio streaming, slot checking, and intelligent escalations for a production ready solution.

- Repository: [Arindam Majumder /awesome-ai-apps](https://github.com/Arindam200/awesome-ai-apps)
- Tags: how-to-guide
- Published: 2026-05-06

---

**You can build a production-ready healthcare voice agent for appointment booking by orchestrating Cartesia STT/TTS, Nebius LLM with tool calling, and Pipecat pipelines to handle audio streaming, slot checking, and intelligent escalations.**

The **Healthcare Voice Contact Center** example in the `Arindam200/awesome-ai-apps` repository demonstrates a complete end-to-end implementation of building healthcare voice agents for appointment booking. Built on the **Pipecat** framework, this system streams real-time audio through a pipeline that transcribes speech, processes intent via an LLM with function calling, and synthesizes responses while managing appointment data and escalation logic.

## Architecture Overview

The voice agent follows a modular pipeline architecture that processes audio in real-time:

```

Caller audio → Cartesia STT → Nebius LLM (tools) → Cartesia TTS → Caller audio

```

According to the source code in [`voice_agents/healthcare_contact_center/main.py`](https://github.com/Arindam200/awesome-ai-apps/blob/main/voice_agents/healthcare_contact_center/main.py), the pipeline is assembled using Pipecat's `Pipeline` class and executed by a `PipelineRunner`. The system supports two transport options: **WebRTC** for local development (default) or **Daily** rooms for production deployment via the `-t daily` flag.

The core services include:
- **STT Service**: `CartesiaSTTService` using the *ink-whisper* model for transcription
- **LLM Service**: `NebiusLLMService` running *openai/gpt-oss-120b* with tool-calling capabilities
- **TTS Service**: `CartesiaTTSService` utilizing the `FRONT_DESK_VOICE` personality by default

## Implementing the Voice Pipeline

### Transport and Audio Streaming

The transport layer is initialized through `create_transport()` in [`main.py`](https://github.com/Arindam200/awesome-ai-apps/blob/main/main.py), which configures either a local WebRTC client or a Daily.co room connection. This transport handles bidirectional audio streaming between the caller and the AI agent.

```python

# From main.py - pipeline construction

transport = create_transport(args.transport)
stt = CartesiaSTTService(api_key=os.getenv("CARTESIA_API_KEY"))
llm = NebiusLLMService(
    api_key=os.getenv("NEBIUS_API_KEY"),
    model="openai/gpt-oss-120b"
)
tts = CartesiaTTSService(
    api_key=os.getenv("CARTESIA_API_KEY"),
    voice=FRONT_DESK_VOICE
)

```

### LLM Context and Tool Registration

The `LLMContext` object manages the conversation state, holding the system prompt (`FRONT_DESK_PROMPT`) and available tools. Tools are defined using `FunctionSchema` objects that specify the function name, description, and JSON schema for arguments.

The appointment booking system implements four primary tools:

- **`check_availability`**: Queries open slots filtered by date and doctor
- **`book_appointment`**: Persists a selected slot for the patient
- **`lookup_faq`**: Performs keyword searches against [`data/faq.json`](https://github.com/Arindam200/awesome-ai-apps/blob/main/data/faq.json)
- **`escalate_to_human`**: Triggers voice/personality switching for complex cases

```python

# Tool schema definition from main.py

check_availability_tool = FunctionSchema(
    name="check_availability",
    description="List open appointment slots for a given date and doctor",
    properties={
        "date": {"type": "string"},
        "doctor": {"type": "string"}
    },
    required=[]
)

```

## Appointment Booking Logic

### Data Layer Implementation

The mock data layer consists of two JSON files in the `data/` directory:
- [`slots.json`](https://github.com/Arindam200/awesome-ai-apps/blob/main/slots.json): Stores available appointment slots
- [`faq.json`](https://github.com/Arindam200/awesome-ai-apps/blob/main/faq.json): Contains question-answer pairs for common queries

The tool implementations in [`tools/appointments.py`](https://github.com/Arindam200/awesome-ai-apps/blob/main/tools/appointments.py) handle reading from and writing to these files. In production, you replace these with REST API calls to your scheduling system.

```python

# Example slot checker (tools/appointments.py)

def check_availability(date: str | None, doctor: str | None):
    with open("data/slots.json") as f:
        slots = json.load(f)
    # Filtering logic here

    return available_slots

```

### Booking Flow

When the LLM detects intent to book an appointment, it invokes the `book_appointment` function with parameters like `date`, `time`, `doctor`, and `patient_name`. The handler in [`tools/appointments.py`](https://github.com/Arindam200/awesome-ai-apps/blob/main/tools/appointments.py) marks the slot as occupied in [`slots.json`](https://github.com/Arindam200/awesome-ai-apps/blob/main/slots.json) and returns a confirmation ID to the LLM, which then verbally confirms the booking to the caller.

## Handling Escalations and Personality Switching

A critical feature of production healthcare voice agents is seamless escalation to human supervisors. The implementation in [`main.py`](https://github.com/Arindam200/awesome-ai-apps/blob/main/main.py) demonstrates dynamic personality switching through two mechanisms:

1. **Voice Change**: Sending a `TTSUpdateSettingsFrame` to switch from `FRONT_DESK_VOICE` to `SUPERVISOR_VOICE`
2. **Prompt Injection**: Appending `SUPERVISOR_PROMPT` to the conversation context to alter the LLM's persona

```python
async def handle_escalate(params: FunctionCallParams):
    result = escalate_to_human(params.arguments)
    
    # Switch voice identity

    await tts.push_frame(
        TTSUpdateSettingsFrame(
            settings=CartesiaTTSService.Settings(voice=SUPERVISOR_VOICE)
        )
    )
    
    # Change system personality

    messages.append({
        "role": "system",
        "content": SUPERVISOR_PROMPT
    })
    await params.result_callback(result)

```

The [`personalities.py`](https://github.com/Arindam200/awesome-ai-apps/blob/main/personalities.py) file contains the full text of `FRONT_DESK_PROMPT` and `SUPERVISOR_PROMPT`, allowing you to customize the agent's tone, empathy level, and escalation criteria.

## Setup and Customization

### Running the Agent Locally

To test the healthcare voice agent with WebRTC transport:

```bash
cd voice_agents/healthcare_contact_center
uv sync  # Install dependencies

uv run python main.py  # Defaults to WebRTC

```

Open the printed URL (typically `http://localhost:7860/client`) in your browser to access the test client with microphone input.

For Daily.co integration:

```bash
export DAILY_API_KEY=your_key
export DAILY_ROOM_URL=your_room_url
uv run python main.py -t daily

```

### Production Customization Points

To adapt this template for live deployment:

- **Backend Integration**: Replace JSON file operations in [`tools/appointments.py`](https://github.com/Arindam200/awesome-ai-apps/blob/main/tools/appointments.py) with calls to your EMR or scheduling API
- **Knowledge Base**: Extend [`data/faq.json`](https://github.com/Arindam200/awesome-ai-apps/blob/main/data/faq.json) or implement vector search in [`tools/knowledge_base.py`](https://github.com/Arindam200/awesome-ai-apps/blob/main/tools/knowledge_base.py) using embeddings
- **Voice Configuration**: Modify `FRONT_DESK_VOICE` and `SUPERVISOR_VOICE` in [`personalities.py`](https://github.com/Arindam200/awesome-ai-apps/blob/main/personalities.py) to match your brand
- **Escalation Bridge**: Replace the stub logger in [`tools/escalation.py`](https://github.com/Arindam200/awesome-ai-apps/blob/main/tools/escalation.py) with Twilio bridges, Daily SIP interconnects, or webhook notifications to human agents

## Summary

- **Pipecat Framework**: The [`main.py`](https://github.com/Arindam200/awesome-ai-apps/blob/main/main.py) implementation demonstrates how to orchestrate STT, LLM, and TTS services into a real-time voice pipeline using `Pipeline` and `PipelineRunner` classes.
- **Tool-Enabled LLM**: The Nebius LLM service uses `FunctionSchema` definitions to handle appointment checking, booking, FAQ lookups, and escalations through structured function calling.
- **Dynamic Personalization**: The system supports runtime voice and personality switching via `TTSUpdateSettingsFrame` and context manipulation, enabling supervisor escalation flows.
- **Extensible Data Layer**: Mock JSON backends in [`tools/appointments.py`](https://github.com/Arindam200/awesome-ai-apps/blob/main/tools/appointments.py) and [`tools/knowledge_base.py`](https://github.com/Arindam200/awesome-ai-apps/blob/main/tools/knowledge_base.py) can be swapped for production APIs without modifying the core pipeline logic.

## Frequently Asked Questions

### What hardware requirements are needed to run this healthcare voice agent?

The voice agent runs on standard consumer hardware with a stable internet connection. The heavy processing (STT, LLM inference, TTS) is handled by cloud APIs (Cartesia and Nebius), so local compute requirements are minimal—only sufficient to run the Python async event loop and audio streaming transport.

### Can I integrate this with my existing scheduling software instead of the JSON files?

Yes. The [`tools/appointments.py`](https://github.com/Arindam200/awesome-ai-apps/blob/main/tools/appointments.py) file is designed as a pluggable layer. Replace the JSON read/write operations with HTTP requests to your scheduling API. Ensure the function signatures remain unchanged (accepting `date` and `doctor` parameters, returning lists of slots) to maintain compatibility with the LLM tool schema defined in [`main.py`](https://github.com/Arindam200/awesome-ai-apps/blob/main/main.py).

### How does the system handle patient privacy and HIPAA compliance?

The open-source example uses mock data and cloud API providers. For HIPAA compliance, you must configure the transport to use encrypted connections (WSS for WebRTC), ensure your Cartesia and Nebius accounts have BAA agreements, replace the JSON data stores with secure databases, and implement proper authentication in the transport layer. The Pipecat framework itself supports these configurations but requires proper infrastructure setup.

### Is it possible to add more complex appointment types or multi-step booking flows?

Yes. You can extend the `FunctionSchema` definitions in [`main.py`](https://github.com/Arindam200/awesome-ai-apps/blob/main/main.py) to include additional parameters like `appointment_type`, `insurance_provider`, or `preferred_location`. For multi-step flows, modify the tool handlers in [`tools/appointments.py`](https://github.com/Arindam200/awesome-ai-apps/blob/main/tools/appointments.py) to return intermediate confirmations, allowing the LLM to ask clarifying questions before finalizing the booking. The conversational state is maintained in the `LLMContext`, supporting arbitrary complexity in the dialogue flow.