# How to Configure PAI's Voice System with ElevenLabs and Qwen3 TTS

> Learn to configure PAI's voice system with ElevenLabs and Qwen3 TTS. Our guide details using VoiceConfig.json and the unified notify endpoint for seamless TTS integration on localhost:8888.

- Repository: [Daniel Miessler 🛡️/Personal_AI_Infrastructure](https://github.com/danielmiessler/personal_ai_infrastructure)
- Tags: how-to-guide
- Published: 2026-02-16

---

**PAI (Personal AI Infrastructure) supports both cloud-based ElevenLabs and local Qwen3 TTS through a modular voice server architecture that uses [`VoiceConfig.json`](https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/VoiceConfig.json) for voice mappings and exposes a unified `/notify` endpoint on `localhost:8888`.**

The **Personal AI Infrastructure (PAI)** repository by Daniel Miessler provides a flexible, modular voice synthesis layer that lets you choose between high-quality cloud text-to-speech via **ElevenLabs** or privacy-preserving local generation with **Qwen3 TTS**. This guide explains how to configure PAI's voice system with ElevenLabs and Qwen3 TTS using the server implementations found in [`VoiceServer/server.ts`](https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/VoiceServer/server.ts) and [`VoiceServer/server.py`](https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/VoiceServer/server.py).

## Understanding the Voice Architecture

PAI’s voice system decouples voice configuration from synthesis execution. At its core, the architecture relies on three components:

- **Voice Configuration** – A [`VoiceConfig.json`](https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/VoiceConfig.json) file (generated from `skills/Prompting/Templates/Primitives/Voice.hbs`) maps agent IDs to specific voice IDs, stability settings, and prosody presets.
- **Voice Server** – An HTTP service running on `localhost:8888` that accepts JSON payloads at `/notify` or `/notify/personality`, resolves the appropriate voice settings via `loadVoiceConfig()`, and synthesizes audio.
- **TTS Engine** – Either the **ElevenLabs** cloud API (TypeScript/Bun server) or the **Qwen3** local model (Python/FastAPI server).

## Configuring ElevenLabs Cloud TTS

The ElevenLabs integration provides high-fidelity speech synthesis with emotional prosody control. It is implemented in [`VoiceServer/server.ts`](https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/VoiceServer/server.ts) and managed as a macOS LaunchAgent via [`VoiceServer/install.sh`](https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/VoiceServer/install.sh).

### Set Environment Credentials

ElevenLabs requires an API key for authentication. Add it to your environment file:

```bash
echo 'ELEVENLABS_API_KEY=YOUR_KEY_HERE' >> ~/.env

```

The server loads `~/.env` at startup (lines 23‑31 in [`server.ts`](https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/server.ts)). If the key is missing, the installer falls back to macOS `say` (see [`install.sh`](https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/install.sh) lines 52‑63).

### Map Voices to Agents

Define voice mappings in [`VoiceConfig.json`](https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/VoiceConfig.json) using the Handlebars template at `skills/Prompting/Templates/Primitives/Voice.hbs`:

```bash
bun run RenderTemplate.ts \
    -t Prompting/Templates/Primitives/Voice.hbs \
    -d skills/Agents/Data/Agents.yaml \
    -o VoiceConfig.json

```

Place the resulting JSON in `~/.claude/VoiceServer`. The `loadVoiceConfig()` function (lines 4‑9 of [`server.ts`](https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/server.ts)) reads this file to resolve `voice_id`, `stability`, and `similarity_boost` per agent.

Example agent definition in [`Agents.yaml`](https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/Agents.yaml):

```yaml
agents:
  serena:
    name: Serena
    voice:
      voice_id: "EXAVITQu4vr4xnSDxMaL"
      voice_name: "Serena"
      rate_wpm: 180
      stability: 0.7
      similarity_boost: 0.9

```

### Deploy the TypeScript Server

Install and start the Bun-based server:

```bash
cd Releases/v3.0/.claude/VoiceServer
./install.sh

```

The script creates a macOS LaunchAgent (`com.pai.voice-server`) that runs `bun run server.ts` (lines 90‑106 of [`install.sh`](https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/install.sh)).

## Configuring Qwen3 Local TTS

For offline, privacy-preserving synthesis, PAI supports the **Qwen3** model via a Python FastAPI server defined in [`VoiceServer/server.py`](https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/VoiceServer/server.py) and configured in [`VoiceServer/config.py`](https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/VoiceServer/config.py).

### Install Python Dependencies

Ensure Python 3.10+ is installed, then install requirements:

```bash
cd Releases/v2.5/.claude/VoiceServer
pip install -r requirements.txt

```

The `Qwen3TTSEngine` class (lines 78‑84 of [`server.py`](https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/server.py)) handles model loading and inference.

### Launch the FastAPI Server

Start the local server on the same port (`8888`):

```bash
python server.py

```

The server reads configuration from [`VoiceServer/config.py`](https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/VoiceServer/config.py) (lines 12‑23), including model size and audio format settings. Once running, it exposes the same `/notify` endpoint as the ElevenLabs server, allowing seamless switching between cloud and local TTS without changing client code.

## Sending Voice Notifications

Both servers accept JSON payloads at `http://localhost:8888/notify` (or `/notify/personality` for emotion-aware synthesis).

### From TypeScript Skills

Use the helper in [`skills/PAI/Tools/pai.ts`](https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/skills/PAI/Tools/pai.ts) or send requests directly:

```typescript
const payload = {
  message: "[🚀 excited] Build completed successfully!",
  title: "CI",
  voice_id: "YOUR_ELEVENLABS_VOICE_ID"  // optional fallback
};

await fetch("http://localhost:8888/notify/personality", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify(payload)
});

```

The `extractEmotionalMarker` function (lines 62‑90 of [`server.ts`](https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/server.ts)) parses emoji tags like `[🚀 excited]` and applies corresponding prosody presets from `EMOTIONAL_PRESETS` (lines 31‑54).

### From Shell Scripts

For quick notifications from bash:

```bash
#!/usr/bin/env bash
msg="[🔥 urgent] Critical error in deployment!"
json=$(jq -n --arg m "$msg" '{message:$m, title:"Ops"}')
curl -s -X POST http://localhost:8888/notify/personality \
     -H "Content-Type: application/json" \
     -d "$json"

```

The "urgent" marker triggers `EMOTIONAL_PRESETS['urgent']`, reducing stability to 0.3 for a more agitated tone.

## Summary

Configuring PAI's voice system with ElevenLabs and Qwen3 TTS involves three core steps:

- **Credential Management** – Set `ELEVENLABS_API_KEY` in `~/.env` for cloud synthesis; no API key is required for local Qwen3 operation.
- **Voice Mapping** – Generate [`VoiceConfig.json`](https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/VoiceConfig.json) from `skills/Prompting/Templates/Primitives/Voice.hbs` to bind agents to specific voice IDs, stability, and similarity settings.
- **Server Deployment** – Run the Bun-based TypeScript server ([`VoiceServer/server.ts`](https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/VoiceServer/server.ts)) for ElevenLabs or the Python FastAPI server ([`VoiceServer/server.py`](https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/VoiceServer/server.py)) for Qwen3, both exposing a unified `localhost:8888/notify` endpoint.

## Frequently Asked Questions

### What is the difference between ElevenLabs and Qwen3 TTS in PAI?

**ElevenLabs** provides cloud-based, high-fidelity speech synthesis with advanced prosody control and requires an `ELEVENLABS_API_KEY`. **Qwen3 TTS** runs locally via a Python FastAPI server ([`VoiceServer/server.py`](https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/VoiceServer/server.py)), offering privacy-preserving, offline generation without API costs, but requires sufficient local GPU resources for the Qwen3 model.

### How do I switch between ElevenLabs and Qwen3 servers?

To switch providers, stop the currently running service and start the alternative. For ElevenLabs, unload the LaunchAgent with `launchctl unload "$HOME/Library/LaunchAgents/com.pai.voice-server.plist"`, then start the Qwen3 Python server with `python VoiceServer/server.py`. Both servers use the same port (`8888`) and endpoint structure, so client code in [`skills/PAI/Tools/pai.ts`](https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/skills/PAI/Tools/pai.ts) requires no changes.

### Can I use emotional markers with both TTS providers?

Emotional markers (e.g., `[🚀 excited]`, `[🔥 urgent]`) are parsed by the **ElevenLabs server** ([`server.ts`](https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/server.ts)) via `extractEmotionalMarker` and applied as prosody overlays using `EMOTIONAL_PRESETS`. The **Qwen3 server** ([`server.py`](https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/server.py)) may not implement identical emotional preset logic; check the `Qwen3TTSEngine` implementation in [`VoiceServer/server.py`](https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/VoiceServer/server.py) lines 78‑84 for current capabilities.

### Where is the voice configuration stored?

Voice mappings are defined in **[`VoiceConfig.json`](https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/VoiceConfig.json)**, typically located in `~/.claude/VoiceServer`. This file is generated from the Handlebars template at `skills/Prompting/Templates/Primitives/Voice.hbs` using agent data from [`skills/Agents/Data/Agents.yaml`](https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/skills/Agents/Data/Agents.yaml). The `loadVoiceConfig()` function in [`VoiceServer/server.ts`](https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/VoiceServer/server.ts) (lines 4‑9) reads this JSON at runtime to resolve voice IDs and stability settings.