How to Configure PAI's Voice System with ElevenLabs and Qwen3 TTS

PAI (Personal AI Infrastructure) supports both cloud-based ElevenLabs and local Qwen3 TTS through a modular voice server architecture that uses VoiceConfig.json for voice mappings and exposes a unified /notify endpoint on localhost:8888.

The Personal AI Infrastructure (PAI) repository by Daniel Miessler provides a flexible, modular voice synthesis layer that lets you choose between high-quality cloud text-to-speech via ElevenLabs or privacy-preserving local generation with Qwen3 TTS. This guide explains how to configure PAI's voice system with ElevenLabs and Qwen3 TTS using the server implementations found in VoiceServer/server.ts and VoiceServer/server.py.

Understanding the Voice Architecture

PAI’s voice system decouples voice configuration from synthesis execution. At its core, the architecture relies on three components:

• Voice Configuration – A VoiceConfig.json file (generated from skills/Prompting/Templates/Primitives/Voice.hbs) maps agent IDs to specific voice IDs, stability settings, and prosody presets.
• Voice Server – An HTTP service running on localhost:8888 that accepts JSON payloads at /notify or /notify/personality, resolves the appropriate voice settings via loadVoiceConfig(), and synthesizes audio.
• TTS Engine – Either the ElevenLabs cloud API (TypeScript/Bun server) or the Qwen3 local model (Python/FastAPI server).

Configuring ElevenLabs Cloud TTS

The ElevenLabs integration provides high-fidelity speech synthesis with emotional prosody control. It is implemented in VoiceServer/server.ts and managed as a macOS LaunchAgent via VoiceServer/install.sh.

Set Environment Credentials

ElevenLabs requires an API key for authentication. Add it to your environment file:

echo 'ELEVENLABS_API_KEY=YOUR_KEY_HERE' >> ~/.env

The server loads ~/.env at startup (lines 23‑31 in server.ts). If the key is missing, the installer falls back to macOS say (see install.sh lines 52‑63).

Map Voices to Agents

Define voice mappings in VoiceConfig.json using the Handlebars template at skills/Prompting/Templates/Primitives/Voice.hbs:

bun run RenderTemplate.ts \
    -t Prompting/Templates/Primitives/Voice.hbs \
    -d skills/Agents/Data/Agents.yaml \
    -o VoiceConfig.json

Place the resulting JSON in ~/.claude/VoiceServer. The loadVoiceConfig() function (lines 4‑9 of server.ts) reads this file to resolve voice_id, stability, and similarity_boost per agent.

Example agent definition in Agents.yaml:

agents:
  serena:
    name: Serena
    voice:
      voice_id: "EXAVITQu4vr4xnSDxMaL"
      voice_name: "Serena"
      rate_wpm: 180
      stability: 0.7
      similarity_boost: 0.9

Deploy the TypeScript Server

Install and start the Bun-based server:

cd Releases/v3.0/.claude/VoiceServer
./install.sh

The script creates a macOS LaunchAgent (com.pai.voice-server) that runs bun run server.ts (lines 90‑106 of install.sh).

Configuring Qwen3 Local TTS

For offline, privacy-preserving synthesis, PAI supports the Qwen3 model via a Python FastAPI server defined in VoiceServer/server.py and configured in VoiceServer/config.py.

Install Python Dependencies

Ensure Python 3.10+ is installed, then install requirements:

cd Releases/v2.5/.claude/VoiceServer
pip install -r requirements.txt

The Qwen3TTSEngine class (lines 78‑84 of server.py) handles model loading and inference.

Launch the FastAPI Server

Start the local server on the same port (8888):

python server.py

The server reads configuration from VoiceServer/config.py (lines 12‑23), including model size and audio format settings. Once running, it exposes the same /notify endpoint as the ElevenLabs server, allowing seamless switching between cloud and local TTS without changing client code.

Sending Voice Notifications

Both servers accept JSON payloads at http://localhost:8888/notify (or /notify/personality for emotion-aware synthesis).

From TypeScript Skills

Use the helper in skills/PAI/Tools/pai.ts or send requests directly:

const payload = {
  message: "[🚀 excited] Build completed successfully!",
  title: "CI",
  voice_id: "YOUR_ELEVENLABS_VOICE_ID"  // optional fallback
};

await fetch("http://localhost:8888/notify/personality", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify(payload)
});

The extractEmotionalMarker function (lines 62‑90 of server.ts) parses emoji tags like [🚀 excited] and applies corresponding prosody presets from EMOTIONAL_PRESETS (lines 31‑54).

From Shell Scripts

For quick notifications from bash:

#!/usr/bin/env bash
msg="[🔥 urgent] Critical error in deployment!"
json=$(jq -n --arg m "$msg" '{message:$m, title:"Ops"}')
curl -s -X POST http://localhost:8888/notify/personality \
     -H "Content-Type: application/json" \
     -d "$json"

The "urgent" marker triggers EMOTIONAL_PRESETS['urgent'], reducing stability to 0.3 for a more agitated tone.

Summary

Configuring PAI's voice system with ElevenLabs and Qwen3 TTS involves three core steps:

• Credential Management – Set ELEVENLABS_API_KEY in ~/.env for cloud synthesis; no API key is required for local Qwen3 operation.
• Voice Mapping – Generate VoiceConfig.json from skills/Prompting/Templates/Primitives/Voice.hbs to bind agents to specific voice IDs, stability, and similarity settings.
• Server Deployment – Run the Bun-based TypeScript server (VoiceServer/server.ts) for ElevenLabs or the Python FastAPI server (VoiceServer/server.py) for Qwen3, both exposing a unified localhost:8888/notify endpoint.

Frequently Asked Questions

What is the difference between ElevenLabs and Qwen3 TTS in PAI?

ElevenLabs provides cloud-based, high-fidelity speech synthesis with advanced prosody control and requires an ELEVENLABS_API_KEY. Qwen3 TTS runs locally via a Python FastAPI server (VoiceServer/server.py), offering privacy-preserving, offline generation without API costs, but requires sufficient local GPU resources for the Qwen3 model.

How do I switch between ElevenLabs and Qwen3 servers?

To switch providers, stop the currently running service and start the alternative. For ElevenLabs, unload the LaunchAgent with launchctl unload "$HOME/Library/LaunchAgents/com.pai.voice-server.plist", then start the Qwen3 Python server with python VoiceServer/server.py. Both servers use the same port (8888) and endpoint structure, so client code in skills/PAI/Tools/pai.ts requires no changes.

Can I use emotional markers with both TTS providers?

Emotional markers (e.g., [🚀 excited], [🔥 urgent]) are parsed by the ElevenLabs server (server.ts) via extractEmotionalMarker and applied as prosody overlays using EMOTIONAL_PRESETS. The Qwen3 server (server.py) may not implement identical emotional preset logic; check the Qwen3TTSEngine implementation in VoiceServer/server.py lines 78‑84 for current capabilities.

Where is the voice configuration stored?

Voice mappings are defined in VoiceConfig.json, typically located in ~/.claude/VoiceServer. This file is generated from the Handlebars template at skills/Prompting/Templates/Primitives/Voice.hbs using agent data from skills/Agents/Data/Agents.yaml. The loadVoiceConfig() function in VoiceServer/server.ts (lines 4‑9) reads this JSON at runtime to resolve voice IDs and stability settings.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}