# Voice-first Agent Architectures: Three Implementation Patterns from the AI Agent Book

> Explore voice-first agent architectures with three key patterns: cascaded, end-to-end, and full-duplex. Learn to balance modularity, latency, and observability.

- Repository: [Bojie Li/ai-agent-book](https://github.com/bojieli/ai-agent-book)
- Tags: deep-dive
- Published: 2026-08-01

---

**Voice-first agent architectures employ one of three distinct patterns—cascaded (turn-based), end-to-end (omni-modal), or full-duplex (streaming)—to balance the trade-offs between system modularity, end-to-end latency, and operational observability.**

The bojieli/ai-agent-book repository explores voice-first agents as a central case study of multimodal interaction, decomposing the pipeline into four discrete layers: **Audio Capture → Speech Processing → Language Model → Speech Synthesis**. This analysis examines the concrete implementations, configuration patterns, and production considerations documented across Chapters 6 through 10 of the repository.

## Three Architectural Paradigms for Voice-First Agents

The repository categorizes voice-first implementations into three paradigms, each with distinct engineering implications for latency and debuggability.

### Cascaded (Turn-Based) Architecture

The **cascaded** paradigm splits the pipeline into discrete, modular stages: Voice Activity Detection (VAD) → Automatic Speech Recognition (ASR) → Large Language Model (LLM) → Text-to-Speech (TTS). Each component exchanges complete utterances, maximizing modularity and allowing independent swapping of providers. This design aligns with the book’s central formula—*Agent = LLM + Context + Tools*—treating voice as both an input context source and an action output mechanism. The reference implementation resides in `chapter9/live-audio/`, where a WebSocket-based backend orchestrates the stage transitions.

### End-to-End (Omni-Modal) Architecture

In the **end-to-end** approach, a single foundation model consumes raw audio and directly emits spoken replies without intermediate text representation. This minimizes pipeline latency but significantly reduces observability and debuggability, as developers cannot inspect the ASR transcription or LLM reasoning steps independently.

### Full-Duplex (Streaming) Architecture

The **full-duplex** pattern enables real-time "barge-in" capabilities by streaming audio fragments to the LLM while the user is still speaking. Voice Activity Detection runs continuously, pushing partial utterances to downstream components rather than waiting for explicit end-of-speech markers. The `chapter10/voice-werewolf/` experiment demonstrates this architecture in a multi-agent setting where participants communicate simultaneously via synthesized audio.

## Implementing the Cascaded Pipeline

The `chapter9/live-audio/` directory contains a production-grade implementation of the cascaded architecture. The backend server, defined in [`chapter9/live-audio/main.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter9/live-audio/main.py), coordinates the four-stage pipeline through a WebSocket interface.

### Voice Activity Detection (VAD) Layer

The system utilizes **Silero VAD** to detect end-of-speech conditions, typically triggered by approximately 500ms of silence. This segmentation logic is documented in [`chapter9/live-audio/backend/validation/real_pipeline_20260729_localwhisper_ark_fish/report.md`](https://github.com/bojieli/ai-agent-book/blob/main/chapter9/live-audio/backend/validation/real_pipeline_20260729_localwhisper_ark_fish/report.md). The VAD layer ensures that only complete utterances are passed to the ASR component, reducing transcription errors from mid-sentence cuts.

### ASR and LLM Integration

After segmentation, audio routes to a configurable ASR provider. The repository supports **OpenAI Whisper**, **OpenRouter**, **ARK**, and **Siliconflow**, among others. The resulting transcription feeds into the LLM alongside the agent’s toolset—such as code interpreters or web-search modules—mirroring the implementation patterns shown in [`book/chapter5.md`](https://github.com/bojieli/ai-agent-book/blob/main/book/chapter5.md) (the Coding Agent chapter).

To launch the cascaded demo:

```bash

# Install chapter-9 dependencies (includes all audio processing libraries)

uv sync --locked --extra ch9

# Start the backend with a specific provider

uv run python chapter9/live-audio/main.py --provider openai

```

The frontend streams raw PCM frames via WebSocket and plays back synthesized audio returned by the server.

### Configuring Text-to-Speech (TTS) Providers

The TTS layer supports multiple synthesis backends, including **ElevenLabs**, **Fish Audio S1** (for zero-shot voice cloning), and **OpenRouter**. Provider configurations and quality evaluation metrics are detailed in [`chapter6/tts-quality-eval/README.md`](https://github.com/bojieli/ai-agent-book/blob/main/chapter6/tts-quality-eval/README.md). This modularity allows developers to optimize for latency, cost, or voice quality without modifying the upstream pipeline logic.

## Advanced Voice Control Techniques

Beyond the basic pipeline, the repository implements techniques for maintaining speaker consistency and controlling prosodic features.

### Speaker Consistency with Orpheus

Chapter 7 introduces **Orpheus**, a voice-cloning system that preserves speaker characteristics across conversation turns using dataset context. The training script selects speaker-specific subsets to ensure timbre stability.

```bash

# Install Orpheus dependencies

uv sync --locked --extra orpheus

# Train on a specific speaker index (e.g., index 3)

uv run python chapter7/orpheus/train.py --dataset-index 3

# Inference with consistent voice identity

uv run python chapter7/orpheus/infer.py --voice "speaker_03"

```

This approach ensures that the agent maintains a recognizable vocal identity throughout extended interactions, as implemented in [`chapter7/orpheus/train.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter7/orpheus/train.py).

### Paralinguistic Control with Sesame

The **Sesame** module (also in Chapter 7) enables control over non-textual speech characteristics through tag-based prompting. Developers can specify emotional or prosodic tags to influence the generated speech patterns.

```bash

# Install Sesame dependencies

uv sync --locked --extra sesame

# Train with specific paralinguistic tags

uv run python chapter7/sesame/train.py --tags "enthusiastic,slow"

# Generate speech with controlled prosody

uv run python chapter7/sesame/infer.py --tag enthusiastic

```

This capability allows voice-first agents to convey enthusiasm, hesitation, or other affective states beyond the literal text content.

## Full-Duplex Multi-Agent Coordination

The **voice-werewolf** experiment in `chapter10/voice-werewolf/` validates full-duplex architectures through a multi-agent game where each participant (human or AI) communicates solely via real-time audio. This implementation tests end-to-end latency, tool-calling via voice, and strict rule-based victory conditions.

To run the full-duplex simulation:

```bash

# Install voice-werewolf dependencies

uv sync --locked --extra voice-werewolf

# Launch with full-duplex streaming mode

uv run python chapter10/voice-werewolf/run_game.py \
    --asr-provider openrouter \
    --tts-provider fishaudio \
    --mode full-duplex

```

The script coordinates multiple agents simultaneously, demonstrating how voice-first architectures scale to complex, real-time collaborative scenarios.

## Summary

- **Voice-first agents** in the bojieli/ai-agent-book repository implement three distinct architectural paradigms: cascaded (modular turn-based), end-to-end (low-latency omni-modal), and full-duplex (streaming with barge-in).

- The **cascaded pipeline** ([`chapter9/live-audio/main.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter9/live-audio/main.py)) provides maximum flexibility by separating VAD, ASR, LLM, and TTS into interchangeable components, supporting providers like Silero VAD, OpenAI Whisper, and Fish Audio.

- **Voice consistency** is achieved through Orpheus (dataset-context cloning) and **prosodic control** through Sesame (paralinguistic tag modeling), both located in `chapter7/`.

- The **full-duplex** architecture is validated through the voice-werewolf multi-agent experiment ([`chapter10/voice-werewolf/run_game.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter10/voice-werewolf/run_game.py)), which tests real-time audio interaction under competitive game conditions.

## Frequently Asked Questions

### What is the primary difference between cascaded and end-to-end voice architectures?

**Cascaded architectures** decompose the pipeline into discrete stages (VAD → ASR → LLM → TTS) that exchange complete utterances, maximizing modularity and debuggability at the cost of higher latency. **End-to-end architectures** use a single model to consume audio and emit speech directly, minimizing latency but sacrificing the ability to inspect intermediate representations like transcriptions or reasoning traces.

### How does Voice Activity Detection (VAD) work in the live-audio demo?

The demo uses **Silero VAD** to monitor incoming audio streams for approximately 500ms of silence, which triggers an end-of-speech event. This segments the audio into complete utterances before passing them to the ASR component, as documented in [`chapter9/live-audio/backend/validation/real_pipeline_20260729_localwhisper_ark_fish/report.md`](https://github.com/bojieli/ai-agent-book/blob/main/chapter9/live-audio/backend/validation/real_pipeline_20260729_localwhisper_ark_fish/report.md).

### Which TTS providers are supported for voice synthesis?

The repository supports multiple synthesis backends including **ElevenLabs**, **Fish Audio S1** (for zero-shot voice cloning), and **OpenRouter**. Configuration details and quality benchmarks are available in [`chapter6/tts-quality-eval/README.md`](https://github.com/bojieli/ai-agent-book/blob/main/chapter6/tts-quality-eval/README.md).

### How can I maintain consistent voice identity across agent interactions?

Use the **Orpheus** module in `chapter7/orpheus/` to train on a specific speaker dataset (via `--dataset-index`), then reference that speaker during inference (`--voice "speaker_03"`). This ensures the agent maintains stable timbre and vocal characteristics across conversation turns, preventing the "voice drift" common in generic TTS systems.