Speech Interaction Paradigms for AI Agents: Live, Streaming, and End-to-End Architectures

The bojieli/ai-agent-book repository implements three distinct speech interaction paradigms—Live-Audio cascaded pipelines, Streaming-Speech with incremental ASR, and End-to-End Speech—to demonstrate real-time voice AI architectures ranging from modular provider chains to single-pass neural inference.

The ability to perceive and generate spoken language defines the next generation of autonomous systems. In the bojieli/ai-agent-book repository, developers explore speech interaction paradigms for AI agents through production-grade implementations that balance latency, accuracy, and computational cost. These paradigms illustrate how modern agents can process voice input using everything from modular cloud APIs to unified local inference models.

The Three Speech Interaction Paradigms

The repository organizes its voice capabilities into three distinct architectural approaches, each targeting different trade-offs between responsiveness and integration complexity.

Live-Audio Cascaded Pipeline

The Live-Audio paradigm implements a real-time voice chat system using a sequential processing chain: Voice Activity Detection (VAD) → Automatic Speech Recognition (ASR) → Large Language Model (LLM) → Text-to-Speech (TTS). This modular approach allows developers to swap individual components without redesigning the entire pipeline.

The frontend captures audio via the Web Audio API and streams raw PCM frames over a WebSocket connection (ws://localhost:8848). The Node.js backend processes this stream through Silero VAD (backend/utils/vad.js) to detect end-of-speech (~500 ms silence), then routes the audio through pluggable providers. Supported configurations include OpenAI Whisper or Siliconflow SenseVoice for ASR, OpenAI GPT-4o, OpenRouter, or ARK Doubao for LLM inference, and Siliconflow CosyVoice2 for speech synthesis.

Repository reference: chapter9/live-audio/README.md

Streaming-Speech with Incremental ASR

The Streaming-Speech paradigm demonstrates the latency-accuracy trade-off by feeding continuously growing audio chunks to an ASR model. Using Qwen2-Audio with incremental prefix encoding, each chunk yields a partial recognition result, enabling ultra-low first-chunk latency at the potential cost of early transcription errors.

This implementation includes a VAD + Whisper baseline for comparison and ships with validation manifests that record raw model outputs, timestamps, and acceptance criteria. The approach targets scenarios where immediate feedback outweighs the need for perfect initial accuracy.

Repository reference: chapter9/streaming-speech/README.md

End-to-End Speech Integration

The End-to-End Speech paradigm eliminates intermediate stages entirely by using MiniCPM-o 4.5, a fully integrated model that maps raw audio directly to text and then to speech within a single forward pass. This architecture avoids explicit VAD, ASR, and TTS modules, maximizing accuracy while requiring significant local compute.

The experiment runs on a single RTX PRO 6000 GPU and produces 24 kHz audio output. Unlike the cascaded approach, this paradigm processes the entire acoustic signal through one neural network, reducing compounding errors between pipeline stages.

Repository reference: chapter9/end-to-end-speech/README.md

System Architecture and Provider Registry

Frontend and Backend Data Flow

The Live-Audio implementation uses a Next.js frontend to capture microphone input and stream raw PCM frames over WebSocket to a Node.js backend. The backend architecture consists of three key layers:

1. Voice Activity Detection – The backend/utils/vad.js module implements server-side Silero VAD to segment speech streams.
2. Provider Factories – Dynamically instantiated ASR, LLM, and TTS providers based on backend/config.js enable seamless vendor swapping.
3. Data Pipeline – User speech flows through VAD → ASR → LLM → TTS → Audio Response with per-stage latency logging.

Pluggable Provider System

The repository centralizes provider metadata in agentbook/providers/models.py. Each Provider dataclass declares a canonical name, default endpoint, default model, environment-variable keys, and namespace requirements. The resolver in agentbook/providers/resolution.py constructs concrete Backend objects containing resolved API keys, endpoints, and model IDs.

from agentbook.providers import resolve_backend

# Resolve the "kimi" provider (Moonshot) – pulls MOONSHOT_API_KEY

backend = resolve_backend("kimi")
print(backend.api_key, backend.base_url, backend.model, backend.using_openrouter)

# Output: ('<key>', 'https://api.moonshot.cn/v1', 'kimi-k3', False)

This architecture supports both commercial APIs (OpenAI, OpenRouter) and locally hosted models (Ollama, custom ONNX pipelines), with OpenRouter serving as a fallback routing layer when specified.

Practical Implementation Guide

Configuring Speech Providers

Copy the example configuration and edit environment variables to select your provider stack:

// backend/config.js.example
const config = {
  ASR_PROVIDER: 'siliconflow',      // 'openai' (whisper) or 'siliconflow' (SenseVoice)
  LLM_PROVIDER: 'openrouter',       // 'openrouter', 'openai', 'ark', 'openrouter-gemini'
  TTS_PROVIDER: 'siliconflow',      // 'siliconflow' (CosyVoice2)

  // Environment-variable helpers
  OPENAI_API_KEY: process.env.OPENAI_API_KEY,
  OPENROUTER_API_KEY: process.env.OPENROUTER_API_KEY,
  ARK_API_KEY: process.env.ARK_API_KEY,
  SILICONFLOW_API_KEY: process.env.SILICONFLOW_API_KEY,
};
module.exports = config;

Running the Live-Audio Demo

Install dependencies and verify your environment before starting the servers:

# Install dependencies

cd backend && npm install
cd ../frontend && npm install

# Verify environment (Node ≥16, FFmpeg, VAD model)

cd backend && npm run check   # or `node check-setup.js`

# Start servers

cd backend && npm start            # WebSocket on port 8848

cd ../frontend && npm run dev      # Next.js dev server on port 3000

Open http://localhost:3000 in Chrome, click Start Recording, and speak. The backend displays per-stage latency logs and plays back the synthesized response.

Resolving Providers Programmatically

For Python-based agent implementations, use the provider resolver to handle API key management and endpoint selection:

from agentbook.providers import resolve_backend

# Resolve OpenRouter fallback for GPT-4o

backend = resolve_backend("openrouter")
print(backend.model)  # 'openai/gpt-4o'

print(backend.using_openrouter)  # True

Summary

• Three paradigms cover the spectrum of speech interaction: modular cascaded pipelines for flexibility, streaming ASR for low latency, and end-to-end models for accuracy.
• Pluggable architecture in agentbook/providers/resolution.py enables swapping between OpenAI, Siliconflow, ARK, and OpenRouter without code changes.
• Real-time processing uses Silero VAD (backend/utils/vad.js) and WebSocket streaming to achieve conversational response times.
• Validation frameworks accompany each paradigm, providing reproducible benchmarks for latency and transcription accuracy.
• Hardware requirements vary from standard cloud APIs (cascaded) to high-end local GPUs like the RTX PRO 6000 (end-to-end).

Frequently Asked Questions

What is the difference between cascaded and end-to-end speech interaction?

Cascaded pipelines process speech through discrete stages (VAD → ASR → LLM → TTS), allowing component swapping and cloud-provider flexibility. End-to-end speech uses a single model like MiniCPM-o 4.5 to map audio directly to speech output, eliminating intermediate representations and reducing error propagation but requiring significant local compute resources.

How does the streaming speech paradigm reduce latency?

The streaming speech implementation feeds continuously growing audio chunks to Qwen2-Audio using incremental prefix encoding. This yields partial recognition results before the user finishes speaking, achieving sub-second first-chunk latency compared to waiting for a complete utterance in standard ASR pipelines.

Which hardware is required for the end-to-end speech demo?

The End-to-End Speech paradigm requires an RTX PRO 6000 GPU to run MiniCPM-o 4.5 inference locally. This hardware supports the single-pass processing of raw audio to 24 kHz speech output, whereas the cascaded and streaming paradigms can run on standard cloud APIs without dedicated local GPUs.

How do I switch between different ASR or TTS providers?

Modify backend/config.js to change the ASR_PROVIDER and TTS_PROVIDER values. Valid options include 'openai' (Whisper), 'siliconflow' (SenseVoice/CosyVoice2), and 'ark' for specific Chinese-language models. The provider resolver in agentbook/providers/resolution.py automatically handles API key selection and endpoint routing based on these configuration values.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}