Three Paradigms for Voice Interaction in Multimodal AI Agents: Cascaded, End-to-End, and Full-Duplex Explained

Multimodal AI agents implement voice interaction through three distinct architectural paradigms—Cascaded pipelines (ASR → LLM → TTS), End-to-End models (Audio → LLM → Audio), and Full-Modal/Full-Duplex systems (bidirectional streaming)—each offering specific trade-offs between modularity, latency, and the preservation of paralinguistic cues.

Voice-capable AI agents require sophisticated architectures to handle the audio-to-text-to-audio pipeline while maintaining conversational flow. According to the bojieli/ai-agent-book repository, specifically Chapter 9, these systems employ three paradigms for voice interaction that define whether processing occurs through discrete stages or unified models. These patterns determine latency characteristics, turn-taking capabilities, and whether nuances like tone and emotion survive the transcription process.

1. Cascaded Pipeline (ASR → LLM → TTS)

The Cascaded paradigm represents the classic modular approach where three separate components communicate via plain text. Speech is first transcribed by an Automatic Speech Recognition (ASR) model, the resulting transcript is processed by a language model (LLM) for reasoning, and the LLM’s output is converted to speech by a Text-to-Speech (TTS) model.

This architecture prioritizes modularity and easy swapping of individual components. You can upgrade the ASR engine without retraining the LLM, or switch TTS providers independently. However, latency accumulates across each stage, and paralinguistic information like emotion or tone is typically lost after the ASR stage converts audio to text.

According to the repository’s chapter9/live-audio/README.md, this implementation uses distinct providers for each stage:

from live_audio import LiveAudioAgent

agent = LiveAudioAgent(
    asr_provider="openai",      # ASR service

    llm_provider="openrouter",  # LLM service

    tts_provider="siliconflow" # TTS service

)

# Runs the full cascade: speech → text → LLM → speech

agent.run()

This pattern suits voice assistants that prioritize flexibility over millisecond-level latency, such as classic smart-speaker bots where component independence matters more than conversational fluidity.

2. End-to-End Architecture (Audio → LLM → Audio)

The End-to-End paradigm employs a single monolithic model that consumes raw audio waveforms and directly produces audio output. The model jointly learns acoustic, linguistic, and prosodic features, internalizing the "thinking" step within its transformer architecture.

By eliminating the text-based hand-off between ASR and TTS, this approach reduces the number of transition points and preserves paralinguistic cues including tone, emotion, and background sounds. The model receives a waveform, encodes it into a learned representation, runs the LLM on that representation, and decodes the response back to a waveform.

As implemented in chapter9/end-to-end-speech/README.md, this approach supports Step-Audio R1-style agents that require ultra-low latency and acoustic nuance:

from end_to_end_speech import EndToEndAgent

# The model directly consumes raw audio and returns raw audio

agent = EndToEndAgent(
    model_name="gpt-audio",  # An end-to-end model (audio-in → audio-out)

    device="cuda"
)

agent.run()

While this architecture demands a single monolithic training run and offers less flexibility to swap components, it achieves tighter latency and retains voice characteristics that cascaded pipelines typically lose.

3. Full-Modal and Full-Duplex (Bidirectional Audio)

The Full-Modal or Full-Duplex paradigm enables the agent to listen and speak simultaneously, creating a true conversational loop. This architecture combines a streaming ASR front-end with an LLM that can produce partial outputs while still receiving incoming audio, enabling "thinking while listening" and real-time turn-taking.

Streaming ASR produces incremental transcripts that continuously update the LLM’s internal state. The system outputs audio fragments synthesized on-the-fly, allowing humans to interrupt the agent mid-sentence or engage in rapid back-and-forth dialogue without the "think-then-speak" pause.

The chapter9/phone-agent/README.md demonstrates this with explicit streaming parameters:

from phone_agent import PhoneAgent

agent = PhoneAgent(
    streaming_asr=True,     # Incremental ASR

    llm_provider="openrouter",
    tts_streaming=True      # Stream audio back while still listening

)

# Agent can listen and speak simultaneously during a call

agent.start_call("+1-555-123-4567")

This paradigm is essential for phone agents and live-audio demos where natural conversation flow requires immediate responsiveness and bidirectional audio processing.

Architectural Trade-offs and Selection Criteria

When selecting among these three paradigms for voice interaction, consider the following structural differences:

• Cascaded systems offer maximum modularity—you can independently upgrade the ASR model or swap TTS voices without retraining the entire pipeline. However, the sequential processing introduces higher cumulative latency and strips acoustic nuances.

• End-to-End models provide lowest latency and preserve paralinguistic information (tone, emotion, background sounds), but require monolithic training runs and offer limited customization of individual stages.

• Full-Duplex architectures minimize perceived response delay through bidirectional streaming, enabling interruption and true turn-taking. This requires sophisticated streaming ASR and incremental LLM generation, increasing architectural complexity.

The visualization in book-en/images/fig10-13.svg ("Real-time voice interaction (ASR + LLM + TTS)") illustrates these pipeline differences as implemented in the repository.

Implementation Reference in ai-agent-book

The bojieli/ai-agent-book repository provides runnable implementations for all three paradigms in Chapter 9:

File	Description
chapter9/README.en.md	High-level overview listing the three paradigms and companion projects
chapter9/live-audio/README.md	Cascaded pipeline implementation with separate ASR, LLM, and TTS providers
chapter9/end-to-end-speech/README.md	End-to-end speech model architecture and usage instructions
chapter9/phone-agent/README.md	Full-duplex phone agent demo including tool definitions (make_phone_call)
book-en/images/fig10-13.svg	Visual diagram of the real-time voice interaction paradigms

Each demo is marked as runnable in the chapter’s README after configuring the required API keys for the respective providers.

Summary

• Multimodal agents employ three distinct voice interaction paradigms: Cascaded (ASR → LLM → TTS), End-to-End (Audio → LLM → Audio), and Full-Modal/Full-Duplex (bidirectional streaming).
• Cascaded architectures prioritize modularity and component independence but sacrifice latency and acoustic nuance.
• End-to-End models minimize latency and preserve paralinguistic features through unified audio-to-audio processing.
• Full-Duplex systems enable simultaneous listening and speaking through streaming ASR and incremental generation, essential for natural phone conversations.
• The bojieli/ai-agent-book repository provides working implementations of all three patterns in chapter9/, including specific class implementations like LiveAudioAgent, EndToEndAgent, and PhoneAgent.

Frequently Asked Questions

What is the main latency difference between Cascaded and End-to-End voice agents?

Cascaded agents accumulate latency across three sequential stages (ASR transcription, LLM inference, TTS synthesis), while End-to-End models process audio through a single forward pass, eliminating hand-off delays between discrete components. According to the repository analysis, End-to-End architectures achieve tighter latency specifically because they avoid the text-based intermediate representation required by Cascaded pipelines.

Can Full-Duplex agents really listen and speak at the same time?

Yes, Full-Modal or Full-Duplex agents implement bidirectional audio streams that allow the system to process incoming audio through streaming ASR while simultaneously generating output through streaming TTS. As shown in the PhoneAgent implementation with streaming_asr=True and tts_streaming=True, the LLM updates its internal state continuously, enabling the agent to handle interruptions and rapid turn-taking during live phone calls.

Why would I choose a Cascaded pipeline over an End-to-End model?

Choose a Cascaded architecture when you require modularity to independently upgrade or swap ASR and TTS components. The repository notes that Cascaded pipelines allow easy substitution of providers (e.g., switching from OpenAI to another ASR service) without retraining the entire system, whereas End-to-End models demand monolithic training and offer limited component-level customization despite their superior latency and acoustic feature preservation.

Do End-to-End models preserve emotion and tone better than Cascaded systems?

Yes, End-to-End models inherently retain paralinguistic information including emotion, tone, and background sounds because they process raw audio waveforms directly without converting to text. Cascaded pipelines lose these acoustic nuances at the ASR stage when speech is converted to text, making End-to-End architectures preferable for applications requiring emotional intelligence or acoustic context awareness.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}