# How to Implement Semantic Turn Detection in LiveKit Agents: 3 Methods Explained

> Learn 3 methods to implement semantic turn detection in LiveKit Agents. Configure TurnDetection or MultilingualModel for precise control and enhanced conversational AI.

- Repository: [LiveKit/agents](https://github.com/livekit/agents)
- Tags: how-to-guide
- Published: 2026-03-06

---

**To implement semantic turn detection in LiveKit Agents, pass a `TurnDetection` configuration or `MultilingualModel` instance to the `AgentSession` constructor while disabling the provider's native VAD to avoid conflicts.**

LiveKit Agents is an open-source framework for building real-time voice AI applications. **Semantic turn detection** enables your voice agents to identify when a user has finished speaking by analyzing transcript content rather than relying solely on audio energy levels, resulting in more natural conversations and lower latency.

## What Is Semantic Turn Detection?

Traditional voice activity detection (VAD) relies on audio energy thresholds to determine when someone stops speaking. **Semantic turn detection** (also called semantic VAD) uses language models to predict end-of-utterance (EOU) based on the conversational context. This approach works reliably even with whispered speech, background noise, or across multiple languages.

According to the source code in [`livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/utils.py`](https://github.com/livekit/agents/blob/main/livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/utils.py) (lines 28-33), LiveKit provides a `DEFAULT_TURN_DETECTION` configuration that uses `type="semantic_vad"` with medium eagerness and interrupt capabilities enabled.

## Architecture and Key Components

The semantic turn detection system in LiveKit Agents consists of these core components:

| Component | Source File | Purpose |
|-----------|-------------|---------|
| `to_turn_detection` | [`livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/utils.py`](https://github.com/livekit/agents/blob/main/livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/utils.py) (lines 27-35) | Normalizes user configuration into OpenAI `RealtimeAudioInputTurnDetection` objects |
| `MultilingualModel` | [`livekit-plugins/livekit-plugins-turn-detector/livekit/plugins/turn_detector/multilingual.py`](https://github.com/livekit/agents/blob/main/livekit-plugins/livekit-plugins-turn-detector/livekit/plugins/turn_detector/multilingual.py) | Provides language-aware end-of-turn prediction with optional remote inference |
| `AgentSession` | [`livekit-agents/livekit/agents/voice/speech_handle.py`](https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/voice/speech_handle.py) | Runtime orchestrator that wires VAD, STT, and LLM components |
| `RealtimeModel` | [`livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/realtime_model.py`](https://github.com/livekit/agents/blob/main/livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/realtime_model.py) | OpenAI Realtime API wrapper that consumes turn detection configs |

## Method 1: Using the Multilingual Turn Detector Plugin

The `MultilingualModel` class provides **semantic end-of-utterance detection** across many languages without relying on OpenAI's built-in semantic VAD. This approach is ideal when you need language-specific turn detection or want to use a custom inference service.

Key features:
- Implements `predict_end_of_turn` for context-aware EOU prediction
- Supports remote inference via the `LIVEKIT_REMOTE_EOT_URL` environment variable
- Integrates with `AgentSession` through the `turn_detection` parameter

```python
from livekit.plugins.turn_detector.multilingual import MultilingualModel

session = AgentSession(
    turn_detection=MultilingualModel(),  # Uses LiveKit's semantic detector

    vad=silero.VAD.load(),
    stt=deepgram.STT(),
    llm=openai.realtime.RealtimeModel(
        voice="alloy",
        turn_detection=None,  # Disable OpenAI's native VAD

    ),
)

```

As shown in [`examples/voice_agents/realtime_turn_detector.py`](https://github.com/livekit/agents/blob/main/examples/voice_agents/realtime_turn_detector.py), this configuration disables the built-in OpenAI VAD (`turn_detection=None`) and injects LiveKit's multilingual model instead.

## Method 2: Native OpenAI Realtime Semantic VAD

For OpenAI Realtime API users, you can configure semantic VAD directly through the `TurnDetection` model. The `to_turn_detection` utility in [`utils.py`](https://github.com/livekit/agents/blob/main/utils.py) (lines 90-121) converts these configurations into `SemanticVad` instances.

```python
from openai.types.beta.realtime.session import TurnDetection

custom_vad = TurnDetection(
    type="semantic_vad",
    create_response=True,
    eagerness="medium",
    interrupt_response=True,
)

session = AgentSession(
    turn_detection=custom_vad,
    llm=openai.realtime.RealtimeModel(turn_detection=None),
)

```

When `type="semantic_vad"` is specified, the Realtime API performs end-of-utterance detection in the text domain rather than using raw audio energy.

## Method 3: Default Semantic VAD Configuration

If you provide no `turn_detection` argument to `AgentSession`, the system automatically applies `DEFAULT_TURN_DETECTION`. As defined in [`livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/utils.py`](https://github.com/livekit/agents/blob/main/livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/utils.py) (lines 28-33), this defaults to:

- `type="semantic_vad"`
- `create_response=True`
- `eagerness="medium"`
- `interrupt_response=True`

To explicitly use defaults:

```python
from livekit.plugins.openai.realtime.utils import DEFAULT_TURN_DETECTION

session = AgentSession(
    turn_detection=DEFAULT_TURN_DETECTION,
    llm=openai.realtime.RealtimeModel(turn_detection=None),
)

```

## Complete Implementation Example

The following example from [`examples/voice_agents/realtime_turn_detector.py`](https://github.com/livekit/agents/blob/main/examples/voice_agents/realtime_turn_detector.py) demonstrates a production-ready setup using the multilingual plugin:

```python
import logging
from dotenv import load_dotenv
from livekit.agents import Agent, AgentServer, AgentSession, JobContext, cli
from livekit.plugins import openai, deepgram, silero
from livekit.plugins.turn_detector.multilingual import MultilingualModel

logger = logging.getLogger("semantic-vad-demo")
load_dotenv()

server = AgentServer()

@server.rtc_session()
async def entrypoint(ctx: JobContext):
    session = AgentSession(
        turn_detection=MultilingualModel(),
        vad=ctx.proc.userdata["vad"],
        stt=deepgram.STT(),
        llm=openai.realtime.RealtimeModel(
            voice="alloy",
            turn_detection=None,  # Critical: disable OpenAI's VAD

            input_audio_transcription=None,
        ),
    )
    await session.start(
        agent=Agent(instructions="You are a helpful assistant."),
        room=ctx.room
    )

def prewarm(proc):
    proc.userdata["vad"] = silero.VAD.load()

server.setup_fnc = prewarm

if __name__ == "__main__":
    cli.run_app(server)

```

**Key implementation details:**
- `turn_detection=MultilingualModel()` injects the semantic detector
- `openai.realtime.RealtimeModel(turn_detection=None)` prevents double VAD processing
- The separate `vad` parameter handles low-latency speech activity detection while the semantic detector handles turn completion

## Configuration Parameters

When using `TurnDetection` objects, these parameters control behavior:

- **`eagerness`**: Controls how aggressively the model predicts end-of-turn. Options are `"low"`, `"medium"`, or `"high"`. Higher values reduce latency but may cause interruptions.
- **`create_response`**: When `True`, automatically triggers LLM response generation upon turn detection.
- **`interrupt_response`**: When `True`, allows new user speech to interrupt ongoing LLM responses.

For remote multilingual inference, configure the endpoint via the `LIVEKIT_REMOTE_EOT_URL` environment variable as referenced in [`multilingual.py`](https://github.com/livekit/agents/blob/main/multilingual.py).

## Summary

- **Semantic turn detection** uses language models rather than audio energy to detect when users finish speaking
- Implement via `MultilingualModel()` for cross-language support or `TurnDetection(type="semantic_vad")` for OpenAI Realtime integration
- Always disable the provider's native VAD (`turn_detection=None`) when using LiveKit's semantic detection to prevent conflicts
- Configure `eagerness`, `create_response`, and `interrupt_response` to balance latency against accuracy
- Core logic resides in [`livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/utils.py`](https://github.com/livekit/agents/blob/main/livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/utils.py) and [`livekit-plugins/livekit-plugins-turn-detector/livekit/plugins/turn_detector/multilingual.py`](https://github.com/livekit/agents/blob/main/livekit-plugins/livekit-plugins-turn-detector/livekit/plugins/turn_detector/multilingual.py)

## Frequently Asked Questions

### How does semantic turn detection differ from server VAD?

**Server VAD** relies on audio energy thresholds and silence detection, while **semantic turn detection** analyzes the transcript content using language models to predict when a user has completed their thought. According to the source code, semantic VAD works even when audio energy is low (whispers) or in noisy environments, and it supports multilingual detection through the `MultilingualModel` class.

### Can I use semantic turn detection with providers other than OpenAI?

Yes. While the `TurnDetection` object targets OpenAI's Realtime API structure, the `MultilingualModel` plugin in `livekit-plugins-turn-detector` provides provider-agnostic semantic detection. You can use `MultilingualModel()` with any STT and LLM combination by passing it to `AgentSession(turn_detection=...)`.

### What happens if I enable both OpenAI's VAD and LiveKit's turn detection?

This creates a conflict where both systems compete to detect turns. The [`examples/voice_agents/realtime_turn_detector.py`](https://github.com/livekit/agents/blob/main/examples/voice_agents/realtime_turn_detector.py) file explicitly sets `turn_detection=None` on the `RealtimeModel` when using `MultilingualModel()`, ensuring only LiveKit's detector manages the conversation flow.

### How do I adjust the sensitivity of turn detection?

For OpenAI's semantic VAD, set the `eagerness` parameter to `"low"`, `"medium"`, or `"high"` in the `TurnDetection` configuration (lines 90-121 of [`utils.py`](https://github.com/livekit/agents/blob/main/utils.py)). For the multilingual model, you can adjust thresholds in the `predict_end_of_turn` method or configure the remote inference service via `LIVEKIT_REMOTE_EOT_URL` for custom probability cutoffs.