# How Streaming VAD Handles Speech Segmentation in FluidAudio

> Learn how FluidAudio's streaming VAD segments speech using state machines hysteresis and silence detection for accurate real-time audio processing.

- Repository: [Fluid Inference/fluidaudio](https://github.com/fluidinference/fluidaudio)
- Tags: deep-dive
- Published: 2026-03-02

---

**FluidAudio’s streaming Voice Activity Detection (VAD) implements a state machine that converts per-chunk probabilities into discrete speech-start and speech-end events using hysteresis thresholds and silence detection.**

The fluidinference/fluidaudio repository provides a Swift-based streaming API that processes audio incrementally through a Silero VAD model, enabling real-time speech segmentation without loading entire files into memory. Unlike batch processing methods that analyze complete recordings, this streaming architecture maintains minimal mutable state across 4096-sample chunks to emit speech boundaries with low latency.

## The Streaming State Machine Architecture

The core of FluidAudio’s streaming VAD relies on a state machine that persists context across audio chunks. The system tracks speech detection status incrementally using three coordinated components managed by the `VadManager` actor.

### Per-Chunk Processing

Each incoming audio chunk—a `Float` array of 4096 samples—is fed to the underlying Silero VAD model via `processChunk`. This method returns a speech probability, an updated recurrent state, and processing timing information. The implementation resides in [[`VadManager.swift`](https://github.com/fluidinference/fluidaudio/blob/main/VadManager.swift)](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/VAD/VadManager.swift#L51-L95).

### Streaming State Management

Before processing begins, callers initialize a fresh `VadStreamState` using `makeStreamState()`. Defined in [[`VadTypes.swift`](https://github.com/fluidinference/fluidaudio/blob/main/VadTypes.swift)](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/VAD/VadTypes.swift#L66-L84), this struct tracks whether speech is currently triggered, maintains a temporary end-sample marker, and counts total samples processed. The factory method appears in [`VadManager+Streaming.swift`](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/VAD/VadManager+Streaming.swift#L5-L8).

## Converting Probabilities to Speech Events

After chunk processing, `processStreamingChunk` invokes `streamingStateMachine` to evaluate the probability against configured thresholds. This function, spanning lines 30-90 in [`VadManager+Streaming.swift`](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/VAD/VadManager+Streaming.swift#L30-L90), implements the hysteresis logic that distinguishes genuine speech from background noise.

The state machine emits `VadStreamEvent` objects containing:
- `kind`: Either `.speechStart` or `.speechEnd`
- `sampleIndex`: The precise sample where the transition occurred  
- `time`: Optional timestamp in seconds when `returnSeconds` is enabled

### Time Conversion and Event Creation

When the caller requests temporal coordinates, `makeStreamEvent` in [`VadManager+Streaming.swift`](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/VAD/VadManager+Streaming.swift#L93-L106) converts sample indices to seconds using the specified `timeResolution` parameter.

## Hysteresis, Thresholds, and Silence Detection

The segmentation logic relies on dual thresholds to prevent rapid state toggling and false boundaries.

### Threshold Configuration

- **Positive threshold**: Defaults to `0.85` via `config.defaultThreshold`, triggering speech detection when the model probability exceeds this value
- **Negative threshold**: Calculated dynamically through `effectiveNegativeThreshold` in [[`VadTypes.swift`](https://github.com/fluidinference/fluidaudio/blob/main/VadTypes.swift)](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/VAD/VadTypes.swift#L83-L90), incorporating `negativeThreshold` and `negativeThresholdOffset` parameters to customize hysteresis

### Silence Detection and Padding

The system postpones emitting `speechEnd` until the probability remains below the negative threshold for at least `minSilenceDuration` samples. It then applies `speechPadding` to extend segment boundaries, ensuring trailing phonemes are not truncated. These parameters are defined in [`VadSegmentationConfig`](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/VAD/VadTypes.swift#L24-L48) within [`VadTypes.swift`](https://github.com/fluidinference/fluidaudio/blob/main/VadTypes.swift).

## Implementation Examples

### Basic Streaming Loop

The following pattern demonstrates continuous processing of audio chunks:

```swift
import FluidAudio

// Initialize the manager (async because the model is downloaded/loaded)
let vad = try await VadManager()

// Create a fresh streaming state
var streamState = vad.makeStreamState()

// Process chunks from a microphone or file reader
for chunk in audioChunkSequence {
    // Process with millisecond timestamp resolution
    let result = try await vad.processStreamingChunk(
        chunk,
        state: streamState,
        config: .default,
        returnSeconds: true,
        timeResolution: 3
    )
    
    streamState = result.state
    
    if let ev = result.event {
        switch ev.kind {
        case .speechStart:
            print("Speech started at \(ev.time ?? 0)s")
        case .speechEnd:
            print("Speech ended at \(ev.time ?? 0)s")
        }
    }
}

```

### Customizing Segmentation Behavior

Adjust hysteresis and padding through `VadSegmentationConfig`:

```swift
var cfg = VadSegmentationConfig.default
cfg.speechPadding = 0.2          // 200ms padding
cfg.minSilenceDuration = 0.5     // Require 500ms silence to close
cfg.negativeThreshold = 0.6      // Stricter hysteresis

let result = try await vad.processStreamingChunk(
    audioChunk,
    state: streamState,
    config: cfg,
    returnSeconds: true,
    timeResolution: 2
)

```

### Live Microphone Integration

For real-time capture using AVAudioEngine:

```swift
import AVFoundation
import FluidAudio

let audioEngine = AVAudioEngine()
let inputNode = audioEngine.inputNode

var streamState = vad.makeStreamState()

inputNode.installTap(onBus: 0, bufferSize: 4096, format: inputNode.inputFormat(forBus: 0)) {
    buffer, when in
    Task {
        // Convert buffer to Float array (resampled to 16kHz internally)
        let floatChunk = try audioConverter.resampleBuffer(buffer)
        let result = try await vad.processStreamingChunk(
            floatChunk,
            state: streamState,
            returnSeconds: true,
            timeResolution: 3
        )
        streamState = result.state
        // Handle result.event...
    }
}
try audioEngine.start()

```

## Summary

- **State machine architecture**: FluidAudio uses `VadStreamState` to track speech triggers and temporary end markers across chunks, avoiding full-file reprocessing
- **Dual-threshold hysteresis**: The system applies positive (0.85 default) and configurable negative thresholds via `effectiveNegativeThreshold` to prevent oscillation
- **Silence detection**: Segments close only after `minSilenceDuration` of sub-threshold audio, with configurable `speechPadding` to preserve trailing audio
- **Actor-based thread safety**: `VadManager` ensures safe concurrent access to the CoreML model across streaming iterations
- **Real-time events**: `processStreamingChunk` returns `VadStreamResult` containing updated state and optional `VadStreamEvent` with sample-accurate timestamps

## Frequently Asked Questions

### What is the optimal chunk size for streaming VAD in FluidAudio?

The implementation expects chunks of 4096 samples when processing at 16kHz, which equals approximately 256 milliseconds of audio. This size balances latency and model performance, as defined in the `processChunk` implementation within [`VadManager.swift`](https://github.com/fluidinference/fluidaudio/blob/main/VadManager.swift).

### How does the negative threshold differ from the default threshold?

While the default threshold (0.85) triggers speech detection when exceeded, the negative threshold—calculated via `effectiveNegativeThreshold` in [`VadTypes.swift`](https://github.com/fluidinference/fluidaudio/blob/main/VadTypes.swift)—must be maintained for `minSilenceDuration` before emitting a `speechEnd` event. This hysteresis prevents brief audio dips from prematurely terminating speech segments.

### Can streaming VAD state persist across multiple audio sessions?

Yes. The `VadStreamState` struct returned by `processStreamingChunk` can be preserved and passed to subsequent calls, allowing speech segmentation to continue seamlessly across disconnected audio streams or application restarts, provided the `VadManager` actor instance remains consistent.

### How does speech padding affect segmentation accuracy?

The `speechPadding` parameter adds buffer time to segment boundaries, compensating for the VAD model's inherent reaction latency. According to the configuration in `VadSegmentationConfig`, typical values range from 0.1 to 0.3 seconds, ensuring initial and final phonemes remain captured without extending silence into adjacent segments.