How Streaming VAD Handles Speech Segmentation in FluidAudio

FluidAudio’s streaming Voice Activity Detection (VAD) implements a state machine that converts per-chunk probabilities into discrete speech-start and speech-end events using hysteresis thresholds and silence detection.

The fluidinference/fluidaudio repository provides a Swift-based streaming API that processes audio incrementally through a Silero VAD model, enabling real-time speech segmentation without loading entire files into memory. Unlike batch processing methods that analyze complete recordings, this streaming architecture maintains minimal mutable state across 4096-sample chunks to emit speech boundaries with low latency.

The Streaming State Machine Architecture

The core of FluidAudio’s streaming VAD relies on a state machine that persists context across audio chunks. The system tracks speech detection status incrementally using three coordinated components managed by the VadManager actor.

Per-Chunk Processing

Each incoming audio chunk—a Float array of 4096 samples—is fed to the underlying Silero VAD model via processChunk. This method returns a speech probability, an updated recurrent state, and processing timing information. The implementation resides in [VadManager.swift](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/VAD/VadManager.swift#L51-L95).

Streaming State Management

Before processing begins, callers initialize a fresh VadStreamState using makeStreamState(). Defined in [VadTypes.swift](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/VAD/VadTypes.swift#L66-L84), this struct tracks whether speech is currently triggered, maintains a temporary end-sample marker, and counts total samples processed. The factory method appears in VadManager+Streaming.swift.

Converting Probabilities to Speech Events

After chunk processing, processStreamingChunk invokes streamingStateMachine to evaluate the probability against configured thresholds. This function, spanning lines 30-90 in VadManager+Streaming.swift, implements the hysteresis logic that distinguishes genuine speech from background noise.

The state machine emits VadStreamEvent objects containing:

• kind: Either .speechStart or .speechEnd
• sampleIndex: The precise sample where the transition occurred
• time: Optional timestamp in seconds when returnSeconds is enabled

Time Conversion and Event Creation

When the caller requests temporal coordinates, makeStreamEvent in VadManager+Streaming.swift converts sample indices to seconds using the specified timeResolution parameter.

Hysteresis, Thresholds, and Silence Detection

The segmentation logic relies on dual thresholds to prevent rapid state toggling and false boundaries.

Threshold Configuration

• Positive threshold: Defaults to 0.85 via config.defaultThreshold, triggering speech detection when the model probability exceeds this value
• Negative threshold: Calculated dynamically through effectiveNegativeThreshold in [VadTypes.swift](https://github.com/fluidinference/fluidaudio/blob/main/Sources/FluidAudio/VAD/VadTypes.swift#L83-L90), incorporating negativeThreshold and negativeThresholdOffset parameters to customize hysteresis

Silence Detection and Padding

The system postpones emitting speechEnd until the probability remains below the negative threshold for at least minSilenceDuration samples. It then applies speechPadding to extend segment boundaries, ensuring trailing phonemes are not truncated. These parameters are defined in VadSegmentationConfig within VadTypes.swift.

Implementation Examples

Basic Streaming Loop

The following pattern demonstrates continuous processing of audio chunks:

import FluidAudio

// Initialize the manager (async because the model is downloaded/loaded)
let vad = try await VadManager()

// Create a fresh streaming state
var streamState = vad.makeStreamState()

// Process chunks from a microphone or file reader
for chunk in audioChunkSequence {
    // Process with millisecond timestamp resolution
    let result = try await vad.processStreamingChunk(
        chunk,
        state: streamState,
        config: .default,
        returnSeconds: true,
        timeResolution: 3
    )
    
    streamState = result.state
    
    if let ev = result.event {
        switch ev.kind {
        case .speechStart:
            print("Speech started at \(ev.time ?? 0)s")
        case .speechEnd:
            print("Speech ended at \(ev.time ?? 0)s")
        }
    }
}

Customizing Segmentation Behavior

Adjust hysteresis and padding through VadSegmentationConfig:

var cfg = VadSegmentationConfig.default
cfg.speechPadding = 0.2          // 200ms padding
cfg.minSilenceDuration = 0.5     // Require 500ms silence to close
cfg.negativeThreshold = 0.6      // Stricter hysteresis

let result = try await vad.processStreamingChunk(
    audioChunk,
    state: streamState,
    config: cfg,
    returnSeconds: true,
    timeResolution: 2
)

Live Microphone Integration

For real-time capture using AVAudioEngine:

import AVFoundation
import FluidAudio

let audioEngine = AVAudioEngine()
let inputNode = audioEngine.inputNode

var streamState = vad.makeStreamState()

inputNode.installTap(onBus: 0, bufferSize: 4096, format: inputNode.inputFormat(forBus: 0)) {
    buffer, when in
    Task {
        // Convert buffer to Float array (resampled to 16kHz internally)
        let floatChunk = try audioConverter.resampleBuffer(buffer)
        let result = try await vad.processStreamingChunk(
            floatChunk,
            state: streamState,
            returnSeconds: true,
            timeResolution: 3
        )
        streamState = result.state
        // Handle result.event...
    }
}
try audioEngine.start()

Summary

• State machine architecture: FluidAudio uses VadStreamState to track speech triggers and temporary end markers across chunks, avoiding full-file reprocessing
• Dual-threshold hysteresis: The system applies positive (0.85 default) and configurable negative thresholds via effectiveNegativeThreshold to prevent oscillation
• Silence detection: Segments close only after minSilenceDuration of sub-threshold audio, with configurable speechPadding to preserve trailing audio
• Actor-based thread safety: VadManager ensures safe concurrent access to the CoreML model across streaming iterations
• Real-time events: processStreamingChunk returns VadStreamResult containing updated state and optional VadStreamEvent with sample-accurate timestamps

Frequently Asked Questions

What is the optimal chunk size for streaming VAD in FluidAudio?

The implementation expects chunks of 4096 samples when processing at 16kHz, which equals approximately 256 milliseconds of audio. This size balances latency and model performance, as defined in the processChunk implementation within VadManager.swift.

How does the negative threshold differ from the default threshold?

While the default threshold (0.85) triggers speech detection when exceeded, the negative threshold—calculated via effectiveNegativeThreshold in VadTypes.swift—must be maintained for minSilenceDuration before emitting a speechEnd event. This hysteresis prevents brief audio dips from prematurely terminating speech segments.

Can streaming VAD state persist across multiple audio sessions?

Yes. The VadStreamState struct returned by processStreamingChunk can be preserved and passed to subsequent calls, allowing speech segmentation to continue seamlessly across disconnected audio streams or application restarts, provided the VadManager actor instance remains consistent.

How does speech padding affect segmentation accuracy?

The speechPadding parameter adds buffer time to segment boundaries, compensating for the VAD model's inherent reaction latency. According to the configuration in VadSegmentationConfig, typical values range from 0.1 to 0.3 seconds, ensuring initial and final phonemes remain captured without extending silence into adjacent segments.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}