How FluidAudio Handles Audio Format Conversion for Inference

FluidAudio normalizes every audio input to 16 kHz, mono, Float32 using the central AudioConverter class that wraps Apple's AVAudioConverter, ensuring consistent model-ready data for ASR, VAD, diarization, and TTS pipelines.

The fluidinference/fluidaudio repository enforces strict audio normalization before any inference occurs. Regardless of source format—whether stereo 48 kHz Int16 PCM or mono 8 kHz files—the pipeline converts inputs to a standardized format consumable by all machine learning models.

The 16 kHz Mono Float32 Target Standard

All inference components in FluidAudio expect audio in 16 kHz sample rate, single-channel mono, and 32-bit floating point (Float32) format. This standard eliminates variability across ASR, VAD, diarization, and TTS modules, allowing models to assume consistent input characteristics.

The conversion guarantee is implemented in Sources/FluidAudio/Shared/AudioConverter.swift, where the AudioConverter type orchestrates three transformations: sample-rate conversion to 16 kHz, channel mixing to mono, and sample-format conversion to Float32.

The AudioConverter Architecture

Format Detection and Validation

Before processing, AudioConverter.isTargetFormat inspects the source AVAudioFormat to determine if conversion is necessary. This method validates sample rate, channel count, sample format, and interleaving status by comparing them against the target specification.

// From Sources/FluidAudio/Shared/AudioConverter.swift#L31-L37
// Checks if the format already matches 16kHz, mono, Float32
if format.sampleRate == targetSampleRate && 
   format.channelCount == 1 && 
   format.commonFormat == .pcmFormatFloat32 {
    return true
}

Channel Mixing Strategy

Multi-channel audio undergoes downmixing to mono. For stereo sources, AudioConverter.extractMonoFloat32 leverages AVAudioConverter to perform the mix. When handling sources with more than two channels, the implementation falls back to linearResample, which applies a manual linear mix to collapse channels into a single mono stream before resampling.

// Fallback linear mix for >2 channels in Sources/FluidAudio/Shared/AudioConverter.swift#L39-L46
private func linearResample(_ buffer: AVAudioPCMBuffer) -> [Float] {
    // Manual channel averaging for multi-channel inputs
    let channelCount = Int(buffer.format.channelCount)
    let frameLength = Int(buffer.frameLength)
    // ... linear mixing implementation
}

Sample-Rate and Sample-Format Conversion

Rather than implementing custom DSP algorithms, FluidAudio delegates resampling to Apple's native AVAudioConverter. The resampleWithAVAudio method constructs a target AVAudioFormat with sampleRate: outputRate (16000) and invokes convertBuffer to handle both frequency conversion and bit-depth transformation (e.g., Int16 → Float32) in a single operation.

According to the source code comments in Sources/FluidAudio/Shared/AudioConverter.swift#L11-L13, this approach "Uses AVAudioConverter for all sample-rate, sample-format, and channel-count conversions," preserving audio quality through Apple's optimized signal processing.

Stateless Design and Thread Safety

Each conversion operation instantiates a fresh AVAudioConverter instance, making the AudioConverter class stateless and thread-safe. As documented in the public initializer comments (Sources/FluidAudio/Shared/AudioConverter.swift#L13-L15), this design allows a single AudioConverter instance to be reused safely across concurrent file processing or streaming pipelines without locking or state corruption.

Usage Examples

File-Based Conversion

For batch processing or CLI workflows, the resampleAudioFile(_:) method reads any file supported by AVAudioFile and returns normalized Float32 samples.

import AVFoundation
import FluidAudio

public func loadSamples16kMono(path: String) async throws -> [Float] {
    let converter = AudioConverter()               // Stateless, can be reused
    return try await converter.resampleAudioFile(path: path)
}

This pattern appears throughout Sources/FluidAudioCLI/Commands/*.swift, where CLI entry points invoke this method before passing data to transcription or analysis models.

Streaming Buffer Conversion

Real-time inference pipelines use resampleBuffer(_:) to process captured audio chunks without file I/O overhead.

import AVFoundation
import FluidAudio

let converter = AudioConverter()

func processChunk(_ pcmBuffer: AVAudioPCMBuffer) async throws {
    let samples = try await converter.resampleBuffer(pcmBuffer) // 16 kHz mono Float32
    // Feed `samples` to ASR/VAD/diarization pipelines
}

The VadManager implementation in Sources/FluidAudio/VAD/VadManager.swift#L44-L60 demonstrates this streaming approach, calling either resampleAudioFile(url) for file inputs or resampleBuffer(audioBuffer) for live stream processing.

Integration with Inference Components

All high-level inference modules—VAD, ASR, diarization, and TTS—consume audio exclusively through the AudioConverter interface. This centralized routing ensures that model inputs never contain unexpected sample rates or channel configurations.

As detailed in Documentation/Guides/AudioConversion.md#L7-L10, the design philosophy emphasizes that conversion uses AVAudioConverter for "sample-rate conversion, sample-format conversion (e.g., Int16 → Float32), and channel mixing (stereo → mono)." This unified preprocessing layer prevents format-related errors from reaching the inference engines.

Summary

• FluidAudio enforces a strict 16 kHz, mono, Float32 standard for all model inputs through the AudioConverter class.
• Format conversion wraps AVAudioConverter for high-quality resampling, channel mixing, and bit-depth conversion without custom DSP code.
• Stateless architecture creates a fresh converter per operation, ensuring thread safety across concurrent file and stream processing.
• Dual API surface supports both file-based (resampleAudioFile) and buffer-based (resampleBuffer) workflows used by VAD, ASR, and CLI commands.
• Multi-channel handling automatically downmixes stereo to mono, with a linear mix fallback for sources exceeding two channels.

Frequently Asked Questions

What target format does FluidAudio require for model inference?

FluidAudio requires all audio inputs to be 16 kHz sample rate, single-channel (mono), and 32-bit floating point (Float32). This standard is enforced by the AudioConverter class before data reaches any ASR, VAD, diarization, or TTS component.

How does FluidAudio handle stereo or multi-channel audio?

Stereo sources are downmixed to mono using AVAudioConverter. For audio with more than two channels, AudioConverter falls back to a manual linear mixing algorithm in linearResample that averages all channels into a single mono stream before resampling to 16 kHz.

Is the AudioConverter thread-safe for concurrent processing?

Yes. AudioConverter is designed as a stateless utility that creates a fresh AVAudioConverter instance for each operation. As noted in Sources/FluidAudio/Shared/AudioConverter.swift#L13-L15, this allows the same converter instance to process multiple files or streams concurrently without synchronization issues.

Why does FluidAudio use AVAudioConverter instead of custom resamplers?

The implementation relies exclusively on Apple's AVAudioConverter to ensure broadcast-quality sample-rate conversion and format transformation. According to Documentation/Guides/AudioConversion.md, this choice avoids hand-rolled resampling algorithms, leveraging Apple's optimized signal processing for consistent, artifact-free conversion across all supported audio formats.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}