ThreadSafeAudioBuffer in FluidVoice: Safe Concurrent Audio Processing for Real-Time Speech Recognition

ThreadSafeAudioBuffer is a Swift utility class that wraps a mutable float array with an NSLock to serialize access, enabling safe concurrent writes from a real-time audio thread and reads from the main actor without data races or memory corruption.

FluidVoice (altic-dev/FluidVoice) performs real-time audio capture on a low-latency background thread while running the Automatic Speech Recognition (ASR) service on the main actor (@MainActor). Directly sharing a standard Array<Float> between these contexts would trigger "Array mutation while enumerating" crashes and memory corruption, so the project implements ThreadSafeAudioBuffer to bridge the gap with deterministic thread safety.

Why Concurrent Audio Processing Needs Thread Safety

Real-time audio capture introduces a fundamental concurrency problem. The AVAudioEngine tap runs on a dedicated background thread to minimize latency, while UI updates and transcription logic must execute on @MainActor. Sharing a mutable array between these threads creates data races when the audio thread appends new samples simultaneously with the main thread reading or clearing the buffer. Without synchronization, simultaneous mutations corrupt memory and crash the application.

ThreadSafeAudioBuffer Implementation and API

Located in Sources/Fluid/Services/ThreadSafeAudioBuffer.swift, the class implements a thin wrapper around a private [Float] storage. Every public operation acquires a single NSLock before mutating or reading the buffer, ensuring that producers and consumers never operate on the data simultaneously.

Thread Safety Mechanism

The buffer uses a lightweight, non-reentrant NSLock instance. This lock is sufficient for the relatively short critical sections (array appends, copies, or clears) and avoids the overhead of higher-level concurrency primitives. Because the lock is released immediately after each operation, it prevents priority inversion that could disrupt real-time audio processing.

Public Methods

• append(_:): Locks the buffer, appends new samples via buffer.append(contentsOf:), then unlocks. This method is called by the audio tap to push converted mono-16kHz samples.
• clear(keepingCapacity:): Locks, invokes buffer.removeAll(keepingCapacity:), and unlocks. This resets the buffer between recording sessions without reallocating memory, optimizing performance for repeated recordings.
• count: A computed property that locks, reads the current array count, and unlocks. The ASR service uses this to check if sufficient samples exist for processing a streaming chunk.
• getPrefix(_:): Locks, copies the first N samples via Array(buffer[0..<safeLength]), and unlocks. Returns an immutable snapshot for streaming transcription while the audio thread continues appending new data.
• getAll(): Locks, copies the entire array, and unlocks. Retrieves the complete PCM buffer for final transcription after recording stops.

How ThreadSafeAudioBuffer Fits into the ASR Pipeline

The buffer is owned by ASRService and injected into AudioCapturePipeline, which executes on the unmanaged audio thread. This architecture allows the system to ingest raw audio on a high-priority thread while processing transcription on the main actor.

Writing from the Audio Thread

Inside AudioCapturePipeline.handle(buffer:) in Sources/Fluid/Services/ASRService.swift, the pipeline converts incoming multi-channel audio to mono 16kHz format and safely appends it:

let mono16k = Self.toMono16k(floatBuffer: buffer)
self.audioBuffer.append(mono16k)            // Thread-safe write
let level = self.calculateAudioLevel(mono16k)
self.onLevel(level)                       // UI update on main queue

Reading from the Main Actor

On @MainActor, ASRService.processStreamingChunk() reads snapshots without blocking the audio thread for more than the lock duration:

let currentSampleCount = self.audioBuffer.count               // Thread-safe read
guard currentSampleCount >= minSamples else { return }
let chunk = self.audioBuffer.getPrefix(currentSampleCount)   // Thread-safe copy
let result = try await transcriptionExecutor.run {
    try await provider.transcribeStreaming(chunk)
}

Final Transcription and Cleanup

When recording stops, the service retrieves all accumulated data and clears the buffer for the next session:

var pcm = self.audioBuffer.getAll()   // Thread-safe snapshot
self.audioBuffer.clear()              // Reset for next session
// pcm is handed to the selected transcription provider

Practical Example: Capturing Audio Snippets

The following pattern demonstrates checking sample counts, retrieving specific durations, and efficiently resetting the buffer:

import Fluid

func captureOneSecondSnippet(service: ASRService) async -> [Float] {
    guard service.isRunning else { return [] }
    
    // Wait for 16,000 samples (1 second @ 16kHz)
    while service.audioBuffer.count < 16_000 {
        try? await Task.sleep(nanoseconds: 10_000_000)   // 10ms
    }
    
    let oneSecond = service.audioBuffer.getPrefix(16_000)
    service.audioBuffer.clear(keepingCapacity: true)      // Preserve capacity
    return oneSecond
}

Summary

• ThreadSafeAudioBuffer prevents data races between real-time audio threads and the main actor by serializing every access with an NSLock.
• Immutable snapshots from getPrefix(_:) and getAll() ensure the transcription provider works on stable data while the audio tap continues appending.
• Efficient reuse via clear(keepingCapacity:) eliminates memory reallocation during rapid record/stop cycles.
• The implementation in Sources/Fluid/Services/ThreadSafeAudioBuffer.swift serves as the cornerstone of FluidVoice's safe concurrent audio architecture.

Frequently Asked Questions

Why does ThreadSafeAudioBuffer use NSLock instead of Swift actors?

ThreadSafeAudioBuffer must bridge Objective-C audio callbacks (AVAudioEngine taps) with Swift's structured concurrency. Because audio taps run on unmanaged threads outside of Swift's actor system, a low-level NSLock provides the necessary synchronization without requiring actor isolation that would conflict with the real-time audio context.

Does ThreadSafeAudioBuffer block the audio thread?

The lock is held only for the duration of the array operation—typically microseconds for appending or copying small chunks. According to the FluidVoice source code, these brief critical sections prevent priority inversion and maintain the low-latency requirements of real-time audio capture.

How does ThreadSafeAudioBuffer handle memory allocation?

The clear(keepingCapacity: true) method removes all elements while preserving the underlying storage capacity. This optimization prevents repeated memory allocations during rapid record/stop cycles, reducing GC pressure and improving performance for repeated recordings.

Can ThreadSafeAudioBuffer be used with other audio formats?

While the implementation stores [Float] samples, the class is format-agnostic. In FluidVoice, AudioCapturePipeline converts audio to mono 16kHz PCM before calling append(_:), but the buffer itself accepts any float array, making it reusable for different audio processing pipelines.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}