Building Frame-Accurate Playback Scrubbing with Audio Output in Palmier Pro

Palmier Pro achieves frame-accurate scrubbing by coupling a UI-driven playhead with a dedicated audio pipeline that decodes short PCM windows, caches them in an LRU buffer, and plays 50ms audio grains on demand while maintaining main-thread responsiveness.

Frame-accurate playback scrubbing allows video editors to audition audio at precise frames while dragging the playhead. In the palmier-io/palmier-pro codebase, this feature is implemented through a modular pipeline that separates UI gesture handling from audio decoding and playback. The architecture ensures that heavy audio processing occurs off the main thread, preventing UI lag during rapid scrubbing operations.

Architecture of the Scrubbing Pipeline

The scrubbing system spans three primary layers: gesture interpretation in the timeline, audio window management in the preview engine, and low-latency playback output.

TimelineInputController: Translating Gestures to Frame Requests

User interactions begin in Sources/PalmierPro/Timeline/TimelineInputController.swift. When a user initiates a drag on the playhead, the controller pauses active playback and enters scrubbing mode.

The entry point beginPlayheadScrub(at:) sets the editor’s isScrubbing flag and immediately seeks to the starting frame:

func beginPlayheadScrub(at frame: Int) {
    stopPlayheadAutoScroll()
    dragState = .scrubPlayhead
    scrubWasPlaying = editor.isPlaying
    if scrubWasPlaying { editor.pause() }
    editor.isScrubbing = true               // 👈 marks the editor as scrubbing
    scrubToFrame(frame)                     // 👉 calls editor.seekToFrame(..., .interactiveScrub)
    view.updatePlayheadLayer()
}

During dragging, continuePlayheadScrub(windowPoint:) converts mouse coordinates to timeline frames and forwards them to scrubToFrame(_:), which triggers the audio engine via the interactiveScrub mode.

ScrubAudioEngine: Decoding and Window Management

Located in Sources/PalmierPro/Preview/ScrubAudioEngine.swift, this class handles the heavy lifting of converting CMTime positions into playable audio grains. It maintains an LRU-style cache of up to 256 PCM windows to minimize decoding overhead.

The core method scrub(to:movingForward:) translates the requested time into a sample index, determines playback direction, and either serves a cached window or requests a new decode:

func scrub(to time: CMTime, movingForward: Bool? = nil) {
    guard let source, time.isValid else { return }
    let sample = Int64((time.seconds * Self.sampleRate).rounded())
    guard sample != lastRequestedSample else { return }

    // Determine direction (forward / reverse)
    let direction: Direction = … 

    // Store request and try to serve a cached window
    let request = Request(sample: sample, direction: direction)
    latestRequest = request
    if let window = serveableWindow(for: sample) {
        play(request: request, from: window)               // 🎧 audio grain emitted
        prefetchIfNeeded(sample: sample, direction: direction,
                         from: window, source: source)   // 👈 keep cache warm
    } else {
        requestWindow(around: sample, direction: direction, source: source)
    }
}

Grain generation occurs in makeGrain, which produces a 2,400-frame (approximately 50ms) stereo buffer centered on the requested sample, supporting both forward and reverse directions:

private func makeGrain(request: Request, from window: PCMWindow) -> ScrubAudioGrain {
    let frameCount = Self.grainFrameCount
    var left  = [Float](repeating: 0, count: frameCount)
    var right = [Float](repeating: 0, count: frameCount)

    let halfGrain = Int64(frameCount / 2)
    for outputIndex in 0..<frameCount {
        let sourceSample: Int64 = switch request.direction {
        case .forward:
            request.sample - halfGrain + Int64(outputIndex)
        case .reverse:
            request.sample + halfGrain - 1 - Int64(outputIndex)
        }
        // copy the Int16 samples from the window into Float buffers…
    }
    return ScrubAudioGrain(left: left, right: right)
}

ScrubAudioOutput: Lock-Protected Playback

The ScrubAudioOutput class in Sources/PalmierPro/Preview/ScrubAudioOutput.swift manages an AVAudioEngine graph protected by OSAllocatedUnfairLock. It receives grains via play(_:) and handles silence when tracks lack audio, stopping immediately when stop() is called at scrub session end.

Prefetching and Cache Management

To prevent audible dropouts during rapid scrubbing, the engine implements aggressive prefetching.

Bidirectional Fill Strategy

Upon configuration, startFill(from:source:) launches two concurrent tasks: one fills the cache from the current anchor to the asset’s end, while the other fills from the start back to the anchor. This ensures that jumping to arbitrary positions likely hits cached data.

Edge Prefetching

When the playhead approaches a window boundary (within prefetchMarginFrameCount), prefetchIfNeeded triggers a decode for the next adjacent window:

private func prefetchIfNeeded(sample: Int64, direction: Direction,
                              from window: PCMWindow, source: Source) {
    guard decodeTask == nil else { return }
    let nearEdge = direction == .forward
        ? sample + Self.prefetchMarginFrameCount >= window.endSample
        : sample - Self.prefetchMarginFrameCount <= window.startSample
    guard nearEdge else { return }

    // Jump ahead/back by one cache window and request it
    let next = direction == .forward ? sample + Self.cacheFrameCount - Self.prefetchMarginFrameCount
                                    : sample - Self.cacheFrameCount + Self.prefetchMarginFrameCount
    requestWindow(around: max(0, next), direction: direction, source: source)
}

Mix Invalidation Handling

When users modify audio mixes without changing the underlying asset, scheduleMixInvalidation debounces the change for 250ms, then clears the cache and restarts the fill from the last sample position. This avoids expensive full-asset re-decoding while ensuring effect changes are audible immediately.

State Cleanup and UI Feedback

Scrubbing concludes when stopScrubbing() resets the decode tasks and stops audio output:

func stopScrubbing() {
    resetScrubState()          // cancel decode tasks, clear pending requests
    output.stop()              // halt audio playback
}

During scrubbing, real-time audio levels are analyzed via AudioLevelAnalyzer.analyze and pushed to AudioMeterHub, which provides visual feedback in the waveform overlay and inspector panels.

Summary

• Frame-accurate scrubbing in Palmier Pro relies on a dedicated ScrubAudioEngine that translates UI frame requests into sample-accurate audio grains.
• LRU caching of 256 PCM windows ensures low-latency retrieval, while bidirectional background filling keeps the cache warm for jumps.
• 50ms grains (2,400 frames) provide audible feedback without requiring full decode of the surrounding timeline.
• Lock-protected output via ScrubAudioOutput guarantees thread-safe audio playback on the main audio thread.
• Mix invalidation with debouncing allows real-time effect changes without re-decoding the entire asset.

Frequently Asked Questions

How does Palmier Pro maintain audio sync while scrubbing rapidly?

The engine uses an LRU cache of decoded PCM windows combined with aggressive prefetching. When the playhead nears a cache boundary, prefetchIfNeeded requests the next window before the current grain finishes playing. According to the palmier-io/palmier-pro source code, this window-based approach ensures that even rapid directional changes serve audio from pre-decoded buffers rather than hitting the disk.

What is the latency of the scrubbing audio output?

Each scrub grain represents approximately 50ms of audio (2,400 frames at 48kHz). The ScrubAudioOutput class submits these grains immediately to an AVAudioEngine running on the main audio thread, resulting in sub-100ms end-to-end latency from gesture to audible output.

Why does the audio stop when I release the playhead?

The TimelineInputController calls stopScrubbing() on gesture end, which invokes output.stop() and resets the scrub state machine. This design prevents audio continuation past the intended frame and clears the decode tasks to free CPU resources for regular playback or other editing operations.

How are audio level meters updated during scrubbing?

The AudioMeterHub receives real-time analysis data from AudioLevelAnalyzer.analyze, which processes each grain before it reaches ScrubAudioOutput.play(_:). This allows UI components like the waveform overlay to display accurate level meters even during rapid scrubbing operations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}