How to Implement Custom AVFoundation Video Compositors for Effect Chaining in Palmier Pro

To implement custom AVFoundation video compositors in Palmier Pro, subclass NSObject and conform to AVVideoCompositing—using CustomVideoCompositor as your render engine—then chain visual effects by populating the effects array inside CompositorInstruction objects that drive the frame processing pipeline.

Palmier Pro extends AVFoundation's native rendering capabilities through a modular compositing architecture defined in the palmier-io/palmier-pro repository. The system centers on a custom AVFoundation video compositor implemented in Sources/PalmierPro/Compositing/CustomVideoCompositor.swift, which processes pixel buffers while executing ordered sequences of VideoEffect objects defined in CompositorInstruction structs. This design enables you to construct reusable, chainable visual effects that integrate seamlessly with standard AVPlayer and export workflows.

CustomVideoCompositor Implementation Details

The CustomVideoCompositor class serves as the primary entry point for custom rendering. Defined in Sources/PalmierPro/Compositing/CustomVideoCompositor.swift, it adopts the AVVideoCompositing protocol and manages the lifecycle of composition requests.

Class Structure and Protocol Conformance

The class declaration includes conformance to @unchecked Sendable for thread safety across the render queue:

final class CustomVideoCompositor: NSObject, AVVideoCompositing, @unchecked Sendable {
    var videoComposition: AVVideoComposition!
    var sourceTrackIDForFrameTiming: CMPersistentTrackID = kCMPersistentTrackID_Invalid
    var sourceSampleDataTrackIDs: [CMPersistentTrackID] = []
    
    private var renderContext: AVVideoCompositionRenderContext!
    private var instructionQueue: [AVVideoCompositionInstruction] = []
    private var cancelled = false
}

Key properties include:

• videoComposition – References the driving composition configuration.
• sourceTrackIDForFrameTiming – Identifies the track providing timing metadata.
• sourceSampleDataTrackIDs – Lists auxiliary sample data tracks available during processing.

Handling Render Context Updates

AVFoundation calls renderContextChanged(_:) whenever the output dimensions or pixel format changes. Store this context to configure pixel buffer pools or Metal textures for your effects:

func renderContextChanged(_ newRenderContext: AVVideoCompositionRenderContext) {
    renderContext = newRenderContext
}

Processing Requests and Managing State

The startRequest(_:) method receives AVVideoCompositionRequest objects and manages an internal instruction queue:

func startRequest(_ request: AVVideoCompositionRequest) {
    instructionQueue.append(request.videoCompositionInstruction)
    processNext()
}

The private processNext() method retrieves the next instruction and iterates through required source tracks. In the source implementation, this method demonstrates where you would execute your effect chain:

private func processNext() {
    guard !cancelled, let instruction = instructionQueue.first else { return }
    
    for trackID in instruction.requiredSourceTrackIDs ?? [] {
        let trackIDValue = trackID.int32Value
        // Retrieve source buffers and apply effects...
    }
    
    instructionQueue.removeFirst()
}

To halt rendering, the class implements cancelAllRequests():

func cancelAllRequests() {
    cancelled = true
    instructionQueue.removeAll()
}

Effect Chaining with CompositorInstruction

The CompositorInstruction struct, located in Sources/PalmierPro/Compositing/CompositorInstruction.swift, bridges the gap between high-level editing timelines and low-level pixel processing.

Instruction Structure

This struct conforms to AVVideoCompositionInstruction and stores both standard composition metadata and a custom effects array:

struct CompositorInstruction: AVVideoCompositionInstruction {
    var timeRange: CMTimeRange
    var enablePostProcessing: Bool = false
    var requiredSourceTrackIDs: [NSValue]? // Track IDs needed for this segment
    var containsTweening: Bool = false
    
    // Custom properties for effect chaining
    var effects: [VideoEffect] = []
    
    init(timeRange: CMTimeRange, effects: [VideoEffect] = []) {
        self.timeRange = timeRange
        self.effects = effects
    }
}

The effects array determines the sequential order of transformations applied to each frame, enabling effect chaining where the output of one filter becomes the input for the next.

Defining the VideoEffect Protocol

Since CompositorInstruction stores [VideoEffect], you must define a protocol that standardizes how individual effects transform pixel data. Each implementation receives a CVPixelBuffer and the render context, then returns a modified buffer:

import CoreVideo

protocol VideoEffect {
    /// Processes the input buffer and returns a transformed buffer.
    /// - Parameters:
    ///   - buffer: The source CVPixelBuffer to process.
    ///   - context: The AVVideoCompositionRenderContext providing size and timing data.
    /// - Returns: A new or recycled CVPixelBuffer containing the processed image, or nil on failure.
    func apply(to buffer: CVPixelBuffer, context: AVVideoCompositionRenderContext) -> CVPixelBuffer?
}

Concrete Effect Implementation

Create concrete types conforming to this protocol. For example, a Core Image-based color adjustment:

import CoreImage

struct ColorGradingEffect: VideoEffect {
    let saturation: Double
    let brightness: Double
    
    func apply(to buffer: CVPixelBuffer, context: AVVideoCompositionRenderContext) -> CVPixelBuffer? {
        let ciImage = CIImage(cvPixelBuffer: buffer)
        let filtered = ciImage
            .applyingFilter("CIColorControls", parameters: [
                "inputSaturation": saturation,
                "inputBrightness": brightness
            ])
        
        // Create output buffer matching render context dimensions
        var outputBuffer: CVPixelBuffer?
        let pixelBufferFormat = CVPixelBufferGetPixelFormatType(buffer)
        CVPixelBufferCreate(kCFAllocatorDefault,
                           Int(context.size.width),
                           Int(context.size.height),
                           pixelBufferFormat,
                           nil,
                           &outputBuffer)
        
        guard let output = outputBuffer else { return nil }
        
        let ciContext = CIContext(options: nil)
        ciContext.render(filtered, to: output)
        return output
    }
}

Integrating Text Overlays

CustomVideoCompositor provides hooks for combining pixel-based effects with vector-based overlays. The class exposes an optional CALayer property for text or graphics:

var textOverlayLayer: CALayer?

func setTextOverlay(_ layer: CALayer) {
    textOverlayLayer = layer
}

During frame processing, you can composite this layer onto the final buffer after executing the VideoEffect chain, facilitating workflows such as color grading followed by subtitle rendering.

Complete Integration Example

To assemble a complete custom AVFoundation video compositor workflow:

import AVFoundation

// 1. Define effect chain
let effectChain: [VideoEffect] = [
    ColorGradingEffect(saturation: 1.2, brightness: 0.1),
    // Additional effects...
]

// 2. Create instruction for 5-second duration
let instruction = CompositorInstruction(
    timeRange: CMTimeRange(start: .zero, duration: CMTime(seconds: 5, preferredTimescale: 600)),
    effects: effectChain
)

// 3. Configure video composition
let composition = AVMutableVideoComposition()
composition.renderSize = CGSize(width: 1920, height: 1080)
composition.frameDuration = CMTime(value: 1, timescale: 30)
composition.customVideoCompositorClass = CustomVideoCompositor.self
composition.instructions = [instruction]

// 4. Apply to player or export session
let playerItem = AVPlayerItem(asset: videoAsset)
playerItem.videoComposition = composition

Summary

• CustomVideoCompositor in Sources/PalmierPro/Compositing/CustomVideoCompositor.swift implements the AVVideoCompositing protocol and manages the render queue via startRequest(_:) and processNext().
• CompositorInstruction in Sources/PalmierPro/Compositing/CompositorInstruction.swift conforms to AVVideoCompositionInstruction and stores an ordered effects array for declarative effect chaining across specific time ranges.
• The VideoEffect protocol defines the transformation interface, accepting CVPixelBuffer and AVVideoCompositionRenderContext to produce modified frames.
• Effect chaining executes sequentially in the order defined in the effects array, with each effect receiving the buffer output from the previous stage.
• Text overlay integration is supported through the textOverlayLayer property and setTextOverlay(_:) method, enabling hybrid pixel and vector rendering.

Frequently Asked Questions

How does CustomVideoCompositor handle asynchronous rendering requests?

CustomVideoCompositor processes requests synchronously within startRequest(_:) in the provided implementation, immediately invoking processNext(). To implement asynchronous processing, dispatch pixel buffer operations to a background queue, then call request.finishWithComposedVideoFrame() upon completion. Ensure thread-safe access to the instructionQueue using the @unchecked Sendable conformance and appropriate locking mechanisms.

What is the purpose of the requiredSourceTrackIDs property in CompositorInstruction?

The requiredSourceTrackIDs property—as defined in the AVVideoCompositionInstruction protocol implementation within CompositorInstruction—specifies which video track IDs the compositor requires data from for a given time range. CustomVideoCompositor uses these identifiers to retrieve source frames via request.sourceFrame(byTrackID:), ensuring the compositor only accesses relevant media tracks during processing.

Can Metal shaders be integrated into the VideoEffect chain?

Yes. Since the VideoEffect protocol operates on CVPixelBuffer inputs and outputs, you can implement effects using Metal compute shaders by creating MTLTexture objects from the pixel buffer, executing your shader kernel, then returning the resulting buffer. This approach mixes seamlessly with Core Image-based effects in the same CompositorInstruction.effects array.

How do I integrate text overlays with pixel-based effects?

Set the textOverlayLayer property on your CustomVideoCompositor instance using setTextOverlay(_:). During frame processing inside processNext(), render the CALayer onto the final pixel buffer after executing the VideoEffect chain—typically by creating a Core Image or Metal representation of the layer and compositing it over the processed image.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}