GPU Accelerated Video Effects in Swift: Implementing Metal-Based Processing with Palmier Pro

Palmier Pro achieves real-time GPU accelerated video effects in Swift by compiling Metal kernels into Core Image filters and orchestrating them through a custom AVFoundation compositor that processes frames directly on the GPU.

Palmier Pro is an open-source Swift framework that demonstrates production-grade GPU accelerated video effects in Swift by bridging low-level Metal shaders with high-level Core Image and AVFoundation APIs. The architecture leverages a custom build tool to compile Metal sources, a dynamic kernel loader to bridge Metal and Core Image, and a Metal-aware compositor that maintains pixel data on the GPU throughout the entire rendering pipeline.

Architectural Overview: From Metal Shaders to Rendered Frames

The palmier-io/palmier-pro repository implements a four-layer architecture for GPU video processing:

1. Metal Source Files (*.metal) containing per-pixel shader functions for effects like color wheels, glow, and grain.
2. CIKernelLoader – A lightweight utility that loads compiled .metallib bundles and exposes them as CIColorKernel objects.
3. EffectRegistry – A central registry that maps effect identifiers to their parameter schemas and kernel applications, resolving UI parameters into ResolvedEffectParams structs for each frame.
4. CustomVideoCompositor – An AVVideoCompositing implementation that connects AVFoundation's video playback and export pipelines to the Core Image processing chain.

This design ensures that video frames remain in GPU memory from capture through effects processing to final output, eliminating expensive CPU-GPU transfers.

Compiling Metal Kernels for Core Image

The framework uses a custom Swift Package Manager build tool plugin to transform raw Metal source files into Core Image-compatible libraries at compile time. The MetalCIKernelPlugin (located in Plugins/MetalCIKernelPlugin/MetalCIKernelPlugin.swift) automatically processes all .metal files in the Metal/ directory.

The plugin executes the following build commands during compilation:

let air = context.pluginWorkDirectoryURL.appending(path: "\(stem).air")
let metallib = context.pluginWorkDirectoryURL.appending(path: "\(stem).metallib")
// Command sequence:
"xcrun metal -c -fcikernel '\(metal.path())' -o '\(air.path())' && " +
"xcrun metallib -cikernel '\(air.path())' -o '\(metallib.path())'"

This generates .metallib bundles that Core Image can load directly as CIKernel objects, embedding the compiled GPU code as bundle resources.

Loading GPU Kernels with CIKernelLoader

At runtime, the CIKernelLoader class (Sources/PalmierPro/Compositing/Kernels/CIKernelLoader.swift) bridges the compiled Metal libraries with Core Image. It searches the bundle for .metallib files and instantiates kernel objects using Metal library data:

static func colorKernel(_ lib: String, _ function: String) -> CIColorKernel? {
    data(lib).flatMap { try? CIColorKernel(functionName: function, fromMetalLibraryData: $0) }
}

Effect implementations wrap these kernels in type-safe enums. For example, the color wheels effect in Sources/PalmierPro/Compositing/Kernels/WheelsKernel.swift loads the compiled "Wheels" kernel:

enum WheelsKernel {
    private static let kernel = CIKernelLoader.colorKernel("Wheels", "wheels")

    static func apply(_ image: CIImage, params p: ResolvedEffectParams) -> CIImage {
        guard let kernel, !ColorWheels.isNeutral(p) else { return image }
        let c = ColorWheels.coefficients(for: p)
        func vec(_ v: SIMD3<Float>) -> CIVector { CIVector(x: CGFloat(v.x), y: CGFloat(v.y), z: CGFloat(v.z)) }
        return kernel.apply(extent: image.extent,
                            arguments: [image, vec(c.lift), vec(c.gain), vec(c.invGamma)]) ?? image
    }
}

Registering Effects with EffectRegistry

All video effects are declaratively registered in Sources/PalmierPro/Compositing/EffectRegistry.swift using EffectDescriptor instances. Each descriptor defines the effect's identity, parameter specifications, and the closure that applies the kernel:

EffectDescriptor(
    id: "color.wheels", displayName: "Color Wheels", category: "Color",
    params: [
        EffectParamSpec(key: "lift_x", …),
        // Additional parameters for lift, gain, gamma...
    ],
    apply: { image, p, _ in WheelsKernel.apply(image, params: p) }
)

When rendering, the registry resolves dynamic UI parameters into ResolvedEffectParams and handles color space conversions. The render pipeline applies linearization before effects processing and re-applies the sRGB tone curve afterward:

let params = resolve(effect, atOffset: offset)
var working = image
if linearizes { working = working.applyingFilter("CISRGBToneCurveToLinear") }
working = apply(working, params, extent)
if linearizes { working = working.applyingFilter("CILinearToSRGBToneCurve") }

Real-Time Compositing with CustomVideoCompositor

The CustomVideoCompositor class (Sources/PalmierPro/Compositing/CustomVideoCompositor.swift) implements the AVVideoCompositing protocol to integrate with AVFoundation's AVPlayerItem and export sessions. It maintains a shared CIContext configured for GPU processing with color management disabled to preserve raw pixel data:

static let ciContext: CIContext = {
    let options: [CIContextOption: Any] = [
        .workingColorSpace: NSNull(),
        .outputColorSpace: NSNull(),
        .cacheIntermediates: true,
    ]
    if let device = MTLCreateSystemDefaultDevice() {
        return CIContext(mtlDevice: device, options: options)
    }
    return CIContext(options: options)
}()

During each frame request, the compositor obtains a CompositorInstruction, pulls source frames as CIImage objects, executes FrameRenderer.render to iterate through enabled effects, and writes the result into a Metal-compatible pixel buffer.

Practical Implementation Examples

Applying a Color Wheels Effect Directly

You can apply individual GPU effects without the full compositor by resolving parameters and calling the kernel wrapper directly:

import PalmierPro
import CoreImage

// Load a specific effect from the registry
let wheelsEffect = EffectRegistry.all.first { $0.id == "color.wheels" }!
var params = wheelsEffect.makeEffect()

// Adjust color grading parameters
params.params["lift_x"]?.value = 0.2    // Shift shadows toward red
params.params["gain_m"]?.value = 1.1   // Increase overall gain

// Resolve parameters for frame offset 0
let resolved = wheelsEffect.resolve(params, atOffset: 0)

// Apply to a CIImage (e.g., from AVAssetImageGenerator)
let sourceImage: CIImage = ...
let gradedImage = WheelsKernel.apply(sourceImage, params: resolved)

Integrating with AVPlayer for Preview

For real-time preview, attach the custom compositor to an AVPlayerItem:

import AVFoundation
import PalmierPro

let playerItem = AVPlayerItem(url: videoURL)

// Configure the custom video compositor
let composition = AVVideoComposition()
composition.customVideoCompositorClass = CustomVideoCompositor.self
composition.instructions = [/* CompositorInstruction instances */]
playerItem.videoComposition = composition

let player = AVPlayer(playerItem: playerItem)
player.play()

The compositor automatically processes all registered effects on the GPU for each frame during playback.

Summary

• Metal Integration: Palmier Pro uses a custom SPM plugin (MetalCIKernelPlugin) to compile .metal sources into .metallib bundles compatible with Core Image.
• Kernel Loading: CIKernelLoader.colorKernel instantiates CIColorKernel objects from compiled Metal library data, bridging low-level GPU code with Swift.
• Effect Architecture: EffectRegistry maintains declarative descriptors that map UI parameters to kernel arguments via ResolvedEffectParams.
• Zero-Copy Rendering: CustomVideoCompositor creates a Metal-backed CIContext that keeps video frames in GPU memory throughout the rendering pipeline, enabling real-time processing of high-resolution content.

Frequently Asked Questions

How does Palmier Pro compile Metal shaders for Core Image?

Palmier Pro includes a custom Swift Package Manager build tool plugin (MetalCIKernelPlugin.swift) that automatically runs xcrun metal -c -fcikernel and xcrun metallib -cikernel during the build process. This compiles .metal source files into .metallib bundles that Core Image can load as CIKernel objects at runtime.

Can I use Palmier Pro's GPU effects without the full compositor?

Yes. Individual effects can be applied directly by retrieving the effect descriptor from EffectRegistry.all, resolving parameters into a ResolvedEffectParams struct, and calling the kernel's static apply method (such as WheelsKernel.apply). This bypasses the AVFoundation compositor while still utilizing GPU acceleration through Core Image.

Why does the CustomVideoCompositor use NSNull for color spaces?

The compositor disables Core Image's automatic color management by setting both .workingColorSpace and .outputColorSpace to NSNull(). This prevents Core Image from converting pixel data to standard color spaces, ensuring that the Metal shaders receive raw linear RGB values and avoiding unnecessary GPU computations during effects processing.

What Metal device does the compositor use for rendering?

CustomVideoCompositor attempts to create a CIContext backed by the system default Metal device via MTLCreateSystemDefaultDevice(). If Metal hardware is unavailable, it falls back to a software-rendered CIContext. This ensures GPU acceleration on supported devices while maintaining compatibility with simulators or older hardware.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}