# GPU Accelerated Video Effects in Swift: Implementing Metal-Based Processing with Palmier Pro > Unlock real-time GPU accelerated video effects in Swift with Palmier Pro. Learn how Metal kernels and AVFoundation create seamless visual processing on the GPU. - Repository: [Palmier/palmier-pro](https://github.com/palmier-io/palmier-pro) - Tags: how-to-guide - Published: 2026-06-24 --- **Palmier Pro achieves real-time GPU accelerated video effects in Swift by compiling Metal kernels into Core Image filters and orchestrating them through a custom AVFoundation compositor that processes frames directly on the GPU.** Palmier Pro is an open-source Swift framework that demonstrates production-grade GPU accelerated video effects in Swift by bridging low-level Metal shaders with high-level Core Image and AVFoundation APIs. The architecture leverages a custom build tool to compile Metal sources, a dynamic kernel loader to bridge Metal and Core Image, and a Metal-aware compositor that maintains pixel data on the GPU throughout the entire rendering pipeline. ## Architectural Overview: From Metal Shaders to Rendered Frames The palmier-io/palmier-pro repository implements a four-layer architecture for GPU video processing: 1. **Metal Source Files** (`*.metal`) containing per-pixel shader functions for effects like color wheels, glow, and grain. 2. **`CIKernelLoader`** – A lightweight utility that loads compiled `.metallib` bundles and exposes them as `CIColorKernel` objects. 3. **`EffectRegistry`** – A central registry that maps effect identifiers to their parameter schemas and kernel applications, resolving UI parameters into `ResolvedEffectParams` structs for each frame. 4. **`CustomVideoCompositor`** – An `AVVideoCompositing` implementation that connects AVFoundation's video playback and export pipelines to the Core Image processing chain. This design ensures that video frames remain in GPU memory from capture through effects processing to final output, eliminating expensive CPU-GPU transfers. ## Compiling Metal Kernels for Core Image The framework uses a custom Swift Package Manager build tool plugin to transform raw Metal source files into Core Image-compatible libraries at compile time. The `MetalCIKernelPlugin` (located in [`Plugins/MetalCIKernelPlugin/MetalCIKernelPlugin.swift`](https://github.com/palmier-io/palmier-pro/blob/main/Plugins/MetalCIKernelPlugin/MetalCIKernelPlugin.swift)) automatically processes all `.metal` files in the `Metal/` directory. The plugin executes the following build commands during compilation: ```swift let air = context.pluginWorkDirectoryURL.appending(path: "\(stem).air") let metallib = context.pluginWorkDirectoryURL.appending(path: "\(stem).metallib") // Command sequence: "xcrun metal -c -fcikernel '\(metal.path())' -o '\(air.path())' && " + "xcrun metallib -cikernel '\(air.path())' -o '\(metallib.path())'" ``` This generates `.metallib` bundles that Core Image can load directly as `CIKernel` objects, embedding the compiled GPU code as bundle resources. ## Loading GPU Kernels with CIKernelLoader At runtime, the `CIKernelLoader` class ([`Sources/PalmierPro/Compositing/Kernels/CIKernelLoader.swift`](https://github.com/palmier-io/palmier-pro/blob/main/Sources/PalmierPro/Compositing/Kernels/CIKernelLoader.swift)) bridges the compiled Metal libraries with Core Image. It searches the bundle for `.metallib` files and instantiates kernel objects using Metal library data: ```swift static func colorKernel(_ lib: String, _ function: String) -> CIColorKernel? { data(lib).flatMap { try? CIColorKernel(functionName: function, fromMetalLibraryData: $0) } } ``` Effect implementations wrap these kernels in type-safe enums. For example, the color wheels effect in [`Sources/PalmierPro/Compositing/Kernels/WheelsKernel.swift`](https://github.com/palmier-io/palmier-pro/blob/main/Sources/PalmierPro/Compositing/Kernels/WheelsKernel.swift) loads the compiled "Wheels" kernel: ```swift enum WheelsKernel { private static let kernel = CIKernelLoader.colorKernel("Wheels", "wheels") static func apply(_ image: CIImage, params p: ResolvedEffectParams) -> CIImage { guard let kernel, !ColorWheels.isNeutral(p) else { return image } let c = ColorWheels.coefficients(for: p) func vec(_ v: SIMD3) -> CIVector { CIVector(x: CGFloat(v.x), y: CGFloat(v.y), z: CGFloat(v.z)) } return kernel.apply(extent: image.extent, arguments: [image, vec(c.lift), vec(c.gain), vec(c.invGamma)]) ?? image } } ``` ## Registering Effects with EffectRegistry All video effects are declaratively registered in [`Sources/PalmierPro/Compositing/EffectRegistry.swift`](https://github.com/palmier-io/palmier-pro/blob/main/Sources/PalmierPro/Compositing/EffectRegistry.swift) using `EffectDescriptor` instances. Each descriptor defines the effect's identity, parameter specifications, and the closure that applies the kernel: ```swift EffectDescriptor( id: "color.wheels", displayName: "Color Wheels", category: "Color", params: [ EffectParamSpec(key: "lift_x", …), // Additional parameters for lift, gain, gamma... ], apply: { image, p, _ in WheelsKernel.apply(image, params: p) } ) ``` When rendering, the registry resolves dynamic UI parameters into `ResolvedEffectParams` and handles color space conversions. The render pipeline applies linearization before effects processing and re-applies the sRGB tone curve afterward: ```swift let params = resolve(effect, atOffset: offset) var working = image if linearizes { working = working.applyingFilter("CISRGBToneCurveToLinear") } working = apply(working, params, extent) if linearizes { working = working.applyingFilter("CILinearToSRGBToneCurve") } ``` ## Real-Time Compositing with CustomVideoCompositor The `CustomVideoCompositor` class ([`Sources/PalmierPro/Compositing/CustomVideoCompositor.swift`](https://github.com/palmier-io/palmier-pro/blob/main/Sources/PalmierPro/Compositing/CustomVideoCompositor.swift)) implements the `AVVideoCompositing` protocol to integrate with AVFoundation's `AVPlayerItem` and export sessions. It maintains a shared `CIContext` configured for GPU processing with color management disabled to preserve raw pixel data: ```swift static let ciContext: CIContext = { let options: [CIContextOption: Any] = [ .workingColorSpace: NSNull(), .outputColorSpace: NSNull(), .cacheIntermediates: true, ] if let device = MTLCreateSystemDefaultDevice() { return CIContext(mtlDevice: device, options: options) } return CIContext(options: options) }() ``` During each frame request, the compositor obtains a `CompositorInstruction`, pulls source frames as `CIImage` objects, executes `FrameRenderer.render` to iterate through enabled effects, and writes the result into a Metal-compatible pixel buffer. ## Practical Implementation Examples ### Applying a Color Wheels Effect Directly You can apply individual GPU effects without the full compositor by resolving parameters and calling the kernel wrapper directly: ```swift import PalmierPro import CoreImage // Load a specific effect from the registry let wheelsEffect = EffectRegistry.all.first { $0.id == "color.wheels" }! var params = wheelsEffect.makeEffect() // Adjust color grading parameters params.params["lift_x"]?.value = 0.2 // Shift shadows toward red params.params["gain_m"]?.value = 1.1 // Increase overall gain // Resolve parameters for frame offset 0 let resolved = wheelsEffect.resolve(params, atOffset: 0) // Apply to a CIImage (e.g., from AVAssetImageGenerator) let sourceImage: CIImage = ... let gradedImage = WheelsKernel.apply(sourceImage, params: resolved) ``` ### Integrating with AVPlayer for Preview For real-time preview, attach the custom compositor to an `AVPlayerItem`: ```swift import AVFoundation import PalmierPro let playerItem = AVPlayerItem(url: videoURL) // Configure the custom video compositor let composition = AVVideoComposition() composition.customVideoCompositorClass = CustomVideoCompositor.self composition.instructions = [/* CompositorInstruction instances */] playerItem.videoComposition = composition let player = AVPlayer(playerItem: playerItem) player.play() ``` The compositor automatically processes all registered effects on the GPU for each frame during playback. ## Summary - **Metal Integration**: Palmier Pro uses a custom SPM plugin (`MetalCIKernelPlugin`) to compile `.metal` sources into `.metallib` bundles compatible with Core Image. - **Kernel Loading**: `CIKernelLoader.colorKernel` instantiates `CIColorKernel` objects from compiled Metal library data, bridging low-level GPU code with Swift. - **Effect Architecture**: `EffectRegistry` maintains declarative descriptors that map UI parameters to kernel arguments via `ResolvedEffectParams`. - **Zero-Copy Rendering**: `CustomVideoCompositor` creates a Metal-backed `CIContext` that keeps video frames in GPU memory throughout the rendering pipeline, enabling real-time processing of high-resolution content. ## Frequently Asked Questions ### How does Palmier Pro compile Metal shaders for Core Image? Palmier Pro includes a custom Swift Package Manager build tool plugin ([`MetalCIKernelPlugin.swift`](https://github.com/palmier-io/palmier-pro/blob/main/MetalCIKernelPlugin.swift)) that automatically runs `xcrun metal -c -fcikernel` and `xcrun metallib -cikernel` during the build process. This compiles `.metal` source files into `.metallib` bundles that Core Image can load as `CIKernel` objects at runtime. ### Can I use Palmier Pro's GPU effects without the full compositor? Yes. Individual effects can be applied directly by retrieving the effect descriptor from `EffectRegistry.all`, resolving parameters into a `ResolvedEffectParams` struct, and calling the kernel's static `apply` method (such as `WheelsKernel.apply`). This bypasses the AVFoundation compositor while still utilizing GPU acceleration through Core Image. ### Why does the CustomVideoCompositor use `NSNull` for color spaces? The compositor disables Core Image's automatic color management by setting both `.workingColorSpace` and `.outputColorSpace` to `NSNull()`. This prevents Core Image from converting pixel data to standard color spaces, ensuring that the Metal shaders receive raw linear RGB values and avoiding unnecessary GPU computations during effects processing. ### What Metal device does the compositor use for rendering? `CustomVideoCompositor` attempts to create a `CIContext` backed by the system default Metal device via `MTLCreateSystemDefaultDevice()`. If Metal hardware is unavailable, it falls back to a software-rendered `CIContext`. This ensures GPU acceleration on supported devices while maintaining compatibility with simulators or older hardware.