Creating Chroma Key Compositing with Core Image Kernels in Palmier Pro

Palmier Pro implements chroma key compositing by compiling a Metal shader into a Core Image kernel that removes green or blue backgrounds with adjustable tolerance, softness, and spill suppression parameters.

Palmier Pro implements all image effects—including chroma key (green/blue screen) removal—as Core Image kernels compiled from Metal source files. The pipeline relies on Metal/ChromaKey.metal compiled into a .metallib resource, loaded at runtime by CIKernelLoader, and exposed to the UI through EffectRegistry with four tweakable parameters.

How the Chroma Key Kernel Loads at Runtime

The underlying GPU code lives in Metal/ChromaKey.metal. The CIKernelLoader utility in Sources/PalmierPro/Compositing/Kernels/CIKernelLoader.swift reads the compiled library and instantiates a CIColorKernel using the bundled resource:

// CIKernelLoader.swift
enum CIKernelLoader {
    private static func data(_ lib: String) -> Data? {
        BundledResource.url("\(lib).metallib").flatMap { try? Data(contentsOf: $0) }
    }
    
    static func colorKernel(_ lib: String, _ function: String) -> CIColorKernel? {
        data(lib).flatMap { try? CIColorKernel(functionName: function, fromMetalLibraryData: $0) }
    }
}

The chroma key kernel is instantiated privately within the wrapper by calling CIKernelLoader.colorKernel("ChromaKey", "chromaKey").

Bridging Swift to Metal with ChromaKeyKernel

ChromaKeyKernel.swift provides a type-safe Swift interface in Sources/PalmierPro/Compositing/Kernels/ChromaKeyKernel.swift. The apply(_:keyHue:tolerance:softness:spill:) method validates that tolerance is greater than zero and forwards arguments to the GPU:

// ChromaKeyKernel.swift
enum ChromaKeyKernel {
    private static let kernel = CIKernelLoader.colorKernel("ChromaKey", "chromaKey")

    static func apply(_ image: CIImage,
                      keyHue: Double,
                      tolerance: Double,
                      softness: Double,
                      spill: Double) -> CIImage {
        guard let kernel, tolerance > 0 else { return image }
        return kernel.apply(extent: image.extent,
                            arguments: [image,
                                        Float(keyHue),
                                        Float(tolerance),
                                        Float(softness),
                                        Float(spill)]) ?? image
    }
}

Because the kernel operates on a CIImage, it integrates seamlessly with other Core Image filters in the chain and benefits from GPU acceleration.

Registering the Effect in EffectRegistry

The effect is declared in Sources/PalmierPro/Compositing/EffectRegistry.swift as an EffectDescriptor that defines four controllable parameters and points to the wrapper above:

// EffectRegistry.swift (excerpt)
private static let key: [EffectDescriptor] = [
    EffectDescriptor(
        id: "key.chroma",
        displayName: "Chroma Key",
        category: "Key",
        params: [
            EffectParamSpec(key: "keyHue",   label: "Key Hue",   range: 0...1, defaultValue: 0.333, unit: ""),
            EffectParamSpec(key: "tolerance",label: "Tolerance", range: 0...1, defaultValue: 0,     unit: ""),
            EffectParamSpec(key: "softness", label: "Softness",  range: 0...1, defaultValue: 0.1,   unit: ""),
            EffectParamSpec(key: "spill",    label: "Spill",     range: 0...1, defaultValue: 0.5,   unit: "")
        ],
        apply: { image, p, _ in
            ChromaKeyKernel.apply(image,
                                  keyHue: p.value("keyHue"),
                                  tolerance: p.value("tolerance"),
                                  softness: p.value("softness"),
                                  spill: p.value("spill"))
        })
]

The descriptor is reachable via EffectRegistry.descriptor(id:), allowing the inspector to display sliders for key hue, tolerance, softness, and spill.

Sampling Key Hues from Video Frames

When users click the preview to pick a background color, ChromaKeySamplerOverlayView in Sources/PalmierPro/Preview/ChromaKeySamplerOverlayView.swift converts screen coordinates to normalized space and delegates to VideoEngine.sampleKeyHue(at:frame:):

// VideoEngine.swift – sampling
func sampleKeyHue(at normalizedPoint: CGPoint, frame: Int? = nil) async -> Double? {
    guard let item = player.currentItem else { return nil }
    let time = frame.flatMap(playerTime(forPreviewFrame:)) ?? player.currentTime()
    let generator = AVAssetImageGenerator(asset: item.asset)
    generator.videoComposition = item.videoComposition
    guard let cg = try? await generator.image(at: time).image else { return nil }
    return Self.sampleKeyHue(from: cg, at: normalizedPoint)
}

The engine extracts a 9×9 pixel patch around the click point, computes the average color, converts it to HSV, and returns the hue if saturation is sufficient. The overlay view then commits the value to the editor model.

Practical Implementation Examples

Applying the Kernel to a Static Image

You can invoke the kernel directly on a CIImage without the editor timeline:

import CoreImage
import UIKit   // or AppKit for macOS

let ciImage = CIImage(contentsOf: URL(fileURLWithPath: "myGreenScreen.png"))!
let result = ChromaKeyKernel.apply(
    ciImage,
    keyHue: 0.333,        // green ≈ 1/3
    tolerance: 0.12,
    softness: 0.1,
    spill: 0.4
)

let context = CIContext()
let cgImage = context.createCGImage(result, from: result.extent)!
let uiImage = UIImage(cgImage: cgImage)   // macOS → NSImage

Adding the Effect to a Timeline Clip

Within the Palmier Pro editor, add the effect via the registry:

let descriptor = EffectRegistry.descriptor(id: "key.chroma")!
let effect = descriptor.makeEffect()    // Creates Effect with default params
editor.activeClip?.addEffect(effect)  // Triggers model update and rebuild

Triggering the Hue Sampler

Activate the color picker by setting the sampling clip ID:

editor.chromaKeySamplingClipId = clip.id
// The UI now shows a cross-hair cursor; clicking runs the sampling code
// from ChromaKeySamplerOverlayView.swift and commits via editor.commitChromaKeySample(hue:clipId:)

Summary

• Metal Source: The algorithm lives in Metal/ChromaKey.metal and ships as a .metallib resource.
• Runtime Loading: CIKernelLoader.colorKernel(_:_:) in CIKernelLoader.swift instantiates the CIColorKernel.
• Swift Wrapper: ChromaKeyKernel.apply validates inputs and binds the four parameters (keyHue, tolerance, softness, spill).
• UI Registration: EffectRegistry exposes the effect as "key.chroma" with sliders for real-time adjustment.
• Color Sampling: VideoEngine.sampleKeyHue extracts average hues from 9×9 pixel patches to set the key color automatically.

Frequently Asked Questions

What parameters control the chroma key matte in Palmier Pro?

The effect exposes four parameters defined in EffectRegistry.swift: keyHue (the target color to remove, default 0.333 for green), tolerance (width of the color range to key out), softness (feathering of the matte edge), and spill (suppression of color fringing on foreground edges). All values are normalized from 0 to 1.

How does Palmier Pro sample the key color from a video frame?

When the user activates the sampler and clicks the preview, ChromaKeySamplerOverlayView calculates normalized coordinates and calls VideoEngine.sampleKeyHue(at:frame:). This method uses AVAssetImageGenerator to extract a frame, then analyzes a 9×9 pixel region to compute the average hue, returning it asynchronously to the UI.

Can the chroma key kernel be used outside the Palmier Pro editor?

Yes. The ChromaKeyKernel.apply(_:keyHue:tolerance:softness:spill:) method in Sources/PalmierPro/Compositing/Kernels/ChromaKeyKernel.swift is a pure function that accepts any CIImage and returns a processed CIImage. You can import the module and apply it directly to static images or custom video pipelines without involving the EffectRegistry or timeline system.

Where is the compiled chroma key shader stored?

The compiled GPU code resides in ChromaKey.metallib, which is bundled as a resource in the application package. At runtime, CIKernelLoader reads this file via BundledResource.url(_:) and passes the raw data to CIColorKernel(functionName:fromMetalLibraryData:) to create the executable kernel object.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}