Implementing Compositor Instructions for Effects in Palmier Pro: A Complete Guide

Palmier Pro renders video effects by constructing CompositorInstruction objects that implement AVVideoCompositionInstructionProtocol and executing Core Image kernels through a data-driven pipeline.

Palmier Pro handles video compositing through a sophisticated pipeline that converts timeline data into executable rendering instructions. By leveraging AVVideoCompositionInstructionProtocol and custom Metal-based kernels, the system decouples effect definitions from rendering logic. This guide examines the palmier-io/palmier-pro source code to show how compositor instructions drive the effects system and how to extend it with new visual filters.

Understanding the Compositor Architecture

The rendering system in Palmier Pro separates data modeling from frame execution through three primary layers. The timeline layer defines clips and effects as pure data structures. The instruction layer translates these models into CompositorInstruction objects that conform to AVVideoCompositionInstructionProtocol. Finally, the rendering layer executes these instructions using Core Image contexts.

Key Components:

• CompositorInstruction – Located in Sources/PalmierPro/Compositing/CompositorInstruction.swift, this structure implements AVVideoCompositionInstructionProtocol and stores the layer array, time ranges, and effect kernels for a specific composition segment.

• CustomVideoCompositor – Found in Sources/PalmierPro/Compositing/CustomVideoCompositor.swift, this class receives AVAsynchronousVideoCompositionRequest objects from AVFoundation, extracts the CompositorInstruction, and forwards it to the rendering engine.

• FrameRenderer – Defined in Sources/PalmierPro/Compositing/FrameRenderer.swift, this component iterates over instruction layers, applies transforms and masks, and executes CI kernels to produce the final CVPixelBuffer.

• CompositionBuilder – In Sources/PalmierPro/Preview/CompositionBuilder.swift, this utility converts Timeline models into arrays of CompositorInstruction objects, mapping clip effects to their corresponding kernel references.

How Compositor Instructions Execute Effects

The pipeline follows a strict dataflow pattern that keeps effect implementation declarative. When AVFoundation requests a frame, CustomVideoCompositor receives the presentation time and retrieves the corresponding CompositorInstruction from the video composition.

The Rendering Flow:

1. Instruction Retrieval – The compositor casts videoCompositionInstruction to CompositorInstruction and passes it to FrameRenderer.

2. Layer Compositing – FrameRenderer iterates through the instruction's layers array. For video or image layers, it samples the source pixels; for effect layers, it applies the stored CIKernel.

3. Kernel Execution – Effect layers contain references to Metal-based kernels (such as those in Sources/PalmierPro/Rendering/VignetteKernel.swift). The renderer executes these kernels on the accumulated image buffer, applying the effect to the composited result.

4. Output Generation – The final CIImage writes to the output CVPixelBuffer provided by the composition request.

Because CompositorInstruction stores the kernel object directly, the rendering loop treats all effects uniformly regardless of complexity. This design means new effects require no changes to CustomVideoCompositor or FrameRenderer.

Implementing a New Effect

Adding a custom visual effect requires modifying three specific areas of the codebase. The following example demonstrates adding a pixelation effect using the patterns established in the repository.

Step 1: Create the Metal CI Kernel

Define your kernel in the Sources/PalmierPro/Rendering/ directory following the pattern used by existing effects like VignetteKernel.swift.

// Sources/PalmierPro/Rendering/PixelateKernel.swift
import CoreImage
import Metal

final class PixelateKernel {
    static let ciKernel: CIKernel = {
        let metalSource = """
        kernel vec4 pixelate(sampler image, float blockSize) {
            vec2 d = destCoord();
            vec2 block = floor(d / blockSize) * blockSize + blockSize / 2.0;
            return sample(image, block);
        }
        """
        return try! CIColorKernel(source: metalSource)
    }()
}

Step 2: Expose the Effect in the Model

Extend the Effect enum in Sources/PalmierPro/Models/Effect.swift to include your new kernel.

// Sources/PalmierPro/Models/Effect.swift
enum Effect {
    case vignette
    case glow
    case pixelate   // ← new case

    var ciKernel: CIKernel {
        switch self {
        case .vignette: return VignetteKernel.ciKernel
        case .glow:     return GlowKernel.ciKernel
        case .pixelate: return PixelateKernel.ciKernel
        }
    }
}

Step 3: Integrate with CompositionBuilder

Update Sources/PalmierPro/Preview/CompositionBuilder.swift to instantiate effect layers when building instructions. The builder's makeInstruction(for:clip:) routine checks for attached effects and appends the appropriate layer.

// Sources/PalmierPro/Preview/CompositionBuilder.swift
private func addEffectLayer(to instruction: inout CompositorInstruction,
                           from clip: Clip) {
    guard let effect = clip.effect else { return }

    let effectLayer = CompositorLayer(
        source: .effect(effect.ciKernel),
        transform: .identity,
        opacity: 1.0,
        mask: nil
    )
    instruction.layers.append(effectLayer)
}

The builder automatically calls this method for any clip carrying an effect, so your new pixelate effect becomes available immediately without further modification.

Step 4: Surface in the UI

To make the effect selectable, add it to your effect chooser interface.

// Sources/PalmierPro/UI/EffectChooser.swift
Menu {
    Button("Vignette") { clip.effect = .vignette }
    Button("Glow")     { clip.effect = .glow }
    Button("Pixelate") { clip.effect = .pixelate }   // ← newly added
}

When users select the effect, the clip.effect property updates, triggering a timeline rebuild. The compositor then automatically incorporates the pixelation kernel into the rendering pipeline.

Summary

• CompositorInstruction implements AVVideoCompositionInstructionProtocol to bridge AVFoundation requests with Palmier Pro's rendering logic.

• FrameRenderer executes the instruction's layer stack, treating effect kernels and video sources uniformly through Core Image.

• CompositionBuilder generates instruction arrays from timeline data, converting Effect model instances into executable kernel references.

• New effects require only three changes: a Metal kernel class, an entry in the Effect enum, and layer creation logic in CompositionBuilder.

• The architecture enforces separation of concerns, allowing effects to be data-driven without modifying the compositor or renderer source code.

Frequently Asked Questions

What is the role of CompositorInstruction in the Palmier Pro pipeline?

CompositorInstruction acts as the data contract between AVFoundation and Palmier Pro's rendering engine. It implements AVVideoCompositionInstructionProtocol and stores the time range, layer descriptions, and effect kernels needed to render a specific frame segment. Located in Sources/PalmierPro/Compositing/CompositorInstruction.swift, it allows CustomVideoCompositor to remain agnostic about specific effects while providing FrameRenderer with all necessary execution parameters.

How does CustomVideoCompositor handle asynchronous rendering requests?

CustomVideoCompositor receives AVAsynchronousVideoCompositionRequest objects from AVFoundation during playback or export. It extracts the videoCompositionInstruction property, casts it to CompositorInstruction, and passes it to FrameRenderer. The class is marked @unchecked Sendable to ensure thread-safe operation across AVFoundation's concurrent callback queues, as implemented in Sources/PalmierPro/Compositing/CustomVideoCompositor.swift.

Can I use built-in Core Image filters instead of custom Metal kernels?

Yes. While the example demonstrates custom Metal kernels in Sources/PalmierPro/Rendering/, the Effect.ciKernel property can return any CIKernel or CIFilter chain. Simply create a wrapper that returns the appropriate built-in filter, such as CIFilter(name: "CIGaussianBlur"), and reference it in the Effect enum. FrameRenderer will execute the filter the same way it executes custom Metal kernels.

Why do I not need to modify FrameRenderer when adding new effects?

FrameRenderer operates on the abstraction of CompositorInstruction layers rather than specific effect types. When it encounters a layer with source: .effect(kernel), it executes whatever CIKernel reference the layer contains, regardless of the kernel's implementation. This data-driven approach means FrameRenderer in Sources/PalmierPro/Compositing/FrameRenderer.swift automatically supports any kernel you add to the system, provided you wire it through CompositionBuilder and the Effect model.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}