# Implementing Compositor Instructions for Effects in Palmier Pro: A Complete Guide

> Master Palmier Pro's compositor instructions for video effects. This guide details implementing AVVideoCompositionInstructionProtocol and Core Image kernels for a data-driven pipeline.

- Repository: [Palmier/palmier-pro](https://github.com/palmier-io/palmier-pro)
- Tags: how-to-guide
- Published: 2026-06-24

---

**Palmier Pro renders video effects by constructing `CompositorInstruction` objects that implement `AVVideoCompositionInstructionProtocol` and executing Core Image kernels through a data-driven pipeline.**

Palmier Pro handles video compositing through a sophisticated pipeline that converts timeline data into executable rendering instructions. By leveraging `AVVideoCompositionInstructionProtocol` and custom Metal-based kernels, the system decouples effect definitions from rendering logic. This guide examines the `palmier-io/palmier-pro` source code to show how compositor instructions drive the effects system and how to extend it with new visual filters.

## Understanding the Compositor Architecture

The rendering system in Palmier Pro separates data modeling from frame execution through three primary layers. The **timeline layer** defines clips and effects as pure data structures. The **instruction layer** translates these models into `CompositorInstruction` objects that conform to `AVVideoCompositionInstructionProtocol`. Finally, the **rendering layer** executes these instructions using Core Image contexts.

**Key Components:**

- **`CompositorInstruction`** – Located in [`Sources/PalmierPro/Compositing/CompositorInstruction.swift`](https://github.com/palmier-io/palmier-pro/blob/main/Sources/PalmierPro/Compositing/CompositorInstruction.swift), this structure implements `AVVideoCompositionInstructionProtocol` and stores the layer array, time ranges, and effect kernels for a specific composition segment.

- **`CustomVideoCompositor`** – Found in [`Sources/PalmierPro/Compositing/CustomVideoCompositor.swift`](https://github.com/palmier-io/palmier-pro/blob/main/Sources/PalmierPro/Compositing/CustomVideoCompositor.swift), this class receives `AVAsynchronousVideoCompositionRequest` objects from AVFoundation, extracts the `CompositorInstruction`, and forwards it to the rendering engine.

- **`FrameRenderer`** – Defined in [`Sources/PalmierPro/Compositing/FrameRenderer.swift`](https://github.com/palmier-io/palmier-pro/blob/main/Sources/PalmierPro/Compositing/FrameRenderer.swift), this component iterates over instruction layers, applies transforms and masks, and executes CI kernels to produce the final `CVPixelBuffer`.

- **`CompositionBuilder`** – In [`Sources/PalmierPro/Preview/CompositionBuilder.swift`](https://github.com/palmier-io/palmier-pro/blob/main/Sources/PalmierPro/Preview/CompositionBuilder.swift), this utility converts `Timeline` models into arrays of `CompositorInstruction` objects, mapping clip effects to their corresponding kernel references.

## How Compositor Instructions Execute Effects

The pipeline follows a strict dataflow pattern that keeps effect implementation declarative. When AVFoundation requests a frame, `CustomVideoCompositor` receives the presentation time and retrieves the corresponding `CompositorInstruction` from the video composition.

**The Rendering Flow:**

1. **Instruction Retrieval** – The compositor casts `videoCompositionInstruction` to `CompositorInstruction` and passes it to `FrameRenderer`.

2. **Layer Compositing** – `FrameRenderer` iterates through the instruction's `layers` array. For video or image layers, it samples the source pixels; for effect layers, it applies the stored `CIKernel`.

3. **Kernel Execution** – Effect layers contain references to Metal-based kernels (such as those in [`Sources/PalmierPro/Rendering/VignetteKernel.swift`](https://github.com/palmier-io/palmier-pro/blob/main/Sources/PalmierPro/Rendering/VignetteKernel.swift)). The renderer executes these kernels on the accumulated image buffer, applying the effect to the composited result.

4. **Output Generation** – The final `CIImage` writes to the output `CVPixelBuffer` provided by the composition request.

Because `CompositorInstruction` stores the kernel object directly, the rendering loop treats all effects uniformly regardless of complexity. This design means new effects require no changes to `CustomVideoCompositor` or `FrameRenderer`.

## Implementing a New Effect

Adding a custom visual effect requires modifying three specific areas of the codebase. The following example demonstrates adding a pixelation effect using the patterns established in the repository.

### Step 1: Create the Metal CI Kernel

Define your kernel in the `Sources/PalmierPro/Rendering/` directory following the pattern used by existing effects like [`VignetteKernel.swift`](https://github.com/palmier-io/palmier-pro/blob/main/VignetteKernel.swift).

```swift
// Sources/PalmierPro/Rendering/PixelateKernel.swift
import CoreImage
import Metal

final class PixelateKernel {
    static let ciKernel: CIKernel = {
        let metalSource = """
        kernel vec4 pixelate(sampler image, float blockSize) {
            vec2 d = destCoord();
            vec2 block = floor(d / blockSize) * blockSize + blockSize / 2.0;
            return sample(image, block);
        }
        """
        return try! CIColorKernel(source: metalSource)
    }()
}

```

### Step 2: Expose the Effect in the Model

Extend the `Effect` enum in [`Sources/PalmierPro/Models/Effect.swift`](https://github.com/palmier-io/palmier-pro/blob/main/Sources/PalmierPro/Models/Effect.swift) to include your new kernel.

```swift
// Sources/PalmierPro/Models/Effect.swift
enum Effect {
    case vignette
    case glow
    case pixelate   // ← new case

    var ciKernel: CIKernel {
        switch self {
        case .vignette: return VignetteKernel.ciKernel
        case .glow:     return GlowKernel.ciKernel
        case .pixelate: return PixelateKernel.ciKernel
        }
    }
}

```

### Step 3: Integrate with CompositionBuilder

Update [`Sources/PalmierPro/Preview/CompositionBuilder.swift`](https://github.com/palmier-io/palmier-pro/blob/main/Sources/PalmierPro/Preview/CompositionBuilder.swift) to instantiate effect layers when building instructions. The builder's `makeInstruction(for:clip:)` routine checks for attached effects and appends the appropriate layer.

```swift
// Sources/PalmierPro/Preview/CompositionBuilder.swift
private func addEffectLayer(to instruction: inout CompositorInstruction,
                           from clip: Clip) {
    guard let effect = clip.effect else { return }

    let effectLayer = CompositorLayer(
        source: .effect(effect.ciKernel),
        transform: .identity,
        opacity: 1.0,
        mask: nil
    )
    instruction.layers.append(effectLayer)
}

```

The builder automatically calls this method for any clip carrying an effect, so your new pixelate effect becomes available immediately without further modification.

### Step 4: Surface in the UI

To make the effect selectable, add it to your effect chooser interface.

```swift
// Sources/PalmierPro/UI/EffectChooser.swift
Menu {
    Button("Vignette") { clip.effect = .vignette }
    Button("Glow")     { clip.effect = .glow }
    Button("Pixelate") { clip.effect = .pixelate }   // ← newly added
}

```

When users select the effect, the `clip.effect` property updates, triggering a timeline rebuild. The compositor then automatically incorporates the pixelation kernel into the rendering pipeline.

## Summary

- **CompositorInstruction** implements `AVVideoCompositionInstructionProtocol` to bridge AVFoundation requests with Palmier Pro's rendering logic.

- **FrameRenderer** executes the instruction's layer stack, treating effect kernels and video sources uniformly through Core Image.

- **CompositionBuilder** generates instruction arrays from timeline data, converting `Effect` model instances into executable kernel references.

- New effects require only three changes: a Metal kernel class, an entry in the `Effect` enum, and layer creation logic in `CompositionBuilder`.

- The architecture enforces separation of concerns, allowing effects to be data-driven without modifying the compositor or renderer source code.

## Frequently Asked Questions

### What is the role of CompositorInstruction in the Palmier Pro pipeline?

`CompositorInstruction` acts as the data contract between AVFoundation and Palmier Pro's rendering engine. It implements `AVVideoCompositionInstructionProtocol` and stores the time range, layer descriptions, and effect kernels needed to render a specific frame segment. Located in [`Sources/PalmierPro/Compositing/CompositorInstruction.swift`](https://github.com/palmier-io/palmier-pro/blob/main/Sources/PalmierPro/Compositing/CompositorInstruction.swift), it allows `CustomVideoCompositor` to remain agnostic about specific effects while providing `FrameRenderer` with all necessary execution parameters.

### How does CustomVideoCompositor handle asynchronous rendering requests?

`CustomVideoCompositor` receives `AVAsynchronousVideoCompositionRequest` objects from AVFoundation during playback or export. It extracts the `videoCompositionInstruction` property, casts it to `CompositorInstruction`, and passes it to `FrameRenderer`. The class is marked `@unchecked Sendable` to ensure thread-safe operation across AVFoundation's concurrent callback queues, as implemented in [`Sources/PalmierPro/Compositing/CustomVideoCompositor.swift`](https://github.com/palmier-io/palmier-pro/blob/main/Sources/PalmierPro/Compositing/CustomVideoCompositor.swift).

### Can I use built-in Core Image filters instead of custom Metal kernels?

Yes. While the example demonstrates custom Metal kernels in `Sources/PalmierPro/Rendering/`, the `Effect.ciKernel` property can return any `CIKernel` or `CIFilter` chain. Simply create a wrapper that returns the appropriate built-in filter, such as `CIFilter(name: "CIGaussianBlur")`, and reference it in the `Effect` enum. `FrameRenderer` will execute the filter the same way it executes custom Metal kernels.

### Why do I not need to modify FrameRenderer when adding new effects?

`FrameRenderer` operates on the abstraction of `CompositorInstruction` layers rather than specific effect types. When it encounters a layer with `source: .effect(kernel)`, it executes whatever `CIKernel` reference the layer contains, regardless of the kernel's implementation. This data-driven approach means `FrameRenderer` in [`Sources/PalmierPro/Compositing/FrameRenderer.swift`](https://github.com/palmier-io/palmier-pro/blob/main/Sources/PalmierPro/Compositing/FrameRenderer.swift) automatically supports any kernel you add to the system, provided you wire it through `CompositionBuilder` and the `Effect` model.