How Tool Executors Interact with Timeline Elements in palmier-pro: Clips, Effects, and Text

ToolExecutor acts as the bridge between the Agent/CLI API and the editor's internal model, decoding JSON-RPC-style tool calls and mutating Clip, Effect, TextStyle, and Timeline objects inside undoable EditorViewModel transactions.

In the palmier-io/palmier-pro repository, understanding how tool executors interact with timeline elements is essential for building agent-driven video editing workflows. The ToolExecutor struct serves as the central dispatcher that validates incoming tool arguments and applies changes to the EditorViewModel timeline through a consistent three-stage pipeline: decode and validate arguments via DecodableToolArgs, resolve IDs and collect context from Timeline and Track objects, and apply mutations inside an undo group.

Clip Operations in ToolExecutor

Adding Clips with add_clips

The addClips method in Sources/PalmierPro/Agent/Tools/ToolExecutor+Clips.swift places new media on the timeline by resolving a media asset and inserting a Clip into a target track.

func addClips(_ editor: EditorViewModel, _ args: [String: Any]) throws -> ToolResult {
    let input: AddClipsInput = try decodeToolArgs(args, path: "add_clips")
    // ...
    let asset = try asset(entry.mediaRef, editor: editor)
    // ...
    editor.clearRegion(trackIndex: trackIdx,
                       start: entry.startFrame,
                       end: entry.startFrame + entry.durationFrames,
                       prune: false)
    let ids = editor.placeClip(asset: asset,
                               trackIndex: trackIdx,
                               startFrame: entry.startFrame,
                               durationFrames: entry.durationFrames,
                               trimStartFrame: entry.trimStartFrame,
                               trimEndFrame: entry.trimEndFrame)
}

Under the hood, editor.placeClip creates a Clip instance with default fields such as speed = 1.0 and volume = 1.0, and automatically adds any linked audio. The new clip is inserted into the target Track.clips array, and the editor records an undo step.

Updating Clip Properties with set_clip_properties

The setClipProperties method resolves each clipId to its MediaType and then batches property changes inside an undo group via withUndoGroup.

func setClipProperties(_ editor: EditorViewModel, _ args: [String: Any]) throws -> ToolResult {
    let input: SetClipPropertiesInput = try decodeToolArgs(args, path: "set_clip_properties")
    // ...
    for id in input.clipIds {
        guard let loc = editor.findClip(id: id) else { throw ToolError("Clip not found: \(id)") }
        clipTypes[id] = editor.timeline.tracks[loc.trackIndex].clips[loc.clipIndex].mediaType
    }
    // ...
    let summaries = withUndoGroup(editor, actionName: setActionName) {
        for id in input.clipIds {
            let isText = clipTypes[id] == .text
            _ = Self.applyPropertyChanges(
                durationFrames: input.durationFrames,
                // ...
                blendMode: blendMode,
                setBlendMode: setBlendMode,
                clipId: id,
                editor: editor)
        }
    }
}

The helper applyPropertyChanges calls editor.commitClipProperty(clipId:). Scalar values for volume or opacity overwrite any existing keyframe track, while timing changes trigger clip.clampKeyframesToDuration(). The transform property receives partial updates so only supplied fields replace defaults.

Linked-Clip Propagation for Timing Changes

When the executor modifies timing-related fields such as durationFrames, trimStartFrame, trimEndFrame, or speed, it automatically propagates those values to linked partner clips.

let partners = propagatesTiming
    ? editor.timingPropagationPartners(of: Set(input.clipIds))
    : []
for partnerId in partners {
    // Text partners receive no timing changes
    _ = Self.applyPropertyChanges(
        durationFrames: input.durationFrames,
        // ...
        clipId: partnerId,
        editor: editor)
}

As implemented in palmier-pro, EditorViewModel.timingPropagationPartners(of:) returns the set of linked IDs. Text partners are excluded from timing propagation because captions are not locked to source media timing.

How Effects Are Wired to the Timeline

The Effect Model in Effect.swift

Effects are defined in Sources/PalmierPro/Models/Effect.swift as structs attached to clips via the optional Clip.effects array.

struct Effect: Codable, Sendable, Equatable, Identifiable {
    var id: String = UUID().uuidString
    var type: String                // e.g. "color.exposure"
    var enabled: Bool = true
    var params: [String: EffectParam] = [:]
}

Each EffectParam stores a numeric value, a string, or an animated keyframe track.

struct EffectParam: Codable, Sendable, Equatable {
    var value: Double?
    var string: String?
    var track: KeyframeTrack<Double>?
}

Rendering Pipeline Integration

The agent-facing ToolExecutor does not expose a dedicated tool for effect mutation. Instead, effects are typically edited through the UI or created programmatically via helpers such as Effect.make. During rendering, the engine reads Clip.effects and maps each Effect.type to a Core Image filter or custom shader inside the AVVideoComposition pipeline.

Text Handling and TextStyle Updates

Validating Text-Only Fields on Text Clips

SetClipPropertiesInput enumerates text-only keys: content, fontName, fontSize, color, and alignment. In ToolExecutor+Clips.swift, the executor validates that every target clip has mediaType == .text. If any non-text clip is selected, it throws a ToolError reading:

ToolError: text-only fields 'content', 'fontName' … rejected on non-text clips: …

This guard ensures that only text clips receive TextStyle mutations.

Updating TextStyle and Auto-Fitting Content

Inside applyPropertyChanges, text fields update clip.textContent and reconstruct clip.textStyle:

if content != nil || fontName != nil || fontSize != nil || color != nil || alignment != nil {
    if let c = content { clip.textContent = c }
    var style = clip.textStyle ?? TextStyle()
    if let f = fontName { style.fontName = f }
    if let s = fontSize { style.fontSize = s }
    if let c = color { style.color = c }
    if let a = alignment { style.alignment = a }
    clip.textStyle = style
}

The TextStyle struct, defined in Sources/PalmierPro/Models/TextStyle.swift, carries the visual properties used by the renderer to rasterize text into a texture.

If the caller changes text properties without supplying a new transform, the executor calls editor.fitTextClipToContent(clipId:) to resize the bounding box to match the rendered content.

Practical Tool Execution Examples

Changing Video Opacity and Duration

This set_clip_properties call targets a video clip and updates Clip.opacity and Clip.durationFrames.

{
  "name": "set_clip_properties",
  "args": {
    "clipIds": ["c1234"],
    "opacity": 0.5,
    "durationFrames": 120
  }
}

The executor validates the clip ID, confirms the media type, clears any existing opacityTrack, clamps keyframes to the new duration, and commits the result inside an undoable transaction.

Updating a Caption's Text and Style

Because the clip's mediaType is .text, the executor accepts content and style fields.

{
  "name": "set_clip_properties",
  "args": {
    "clipIds": ["t5678"],
    "content": "Hello, world!",
    "fontSize": 24,
    "color": "#FF00FF",
    "alignment": "center"
  }
}

The hex string is mapped to TextStyle.RGBA, clip.textContent is updated, and editor.fitTextClipToContent recomputes the bounding box so the caption fits its new text.

Adding a Video Clip with Linked Audio

The add_clips tool resolves a media asset and places it on a specific track.

{
  "name": "add_clips",
  "args": {
    "entries": [
      {
        "mediaRef": "media-uuid-abc",
        "trackIndex": 2,
        "startFrame": 300,
        "durationFrames": 150,
        "trimStartFrame": 30,
        "trimEndFrame": 0
      }
    ]
  }
}

ToolExecutor calls editor.clearRegion to remove overlapping frames, then editor.placeClip to create the Clip. If the asset contains audio, a linked audio clip is automatically appended to the corresponding audio track.

Summary

• ToolExecutor decodes agent tool calls and mutates timeline objects inside withUndoGroup so every change is undoable and triggers a UI refresh.
• Clip properties such as durationFrames, speed, and transform are written directly via editor.commitClipProperty; scalar changes to volume or opacity overwrite existing keyframe tracks.
• Linked-clip propagation mirrors timing fields to partner clips automatically, while text captions are intentionally excluded from propagation.
• Effects are stored as [Effect]? on each Clip and consumed by the rendering pipeline; the agent layer does not directly edit them through dedicated tools.
• Text-only fields are strictly validated against mediaType == .text, and the executor triggers fitTextClipToContent to keep bounding boxes synchronized with content changes.

Frequently Asked Questions

How does ToolExecutor ensure timeline changes are undoable?

Every mutating operation wraps its editor calls inside withUndoGroup(editor, actionName:). This records the modification through the editor's undo manager and refreshes the UI, allowing users to revert actions via the standard undo and redo stack.

Can I update effects on a clip through the agent tool interface?

No. As implemented in palmier-pro, the agent-facing ToolExecutor does not expose a tool for direct effect manipulation. Effects reside in Clip.effects and are typically edited through the UI or created programmatically with helpers like Effect.make, then rendered by the CompositionBuilder.

Why are text-only properties rejected on non-text clips?

The executor validates target clip mediaType values before applying content, fontName, fontSize, color, or alignment. If any selected clip is not of type .text, it throws a ToolError to prevent invalid state, since these fields map to clip.textContent and clip.textStyle which only exist on text clips.

What happens to linked audio when I change a video clip's duration?

When timing-related fields are modified, the executor queries editor.timingPropagationPartners(of:) and applies the same timing values to linked partners. Text partners are excluded from this propagation. For audio linked to video, this keeps the tracks synchronized.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}