How to Customize or Add New Agents in Palmier Pro: A Complete Developer Guide

Palmier Pro’s agent architecture is centralized in AgentService and modularized through AgentInstructions, ToolDefinitions, and AgentPanelView, allowing developers to customize behavior by editing the system prompt, adding starter prompts, or registering new native tools in Swift.

Palmier Pro ships with a single AI-driven agent that powers the chat panel, handles tool calls, and drives the video-editing workflow. To customize or add new agents in Palmier Pro, you modify specific Swift components that control the system prompt, user interface, and native tool execution. The architecture cleanly separates concerns between orchestration, prompting, and UI rendering, making targeted customization straightforward without refactoring the entire application.

Understanding the Agent Architecture

The agent system in Palmier Pro consists of four interconnected Swift components that handle different aspects of the AI interaction:

• AgentService – The central model that owns chat sessions, drafts, mentions, streaming state, and selects the backend client (Anthropic API or built-in Palmier service).
• AgentInstructions – A large multiline string sent as the system prompt to the LLM, describing the editor’s data model, available tools, and usage guidelines.
• AgentPanelView – The SwiftUI view rendering the chat UI, including tabs, message lists, and the footer with starter prompts.
• ToolDefinitions.inAppAgent – The enumeration of tools the LLM may call (e.g., get_timeline, add_clips, apply_layout), with implementations in ToolExecutor.

Core Components of the Agent System

AgentService – The Central Orchestrator

Located at Sources/PalmierPro/Agent/AgentService.swift, this class manages the chat lifecycle, model selection via effectiveModel, and tool execution through runPendingToolUses. During startup (lines 11-22), it reads the Anthropic API key from the keychain and registers for change notifications.

When a user sends a message, selectClient() (lines 52-58) determines whether to use AnthropicClient (for custom API keys) or the built-in PalmierClient (for signed-in accounts). The runLoop() method (line 58) concatenates AgentInstructions.serverInstructions with the skill index before streaming begins.

AgentInstructions – The System Prompt

The file Sources/PalmierPro/Agent/Tools/AgentInstructions.swift contains the serverInstructions static string that dictates everything the model "knows" about Palmier Pro. This is the primary location to tweak the agent’s personality, add domain-specific rules, or document new capabilities. The string is sent verbatim to the LLM with each request.

AgentPanelView – The Chat Interface

Found at Sources/PalmierPro/Agent/Panel/AgentPanelView.swift, this SwiftUI view exposes the model picker, starter prompts, and message input. The starterPrompts array (lines 6-42) defines quick-launch commands that inject predefined text into the draft when tapped.

ToolDefinitions and ToolExecutor – Native Capabilities

The tool catalogue lives under Sources/PalmierPro/Agent/Tools/. The ToolDefinitions.inAppAgent enumeration declares available tools, while ToolExecutor provides the native Swift implementations. When the LLM requests a tool via a tool_use block, AgentService parses the JSON and hands execution to ToolExecutor (lines 41-45 in AgentService.swift), which returns a tool_result block appended to the conversation.

How to Customize the Agent

Modify the System Prompt

To change the agent’s behavior or add new domain-specific rules, edit AgentInstructions.serverInstructions in Sources/PalmierPro/Agent/Tools/AgentInstructions.swift. Because this string is sent verbatim to the LLM with every request, any new instructions or constraints take effect immediately without recompiling the model logic.

// In AgentInstructions.swift – append a new rule:
static let serverInstructions: String = """
    …existing text…

    # New custom tool

    - Tool name: export_to_gif
    - Parameters: clipId (String), durationFrames (Int)
    - Returns: a mediaRef that points to the generated GIF.
    - Use this when the user asks for a short looping preview.
"""

Add New Starter Prompts

To expose quick-launch commands in the chat UI, append a new AgentStarterPrompt entry to the starterPrompts array in Sources/PalmierPro/Agent/Panel/AgentPanelView.swift. The UI automatically displays a button that inserts the prompt into the draft.

// In AgentPanelView.swift – extend the static array:
private static let starterPrompts: [AgentStarterPrompt] = [
    // …existing prompts…
    AgentStarterPrompt(
        title: "Create a custom GIF",
        systemImage: "photo.on.rectangle",
        prompt: "Generate a short GIF for my timeline. Use the last 2 seconds of the current video clip as source."
    )
]

Expose Custom AI Models

The model picker in the footer (lines 41-55 of AgentPanelView.swift) lists service.availableModels. To expose additional Anthropic models, update the availableModels property in Sources/PalmierPro/Agent/AgentService.swift (lines 47-50) or provide a new AnthropicModel case.

// In AgentService.swift – expose a new model:
var availableModels: [AnthropicModel] {
    if hasApiKey { return AnthropicModel.allCases }
    // Add a custom model for non-API users
    return AccountService.shared.isPaid ? [.sonnet46, .customModel] : [.haiku45]
}

How to Add New Agent Tools

Define the Tool Interface

First, extend the tool enumeration to register the new capability with the system:

// Somewhere in the tool definition file (e.g. ToolDefinitions.swift)
enum InAppAgentTool: String, CaseIterable {
    // …existing tools…
    case export_to_gif = "export_to_gif"
}

Implement the Swift Logic

Add the execution logic to ToolExecutor.swift within the execute(name:args:) switch statement. This function receives the tool name and parsed arguments, runs the native implementation, and returns a ToolResult:

// In ToolExecutor.swift – handle the call:
func execute(name: String, args: [String: Any]) async -> ToolResult {
    switch name {
    // …existing cases…
    case "export_to_gif":
        // Your Swift implementation that creates the GIF and returns a mediaRef.
        return await generateGIF(clipId: args["clipId"] as! String,
                                 duration: args["durationFrames"] as! Int)
    default:
        return .error("Unknown tool")
    }
}

Update the System Instructions

Finally, document the new tool in AgentInstructions.swift so the LLM understands when and how to invoke it. Include the exact tool name, required parameters, and expected return values in the serverInstructions string.

Swapping the Backend Client

To use a different LLM provider instead of Anthropic or Palmier’s built-in service, replace AnthropicClient or PalmierClient with your own implementation in the selectClient() method of AgentService.swift. Ensure your custom client implements the same streaming API used by runLoop() to maintain compatibility with the message synchronization and tool execution pipeline.

Summary

• Prompt control resides in AgentInstructions.swift, where serverInstructions defines the LLM’s knowledge and personality.
• UI customization happens in AgentPanelView.swift through the starterPrompts array and model picker configuration.
• Tool extension requires updating ToolDefinitions, implementing logic in ToolExecutor, and documenting capabilities in the system prompt.
• Orchestration is handled by AgentService.swift, which manages client selection, streaming, and tool execution loops.

All customization points are localized to the Sources/PalmierPro/Agent/ directory, leaving the core video-editing application untouched.

Frequently Asked Questions

Can I add multiple specialized agents to Palmier Pro instead of just one?

Currently, Palmier Pro operates with a single centralized AgentService that manages all chat sessions. To create multiple specialized agents, you would need to instantiate separate AgentService configurations with distinct AgentInstructions strings for each specialized role, or modify the existing service to support multiple concurrent instruction sets.

Where does Palmier Pro store the Anthropic API key securely?

The API key is stored in the macOS keychain through AgentPane.swift (Sources/PalmierPro/Settings/AgentPane.swift). AgentService reads this key during initialization (lines 11-22) and uses it to authenticate AnthropicClient requests when selectClient() determines an API key is present.

How do I ensure my new tool is actually invoked by the LLM?

You must update both the implementation and the documentation. First, add the tool case to ToolDefinitions and the execution logic to ToolExecutor. Then, explicitly describe the tool, its parameters, and use cases in AgentInstructions.serverInstructions. The LLM can only invoke tools it knows exist through the system prompt.

What is the difference between starter prompts and the system prompt?

Starter prompts are user-facing shortcuts defined in AgentPanelView.swift that insert predefined text into the chat input field. The system prompt (AgentInstructions.serverInstructions) is invisible to users but guides the LLM’s behavior, capabilities, and tool usage throughout the conversation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}