# How to Customize or Add New Agents in Palmier Pro: A Complete Developer Guide

> Learn how to customize or add new agents in Palmier Pro. This developer guide details editing system prompts, adding starter prompts, and registering native Swift tools.

- Repository: [Palmier/palmier-pro](https://github.com/palmier-io/palmier-pro)
- Tags: how-to-guide
- Published: 2026-06-30

---

**Palmier Pro’s agent architecture is centralized in `AgentService` and modularized through `AgentInstructions`, `ToolDefinitions`, and `AgentPanelView`, allowing developers to customize behavior by editing the system prompt, adding starter prompts, or registering new native tools in Swift.**

Palmier Pro ships with a single AI-driven agent that powers the chat panel, handles tool calls, and drives the video-editing workflow. To customize or add new agents in Palmier Pro, you modify specific Swift components that control the system prompt, user interface, and native tool execution. The architecture cleanly separates concerns between orchestration, prompting, and UI rendering, making targeted customization straightforward without refactoring the entire application.

## Understanding the Agent Architecture

The agent system in Palmier Pro consists of four interconnected Swift components that handle different aspects of the AI interaction:

- **`AgentService`** – The central model that owns chat sessions, drafts, mentions, streaming state, and selects the backend client (Anthropic API or built-in Palmier service).
- **`AgentInstructions`** – A large multiline string sent as the system prompt to the LLM, describing the editor’s data model, available tools, and usage guidelines.
- **`AgentPanelView`** – The SwiftUI view rendering the chat UI, including tabs, message lists, and the footer with starter prompts.
- **`ToolDefinitions.inAppAgent`** – The enumeration of tools the LLM may call (e.g., `get_timeline`, `add_clips`, `apply_layout`), with implementations in `ToolExecutor`.

## Core Components of the Agent System

### AgentService – The Central Orchestrator

Located at [`Sources/PalmierPro/Agent/AgentService.swift`](https://github.com/palmier-io/palmier-pro/blob/main/Sources/PalmierPro/Agent/AgentService.swift), this class manages the chat lifecycle, model selection via `effectiveModel`, and tool execution through `runPendingToolUses`. During startup (lines 11-22), it reads the Anthropic API key from the keychain and registers for change notifications.

When a user sends a message, `selectClient()` (lines 52-58) determines whether to use `AnthropicClient` (for custom API keys) or the built-in `PalmierClient` (for signed-in accounts). The `runLoop()` method (line 58) concatenates `AgentInstructions.serverInstructions` with the skill index before streaming begins.

### AgentInstructions – The System Prompt

The file [`Sources/PalmierPro/Agent/Tools/AgentInstructions.swift`](https://github.com/palmier-io/palmier-pro/blob/main/Sources/PalmierPro/Agent/Tools/AgentInstructions.swift) contains the `serverInstructions` static string that dictates everything the model "knows" about Palmier Pro. This is the primary location to tweak the agent’s personality, add domain-specific rules, or document new capabilities. The string is sent verbatim to the LLM with each request.

### AgentPanelView – The Chat Interface

Found at [`Sources/PalmierPro/Agent/Panel/AgentPanelView.swift`](https://github.com/palmier-io/palmier-pro/blob/main/Sources/PalmierPro/Agent/Panel/AgentPanelView.swift), this SwiftUI view exposes the model picker, starter prompts, and message input. The `starterPrompts` array (lines 6-42) defines quick-launch commands that inject predefined text into the draft when tapped.

### ToolDefinitions and ToolExecutor – Native Capabilities

The tool catalogue lives under `Sources/PalmierPro/Agent/Tools/`. The `ToolDefinitions.inAppAgent` enumeration declares available tools, while `ToolExecutor` provides the native Swift implementations. When the LLM requests a tool via a `tool_use` block, `AgentService` parses the JSON and hands execution to `ToolExecutor` (lines 41-45 in [`AgentService.swift`](https://github.com/palmier-io/palmier-pro/blob/main/AgentService.swift)), which returns a `tool_result` block appended to the conversation.

## How to Customize the Agent

### Modify the System Prompt

To change the agent’s behavior or add new domain-specific rules, edit `AgentInstructions.serverInstructions` in [`Sources/PalmierPro/Agent/Tools/AgentInstructions.swift`](https://github.com/palmier-io/palmier-pro/blob/main/Sources/PalmierPro/Agent/Tools/AgentInstructions.swift). Because this string is sent verbatim to the LLM with every request, any new instructions or constraints take effect immediately without recompiling the model logic.

```swift
// In AgentInstructions.swift – append a new rule:
static let serverInstructions: String = """
    …existing text…

    # New custom tool

    - Tool name: export_to_gif
    - Parameters: clipId (String), durationFrames (Int)
    - Returns: a mediaRef that points to the generated GIF.
    - Use this when the user asks for a short looping preview.
"""

```

### Add New Starter Prompts

To expose quick-launch commands in the chat UI, append a new `AgentStarterPrompt` entry to the `starterPrompts` array in [`Sources/PalmierPro/Agent/Panel/AgentPanelView.swift`](https://github.com/palmier-io/palmier-pro/blob/main/Sources/PalmierPro/Agent/Panel/AgentPanelView.swift). The UI automatically displays a button that inserts the prompt into the draft.

```swift
// In AgentPanelView.swift – extend the static array:
private static let starterPrompts: [AgentStarterPrompt] = [
    // …existing prompts…
    AgentStarterPrompt(
        title: "Create a custom GIF",
        systemImage: "photo.on.rectangle",
        prompt: "Generate a short GIF for my timeline. Use the last 2 seconds of the current video clip as source."
    )
]

```

### Expose Custom AI Models

The model picker in the footer (lines 41-55 of [`AgentPanelView.swift`](https://github.com/palmier-io/palmier-pro/blob/main/AgentPanelView.swift)) lists `service.availableModels`. To expose additional Anthropic models, update the `availableModels` property in [`Sources/PalmierPro/Agent/AgentService.swift`](https://github.com/palmier-io/palmier-pro/blob/main/Sources/PalmierPro/Agent/AgentService.swift) (lines 47-50) or provide a new `AnthropicModel` case.

```swift
// In AgentService.swift – expose a new model:
var availableModels: [AnthropicModel] {
    if hasApiKey { return AnthropicModel.allCases }
    // Add a custom model for non-API users
    return AccountService.shared.isPaid ? [.sonnet46, .customModel] : [.haiku45]
}

```

## How to Add New Agent Tools

### Define the Tool Interface

First, extend the tool enumeration to register the new capability with the system:

```swift
// Somewhere in the tool definition file (e.g. ToolDefinitions.swift)
enum InAppAgentTool: String, CaseIterable {
    // …existing tools…
    case export_to_gif = "export_to_gif"
}

```

### Implement the Swift Logic

Add the execution logic to [`ToolExecutor.swift`](https://github.com/palmier-io/palmier-pro/blob/main/ToolExecutor.swift) within the `execute(name:args:)` switch statement. This function receives the tool name and parsed arguments, runs the native implementation, and returns a `ToolResult`:

```swift
// In ToolExecutor.swift – handle the call:
func execute(name: String, args: [String: Any]) async -> ToolResult {
    switch name {
    // …existing cases…
    case "export_to_gif":
        // Your Swift implementation that creates the GIF and returns a mediaRef.
        return await generateGIF(clipId: args["clipId"] as! String,
                                 duration: args["durationFrames"] as! Int)
    default:
        return .error("Unknown tool")
    }
}

```

### Update the System Instructions

Finally, document the new tool in [`AgentInstructions.swift`](https://github.com/palmier-io/palmier-pro/blob/main/AgentInstructions.swift) so the LLM understands when and how to invoke it. Include the exact tool name, required parameters, and expected return values in the `serverInstructions` string.

## Swapping the Backend Client

To use a different LLM provider instead of Anthropic or Palmier’s built-in service, replace `AnthropicClient` or `PalmierClient` with your own implementation in the `selectClient()` method of [`AgentService.swift`](https://github.com/palmier-io/palmier-pro/blob/main/AgentService.swift). Ensure your custom client implements the same streaming API used by `runLoop()` to maintain compatibility with the message synchronization and tool execution pipeline.

## Summary

- **Prompt control** resides in [`AgentInstructions.swift`](https://github.com/palmier-io/palmier-pro/blob/main/AgentInstructions.swift), where `serverInstructions` defines the LLM’s knowledge and personality.
- **UI customization** happens in [`AgentPanelView.swift`](https://github.com/palmier-io/palmier-pro/blob/main/AgentPanelView.swift) through the `starterPrompts` array and model picker configuration.
- **Tool extension** requires updating `ToolDefinitions`, implementing logic in `ToolExecutor`, and documenting capabilities in the system prompt.
- **Orchestration** is handled by [`AgentService.swift`](https://github.com/palmier-io/palmier-pro/blob/main/AgentService.swift), which manages client selection, streaming, and tool execution loops.

All customization points are localized to the `Sources/PalmierPro/Agent/` directory, leaving the core video-editing application untouched.

## Frequently Asked Questions

### Can I add multiple specialized agents to Palmier Pro instead of just one?

Currently, Palmier Pro operates with a single centralized `AgentService` that manages all chat sessions. To create multiple specialized agents, you would need to instantiate separate `AgentService` configurations with distinct `AgentInstructions` strings for each specialized role, or modify the existing service to support multiple concurrent instruction sets.

### Where does Palmier Pro store the Anthropic API key securely?

The API key is stored in the macOS keychain through [`AgentPane.swift`](https://github.com/palmier-io/palmier-pro/blob/main/AgentPane.swift) ([`Sources/PalmierPro/Settings/AgentPane.swift`](https://github.com/palmier-io/palmier-pro/blob/main/Sources/PalmierPro/Settings/AgentPane.swift)). `AgentService` reads this key during initialization (lines 11-22) and uses it to authenticate `AnthropicClient` requests when `selectClient()` determines an API key is present.

### How do I ensure my new tool is actually invoked by the LLM?

You must update both the implementation and the documentation. First, add the tool case to `ToolDefinitions` and the execution logic to `ToolExecutor`. Then, explicitly describe the tool, its parameters, and use cases in `AgentInstructions.serverInstructions`. The LLM can only invoke tools it knows exist through the system prompt.

### What is the difference between starter prompts and the system prompt?

**Starter prompts** are user-facing shortcuts defined in [`AgentPanelView.swift`](https://github.com/palmier-io/palmier-pro/blob/main/AgentPanelView.swift) that insert predefined text into the chat input field. The **system prompt** (`AgentInstructions.serverInstructions`) is invisible to users but guides the LLM’s behavior, capabilities, and tool usage throughout the conversation.