Building an Export Pipeline with Queue Management and Progress in Palmier Pro

Palmier Pro implements a robust export pipeline using a centralized ExportQueue to schedule jobs on background executors, exposing granular Progress objects that SwiftUI views and Agent tools observe for real-time status updates and cancellation control.

The export system in palmier-io/palmier-pro orchestrates complex media encoding workflows while keeping the UI responsive. By isolating I/O-heavy operations to dedicated background queues and managing state through structured job objects, the architecture supports concurrent operations with fine-grained progress tracking and safe cancellation.

Export Queue Architecture

The core of the pipeline resides in Sources/PalmierPro/Export/ExportQueue.swift, implemented as a lightweight actor-like object that maintains a FIFO list of ExportJob structs.

Job Structure and Scheduling

Each ExportJob encapsulates the target ExportOptions—including format, resolution, and HDR flags—and holds a reference to the ExportService that executes the actual encoding. When you enqueue a job, the queue immediately instantiates a Task on a dedicated background queue defined in ExportService.swift, returning a Progress object without blocking the caller.

let options = ExportOptions(format: .mov, resolution: .p1080, hdr: true)
ExportQueue.shared.enqueue(project: currentProject, options: options) { progress in
    progress.observe { fractionCompleted, description in
        print("Export \(fractionCompleted * 100)% – \(description)")
    }
}

State Management

The queue monitors each task’s lifecycle, updating the Progress object as bytes are written and removing completed or failed jobs from the active list. This design ensures that the main thread remains free for UI interactions while heavy AVFoundation operations proceed asynchronously.

Progress Reporting and UI Binding

Progress flows through a hierarchical Progress object that aggregates status from all export phases: media rendering, audio mixing, metadata generation, and file I/O.

SwiftUI Integration

In Sources/PalmierPro/Export/ExportView.swift, the UI binds directly to the top-level Progress via @Published state. As the underlying Progress updates its fractionCompleted property, SwiftUI automatically re-renders progress bars and status descriptions without manual polling.

Agent-Side Observation

The same Progress hierarchy is exposed to Agent tools, allowing background operations to surface granular status to automation workflows or external monitoring systems.

Background Execution and Cancellation

Export work executes off the main actor using asynchronous AVFoundation APIs, specifically AVAssetExportSession. The implementation respects Swift concurrency patterns by isolating CPU-intensive encoding to background executors.

Safe Cancellation

The queue checks Task.isCancelled at natural pause points—such as after completing a media segment—and aborts early when requested. This triggers cleanup of partially written files in ExportService.swift, preventing corrupted outputs when users dismiss export dialogs or Agent tools issue stop commands.

let cancelToken = ExportQueue.shared.currentTask?.cancel()

Agent Tool Integration

The Agent subsystem interacts with the pipeline through Sources/PalmierPro/Agent/Tools/ToolExecutor+Export.swift. The ExportProjectTool invokes ExportService.export(project:options:) directly, which internally manages queue insertion via ExportQueue.enqueue(job:).

The tool awaits the returned Progress until completion, then returns a structured receipt containing the final file URL, warning logs, and a success flag—fulfilling the Agent-tool contract for programmatic project exports.

let tool = ExportProjectTool()
let receipt = try await tool.run(projectID: project.id, options: options)
// receipt.fileURL points to the exported movie, receipt.success indicates outcome

Testing the Export Pipeline

The repository includes comprehensive unit tests verifying queue behavior under various conditions:

• ExportQueueTests.swift validates FIFO ordering, progress callback sequences, and cancellation handling.
• ExportResolutionTests.swift confirms that resolution options propagate correctly from ExportOptions through to the final encoded output.
• ExportServiceRoundTripTests.swift executes full export-import cycles to verify project integrity preservation.

These tests ensure that jobs serialize correctly, progress updates fire in expected order, and the system recovers gracefully from disk errors or user interruption.

Summary

• Centralized Queue: ExportQueue.swift manages FIFO job scheduling for all export operations, isolating heavy I/O from the UI thread.
• Live Progress: Hierarchical Progress objects bridge background encoding (via AVAssetExportSession) to SwiftUI views and Agent tools.
• Safe Cancellation: The queue respects Task.isCancelled at segment boundaries, cleaning up partial files to prevent corruption.
• Agent Integration: ExportProjectTool exposes the pipeline to automation through ToolExecutor+Export.swift, returning structured receipts upon completion.
• Verified Reliability: Unit tests in ExportQueueTests.swift and companions validate serialization, progress accuracy, and error recovery.

Frequently Asked Questions

How do I enqueue an export job and observe its progress in Palmier Pro?

Call ExportQueue.shared.enqueue(project:options:) with your ExportOptions configuration. The method returns immediately with a Progress object that updates as encoding proceeds. Bind this object to your SwiftUI view or observe it in an Agent tool to display real-time completion percentages and status descriptions.

Can I cancel an export after it starts?

Yes. The queue exposes the current task through ExportQueue.shared.currentTask, which supports standard Swift concurrency cancellation. The pipeline checks Task.isCancelled between media segments in ExportService.swift, aborting gracefully and cleaning up any partially written files to prevent output corruption.

What file formats and resolutions does the export pipeline support?

Configuration is defined in Sources/PalmierPro/Export/ExportOptions.swift, which supports standard video formats like .mov alongside resolution presets from 720p to 4K and HDR toggling. The ExportService maps these options to AVAssetExportSession presets during the encoding phase, as verified by ExportResolutionTests.swift.

How does the Agent tool integrate with the export queue?

The ExportProjectTool class in ToolExecutor+Export.swift acts as a bridge, calling ExportService.export(project:options:) which internally invokes ExportQueue.enqueue(job:). The tool waits for the Progress object to complete, then packages the result into a receipt with the file URL and status flags for programmatic consumption by automation workflows.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}