How Multicam Editing and Synchronization Works in Palmier Pro: A Technical Deep Dive

Palmier Pro implements multicam editing and synchronization through a three-layer architecture that combines audio-based correlation for automatic syncing, frame-accurate offset mapping, and atomic group operations to ensure angle switches maintain perfect alignment across all sources.

Palmier Pro is an open-source video editing framework designed for complex multicamera workflows. The multicam editing and synchronization system centers on three core components—the MulticamSource data model, the MulticamEngine processing layer, and the EditorViewModel+Multicam façade—that together enable real-time angle switching while preserving user transforms and sync relationships.

The Three-Layer Architecture

The multicam system is organized into distinct layers that separate data modeling from execution logic:

• MulticamSource (in Sources/PalmierPro/Models/MulticamSource.swift) defines the group structure, member metadata, and sync offsets.
• MulticamEngine (in Sources/PalmierPro/Timeline/MulticamEngine.swift) performs clip rewriting, overlay management, and fragment merging.
• EditorViewModel+Multicam (in Sources/PalmierPro/Editor/ViewModel/EditorViewModel+Multicam.swift) provides the high-level API for group creation, synchronization, and angle switching.

This separation ensures that all angle-switching logic funnels through a single engine, guaranteeing consistent behavior across the UI, Agent tools, and undo systems.

The Data Model: MulticamSource and Member Sync

At the heart of the system lies the MulticamSource structure, which groups Members representing camera angles, microphones, or combined video-audio tracks.

struct MulticamSource: Codable, Sendable, Equatable, Identifiable {
    struct Member: Codable, Sendable, Equatable, Identifiable {
        var id: String = UUID().uuidString
        var mediaRef: String                 // Asset identifier
        var kind: MemberKind                 // .angle, .mic, or .both
        var angleLabel: String               // Human-readable label
        var sync: SyncMap = SyncMap()        // Offset, confidence, lock flag
        var providesVideo: Bool { kind != .mic }
        var providesAudio: Bool { kind != .angle }
        var usable: Bool { sync.confidence > 0 || sync.locked }
    }
    // ...
}

Sync offsets stored in SyncMap.offsetSeconds express how many seconds a member is shifted relative to the group’s master clock. The usable property ensures only members with valid correlation confidence or manually locked offsets participate in editing. Helper methods like coverage(), offsetFrames(), and anchorFrame() convert between seconds and frame numbers given the timeline’s FPS.

Creating Multicam Groups

Groups are instantiated via EditorViewModel.createMulticamGroup, which orchestrates the setup:

1. Builds members from supplied specs and sync maps.
2. Determines program boundaries by merging the coverage of all video members.
3. Places a program track (video) and optional audio tracks for each microphone.
4. Generates clips for each member, applying trim offsets and default fit transforms.
5. Registers undo metadata enabling the entire group to be removed in a single undo step.

let (groupId, _) = try editor.createMulticamGroup(
    specs: [.init(mediaRef:"camA", kind:.angle, angleLabel:"cam-a"),
            .init(mediaRef:"camB", kind:.angle, angleLabel:"cam-b"),
            .init(mediaRef:"mic1", kind:.mic,   angleLabel:"mic-1")],
    syncMaps: ["camA": .init(offsetSeconds:0, confidence:1),
               "camB": .init(offsetSeconds:5, confidence:0.9),
               "mic1": .init(offsetSeconds:2, confidence:1)],
    masterRef: "mic1",
    name: "MC",
    startFrame: 0)

The implementation uses multicamSourceDurations to fetch source lengths and makeMemberClip to construct each clip. The test createLaysStampedClips verifies that program clips and microphone offsets are correctly laid out on the timeline.

Synchronizing Members with Audio Correlation

Automatic synchronization is performed asynchronously by syncMulticamMembers. The process extracts audio envelopes for each asset, then employs an AudioSyncCorrelator to compute the optimal lag between the master and target sources within a configurable search window. When audio is unavailable, the system falls back to time-code alignment. Finally, it rebases all offsets so the earliest usable member sits at zero seconds, simplifying subsequent calculations.

let outcome = try await editor.syncMulticamMembers(
    specs: specs,
    masterRef: "mic1")

The correlation respects minimum confidence thresholds and handles offset calculations that persist through angle switches. As demonstrated in the test switchRewritesTrimThroughSyncMaps, a sync offset of 5 seconds for cam-b means a switch at frame 600 produces a trim of 450 frames (accounting for the 5-second offset at 30 FPS).

Switching Angles via MulticamEngine

Angle switches are handled by MulticamEngine.apply, which processes an AngleSwitchRequest specifying the target frame range and desired angles. The engine performs several critical operations:

• Program track lookup: Identifies the video track containing the group’s program clips via programTrackId.
• Fragment enumeration: For each program fragment intersecting the request range, calculates the valid sub-range using clampToCoverage.
• Overlay management: Removes conflicting overlays with clearOverlays, then inserts additional angle clips as needed via placeOverlay.
• Clip rewriting: Calls MulticamEngine.rewrite to swap media references and adjust trim based on stored sync offsets.
• Transform preservation: Applies custom placement functions for layouts or falls back to default fit transforms.
• Fragment merging: Collapses adjacent clips forming "through edits" using joinThroughEdits.

let outcome = try editor.switchMulticamAngles(
    groupId: groupId,
    requests: [.init(range: 600..<1200, angle: "cam-b")])

The Outcome structure records statistics—including switched, merged, and clamped counts—for UI feedback. The test switchClampsToAngleCoverage validates that requests outside a member’s coverage are automatically clamped and reported.

Layout-Based Overlays and Picture-in-Picture

When applying multicam layouts such as picture-in-picture, the engine may place extra angle clips as overlays. The VideoLayout structure defines slots that determine where each angle appears. For every additional slot, placeOverlay creates a new Clip, computes its start frame using anchorFrame and the member’s offset, and inserts it in the highest-available video track above the program track.

editor.applyMulticamLayout(clipId: someClip.id, layout: .pip)

Custom transforms survive angle switches, while unframed fragments revert to default fits, as verified by the test userFramingSurvivesAngleSwitch.

Guardrails and Atomic Operations

Multicam editing maintains strict atomic relationships to prevent timeline corruption:

• Ripple edit prevention: Operations that would split a group without shifting linked members are refused via multicamMoveViolation.
• Partial edit blocking: Partial moves or speed changes on stamped program clips are ignored to keep the group synchronized.
• Atomic undo/redo: Undo operations work at the group level, not per-clip. The undo.perform("Create Multicam") block stores both clips and group metadata, while ungroupMulticam removes stamps while preserving underlying media.

These constraints are enforced by tests including partialRippleAcrossGroupRefuses, partialMoveRefused, and speedRefusedOnStampedClips.

Reading the Program Timeline

The UI can display which angle is active over time using multicamProgramRows. This function walks the program clips, merges consecutive fragments with identical angles, and returns a run-length encoded table.

let rows = editor.multicamProgramRows(groupId: groupId)
// Returns: [["cam-a", 0, 600], ["cam-b", 600, 1200], ["cam-a", 1200, 3600]]

The programRowsRunLengthMerge test validates this compression behavior, enabling efficient timeline visualization without processing every individual frame.

Summary

• Single source of truth: All angle-switching logic routes through MulticamEngine.apply, ensuring consistency across UI, Agent tools, and undo layers.
• Sync-first workflow: Offsets are computed once via audio correlation and stored in SyncMap; subsequent edits adjust trim frames using these persistent offsets.
• Atomic guarantees: Guardrails prevent partial edits that would desynchronize the group, preserving timeline integrity during complex operations.
• Extensible layout system: New VideoLayout configurations automatically trigger overlay insertion through the existing engine methods without requiring structural changes.

Frequently Asked Questions

How does Palmier Pro calculate sync offsets between camera angles?

Palmier Pro calculates sync offsets through the syncMulticamMembers method, which extracts audio envelopes and uses an AudioSyncCorrelator to compute the optimal lag between the master source and each target. The system respects a configurable search window and minimum confidence threshold, falling back to time-code alignment when audio is unavailable. All offsets are then rebased so the earliest usable member starts at zero seconds.

What happens when you switch angles outside a member's coverage range?

When an angle switch request extends beyond a member's actual media coverage, the MulticamEngine automatically clamps the operation to the valid range. The switchClampsToAngleCoverage test verifies this behavior, and the Outcome structure reports which portions were clamped so the UI can provide feedback to the user.

How does the system handle undo operations for multicam groups?

Undo operations in Palmier Pro work at the group level rather than per-clip. The createMulticamGroup method registers an undo block that stores both the created clips and the group metadata, allowing the entire multicam structure to be removed or restored as a single atomic unit. This prevents the timeline from entering inconsistent states where some group members are deleted while others remain.

Can custom video transforms survive angle switches in Palmier Pro?

Yes, custom transforms and crops applied by users survive angle switches. When switchMulticamAngles processes a request, it preserves any user-applied framing on the program track. If a custom layout is applied, the engine uses a placement function to maintain these transforms; otherwise, it falls back to the default fit transform. The test userFramingSurvivesAngleSwitch explicitly confirms this preservation behavior.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}