How the Checkpoint System in OpenMontage Enables Resumable State and Project Recovery

The OpenMontage checkpoint system persists pipeline stage state as validated JSON files, enabling workflows to resume after interruptions and recover from failures through immutable history archives and project markers.

OpenMontage implements a robust checkpoint system that transforms long-running video production pipelines into fault-tolerant, resumable workflows. By serializing stage metadata, artifacts, and decision logs to the project workspace according to the source code in calesthio/OpenMontage, the system ensures that hours of AI-driven processing are never lost to unexpected crashes or host shutdowns.

Core Checkpoint Architecture

Persisted Stage State

At the heart of the system is write_checkpoint() in lib/checkpoint.py (L22-L64), which creates a JSON file named <project>/<stage>.json after each pipeline stage completes or pauses. This checkpoint captures:

• Stage metadata including the status field (in_progress, awaiting_human, or completed)
• Produced artifacts dictionary containing references to generated assets
• Timestamps for execution tracking
• Human-approval flags (human_approved=True/False) for gated stages

The function validates every write against a JSON-Schema defined in schemas/checkpoints/checkpoint.schema.json via _load_checkpoint_schema() (L18-L23) and jsonschema.validate (L88-L92), ensuring data integrity before serialization.

Project Identity Markers

Before any checkpoints are written, init_project() (L98-L112) establishes the canonical workspace structure. It creates the directory layout (artifacts/, assets/, renders/) and writes a project marker file (project.json) containing immutable identity data: project_id, title, pipeline_type, and optional style_playbook. This marker guarantees that subsequent runs can locate the correct workspace even after system crashes or directory moves.

Resumable Pipeline Execution

Reading and Resuming State

When the orchestrator restarts, it reconstructs project progress through two critical functions:

• read_checkpoint() (L66-L78): Loads and validates a specific stage's JSON checkpoint from the workspace
• get_latest_checkpoint() (L80-L99): Determines the most recent valid checkpoint across all stages to establish the restart point

These functions enable the pipeline to skip completed stages and resume from the next unfinished task, eliminating redundant computation.

Progress Navigation

The checkpoint system provides utilities for workflow traversal:

• get_completed_stages() (L102-L108): Scans the pipeline manifest and returns all stages with status: "completed"
• get_next_stage() (L122-L130): Calculates the next uncompleted stage based on the ordered list from get_pipeline_stages()

These methods allow the orchestrator to make deterministic decisions about execution order without manual intervention.

Data Integrity and Audit Trails

Validation and Gate Policies

The checkpoint system enforces gate policies for human-in-the-loop workflows. Before writing a checkpoint with status: "completed", the code consults _stage_requires_approval from the pipeline manifest. If a gated stage lacks human_approved=True, write_checkpoint() raises a CheckpointValidationError (L66-L94), preventing premature stage closure.

Checkpoint Archiving

To support recovery and compliance auditing, _archive_superseded_checkpoint() (L49-L61) copies existing checkpoint files to a history/ subdirectory before overwriting them. This creates an immutable timeline of stage changes, enabling replay harnesses and forensic analysis of pipeline evolution.

Decision Logging

When checkpoints contain decision_log entries, the system merges them into a project-level decision_log.json and injects references (decision_log_ref) into related artifacts like proposal_packet or render_report (L27-L35). This preserves the rationale behind creative choices for downstream consumers and regulatory audits.

Practical Implementation

Initialize a project workspace and write a checkpoint after the research stage:

from pathlib import Path
from lib.checkpoint import init_project, write_checkpoint, read_checkpoint

# Create workspace with project marker

pipeline_dir = Path("/data/projects")
proj_dir = init_project(
    project_id="proj_123",
    title="Demo Video",
    pipeline_type="explainer",
    pipeline_dir=pipeline_dir,
)

# Persist completed research stage

research_artifacts = {
    "research_brief": {"summary": "Target audience analysis complete"},
    "decision_log": {"decisions": [{"decision_id": "d1", "text": "Approve direction"}]},
}
write_checkpoint(
    pipeline_dir,
    project_id="proj_123",
    stage="research",
    status="completed",
    artifacts=research_artifacts,
    pipeline_type="explainer",
    human_approved=True,
)

Resume processing by reading the checkpoint and determining the next stage:

from lib.checkpoint import read_checkpoint, get_next_stage

# Load existing state

cp = read_checkpoint(pipeline_dir, "proj_123", "research")
print(cp["status"])  # → "completed"

# Identify continuation point

next_stage = get_next_stage(pipeline_dir, "proj_123", pipeline_type="explainer")
print(next_stage)  # e.g., "proposal"

Key Files

File	Purpose
lib/checkpoint.py	Core checkpoint writer/reader, validation, archiving, and stage-order utilities
schemas/checkpoints/checkpoint.schema.json	JSON-Schema defining required fields and types for checkpoint validation
lib/pipeline_loader.py	Loads pipeline manifests; resolves stage order and gate policies for checkpoint validation
tests/lib/test_checkpoint_prerequisites.py	Unit tests verifying prerequisite enforcement and checkpoint integrity
tests/qa/test_08_end_to_end.py	End-to-end tests demonstrating resumable pipelines across multiple stages

Summary

• Atomic persistence: write_checkpoint() serializes stage state to validated JSON, capturing artifacts, status, and approval flags.
• Fault-tolerant resumption: read_checkpoint() and get_latest_checkpoint() reconstruct pipeline progress after interruptions.
• Immutable history: The _archive_superseded_checkpoint() mechanism preserves prior checkpoint versions in history/ for audit trails.
• Project recovery: The project.json marker file maintains immutable identity data, ensuring workspaces remain locatable across system restarts.
• Gate enforcement: Built-in validation prevents stages marked as completed from being written without required human approvals.

Frequently Asked Questions

How does OpenMontage recover from a corrupted checkpoint?

When read_checkpoint() encounters invalid JSON or schema violations, it raises a CheckpointValidationError, triggering the orchestrator to fall back to the canonical stage list defined in the pipeline manifest. This graceful degradation allows recovery by re-executing only the affected stage rather than the entire pipeline.

Can the checkpoint system handle long-running stages that span multiple days?

Yes. The system writes checkpoints with status: "in_progress" at configurable intervals during lengthy operations. Since checkpoints are atomic file writes validated against JSON-Schema, the pipeline can resume from the last persisted state even if the host shuts down or the process terminates unexpectedly days into execution.

What happens when a human approval gate rejects a stage?

If _stage_requires_approval() detects a gated stage without human_approved=True, write_checkpoint() raises a CheckpointValidationError. The stage retains status: "awaiting_human" until explicit approval is recorded, preventing downstream stages from executing prematurely and ensuring compliance with human-in-the-loop requirements.

Where are historical checkpoint versions stored?

Before overwriting an existing checkpoint, _archive_superseded_checkpoint() (L49-L61) copies the previous version to a history/ subdirectory within the project workspace. These archived files provide an immutable audit trail of stage evolution and enable rollback capabilities for debugging or compliance reviews.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}