# How the Checkpoint System in OpenMontage Enables Resumable State and Project Recovery

> Discover how OpenMontage's checkpoint system uses validated JSON to ensure resumable state and project recovery after interruptions or failures.

- Repository: [Calesthio/OpenMontage](https://github.com/calesthio/OpenMontage)
- Tags: internals
- Published: 2026-08-30

---

**The OpenMontage checkpoint system persists pipeline stage state as validated JSON files, enabling workflows to resume after interruptions and recover from failures through immutable history archives and project markers.**

OpenMontage implements a robust checkpoint system that transforms long-running video production pipelines into fault-tolerant, resumable workflows. By serializing stage metadata, artifacts, and decision logs to the project workspace according to the source code in `calesthio/OpenMontage`, the system ensures that hours of AI-driven processing are never lost to unexpected crashes or host shutdowns.

## Core Checkpoint Architecture

### Persisted Stage State

At the heart of the system is `write_checkpoint()` in **[`lib/checkpoint.py`](https://github.com/calesthio/OpenMontage/blob/main/lib/checkpoint.py)** (L22-L64), which creates a JSON file named `<project>/<stage>.json` after each pipeline stage completes or pauses. This checkpoint captures:

- **Stage metadata** including the `status` field (`in_progress`, `awaiting_human`, or `completed`)
- **Produced artifacts** dictionary containing references to generated assets
- **Timestamps** for execution tracking
- **Human-approval flags** (`human_approved=True/False`) for gated stages

The function validates every write against a JSON-Schema defined in **[`schemas/checkpoints/checkpoint.schema.json`](https://github.com/calesthio/OpenMontage/blob/main/schemas/checkpoints/checkpoint.schema.json)** via `_load_checkpoint_schema()` (L18-L23) and `jsonschema.validate` (L88-L92), ensuring data integrity before serialization.

### Project Identity Markers

Before any checkpoints are written, `init_project()` (L98-L112) establishes the canonical workspace structure. It creates the directory layout (`artifacts/`, `assets/`, `renders/`) and writes a **project marker** file ([`project.json`](https://github.com/calesthio/OpenMontage/blob/main/project.json)) containing immutable identity data: `project_id`, `title`, `pipeline_type`, and optional `style_playbook`. This marker guarantees that subsequent runs can locate the correct workspace even after system crashes or directory moves.

## Resumable Pipeline Execution

### Reading and Resuming State

When the orchestrator restarts, it reconstructs project progress through two critical functions:

- **`read_checkpoint()`** (L66-L78): Loads and validates a specific stage's JSON checkpoint from the workspace
- **`get_latest_checkpoint()`** (L80-L99): Determines the most recent valid checkpoint across all stages to establish the restart point

These functions enable the pipeline to skip completed stages and resume from the next unfinished task, eliminating redundant computation.

### Progress Navigation

The checkpoint system provides utilities for workflow traversal:

- **`get_completed_stages()`** (L102-L108): Scans the pipeline manifest and returns all stages with `status: "completed"`
- **`get_next_stage()`** (L122-L130): Calculates the next uncompleted stage based on the ordered list from `get_pipeline_stages()`

These methods allow the orchestrator to make deterministic decisions about execution order without manual intervention.

## Data Integrity and Audit Trails

### Validation and Gate Policies

The checkpoint system enforces **gate policies** for human-in-the-loop workflows. Before writing a checkpoint with `status: "completed"`, the code consults `_stage_requires_approval` from the pipeline manifest. If a gated stage lacks `human_approved=True`, `write_checkpoint()` raises a `CheckpointValidationError` (L66-L94), preventing premature stage closure.

### Checkpoint Archiving

To support recovery and compliance auditing, `_archive_superseded_checkpoint()` (L49-L61) copies existing checkpoint files to a `history/` subdirectory before overwriting them. This creates an immutable timeline of stage changes, enabling replay harnesses and forensic analysis of pipeline evolution.

### Decision Logging

When checkpoints contain `decision_log` entries, the system merges them into a project-level **[`decision_log.json`](https://github.com/calesthio/OpenMontage/blob/main/decision_log.json)** and injects references (`decision_log_ref`) into related artifacts like `proposal_packet` or `render_report` (L27-L35). This preserves the rationale behind creative choices for downstream consumers and regulatory audits.

## Practical Implementation

Initialize a project workspace and write a checkpoint after the research stage:

```python
from pathlib import Path
from lib.checkpoint import init_project, write_checkpoint, read_checkpoint

# Create workspace with project marker

pipeline_dir = Path("/data/projects")
proj_dir = init_project(
    project_id="proj_123",
    title="Demo Video",
    pipeline_type="explainer",
    pipeline_dir=pipeline_dir,
)

# Persist completed research stage

research_artifacts = {
    "research_brief": {"summary": "Target audience analysis complete"},
    "decision_log": {"decisions": [{"decision_id": "d1", "text": "Approve direction"}]},
}
write_checkpoint(
    pipeline_dir,
    project_id="proj_123",
    stage="research",
    status="completed",
    artifacts=research_artifacts,
    pipeline_type="explainer",
    human_approved=True,
)

```

Resume processing by reading the checkpoint and determining the next stage:

```python
from lib.checkpoint import read_checkpoint, get_next_stage

# Load existing state

cp = read_checkpoint(pipeline_dir, "proj_123", "research")
print(cp["status"])  # → "completed"

# Identify continuation point

next_stage = get_next_stage(pipeline_dir, "proj_123", pipeline_type="explainer")
print(next_stage)  # e.g., "proposal"

```

## Key Files

| File | Purpose |
|------|---------|
| **[`lib/checkpoint.py`](https://github.com/calesthio/OpenMontage/blob/main/lib/checkpoint.py)** | Core checkpoint writer/reader, validation, archiving, and stage-order utilities |
| **[`schemas/checkpoints/checkpoint.schema.json`](https://github.com/calesthio/OpenMontage/blob/main/schemas/checkpoints/checkpoint.schema.json)** | JSON-Schema defining required fields and types for checkpoint validation |
| **[`lib/pipeline_loader.py`](https://github.com/calesthio/OpenMontage/blob/main/lib/pipeline_loader.py)** | Loads pipeline manifests; resolves stage order and gate policies for checkpoint validation |
| **[`tests/lib/test_checkpoint_prerequisites.py`](https://github.com/calesthio/OpenMontage/blob/main/tests/lib/test_checkpoint_prerequisites.py)** | Unit tests verifying prerequisite enforcement and checkpoint integrity |
| **[`tests/qa/test_08_end_to_end.py`](https://github.com/calesthio/OpenMontage/blob/main/tests/qa/test_08_end_to_end.py)** | End-to-end tests demonstrating resumable pipelines across multiple stages |

## Summary

- **Atomic persistence**: `write_checkpoint()` serializes stage state to validated JSON, capturing artifacts, status, and approval flags.
- **Fault-tolerant resumption**: `read_checkpoint()` and `get_latest_checkpoint()` reconstruct pipeline progress after interruptions.
- **Immutable history**: The `_archive_superseded_checkpoint()` mechanism preserves prior checkpoint versions in `history/` for audit trails.
- **Project recovery**: The [`project.json`](https://github.com/calesthio/OpenMontage/blob/main/project.json) marker file maintains immutable identity data, ensuring workspaces remain locatable across system restarts.
- **Gate enforcement**: Built-in validation prevents stages marked as `completed` from being written without required human approvals.

## Frequently Asked Questions

### How does OpenMontage recover from a corrupted checkpoint?

When `read_checkpoint()` encounters invalid JSON or schema violations, it raises a `CheckpointValidationError`, triggering the orchestrator to fall back to the canonical stage list defined in the pipeline manifest. This graceful degradation allows recovery by re-executing only the affected stage rather than the entire pipeline.

### Can the checkpoint system handle long-running stages that span multiple days?

Yes. The system writes checkpoints with `status: "in_progress"` at configurable intervals during lengthy operations. Since checkpoints are atomic file writes validated against JSON-Schema, the pipeline can resume from the last persisted state even if the host shuts down or the process terminates unexpectedly days into execution.

### What happens when a human approval gate rejects a stage?

If `_stage_requires_approval()` detects a gated stage without `human_approved=True`, `write_checkpoint()` raises a `CheckpointValidationError`. The stage retains `status: "awaiting_human"` until explicit approval is recorded, preventing downstream stages from executing prematurely and ensuring compliance with human-in-the-loop requirements.

### Where are historical checkpoint versions stored?

Before overwriting an existing checkpoint, `_archive_superseded_checkpoint()` (L49-L61) copies the previous version to a `history/` subdirectory within the project workspace. These archived files provide an immutable audit trail of stage evolution and enable rollback capabilities for debugging or compliance reviews.