How the Cangjie-Skill Checkpoint and Resume Mechanism Uses PIPELINE_STATE.md

The cangjie-skill pipeline persists progress in a plain markdown checklist—marking stages complete with [x] and skipping them on restart—to enable reliable resume after interruption.

The kangarooking/cangjie-skill repository implements a robust checkpoint and resume system for its seven-stage RIA-TV++ data processing pipeline. Rather than using a binary database or complex state management, the project leverages a human-readable markdown file—PIPELINE_STATE.md—to track which stages have finished and which remain pending.

How PIPELINE_STATE.md Stores Pipeline Progress

The checkpoint file contains a simple markdown task list representing all seven pipeline stages. Each stage appears as a checkbox item:

- [x] Stage 0 – Adler
- [ ] Stage 1 – Parallel Extract
- [ ] Stage 2 – RIA-Plus
- [ ] Stage 3 – Type Flow
- [ ] Stage 4 – Cangjie Optimizer
- [ ] Stage 5 – TVM Compilation
- [ ] Stage 6 – Runtime Deployment

State transitions are literal text changes: [ ] (unchecked) becomes [x] (checked) when a stage completes successfully. This design makes the checkpoint fully inspectable and manually editable without special tools.

The Checkpoint Logic: Writing Progress

When a stage finishes execution, the driver script updates PIPELINE_STATE.md immediately. This atomic write operation ensures that:

• A crash during stage N does not corrupt the checkpoint for stages 0 through N-1
• The file always reflects the last successfully completed stage
• No separate "commit" step is required

The update mechanism uses regex substitution to find the matching stage line and toggle its checkbox status. The file is rewritten in place with the new state.

The Resume Logic: Reading Progress

On pipeline startup, the driver performs three operations:

1. Initialize — If PIPELINE_STATE.md does not exist, create it with all stages unchecked
2. Parse — Scan the file for the pattern - [(.)] (.+) to extract stage names and completion status
3. Filter — Skip any stage where the captured group equals 'x', execute only unchecked stages

This resume behavior is idempotent: running the pipeline twice with no intervening changes executes nothing the second time. Conversely, resetting a single stage by editing its checkbox back to [ ] causes that stage and all subsequent stages to re-execute on the next run.

Practical Code Example

Below is a simplified Python implementation matching the actual logic in SKILL.md:

import pathlib
import re

STATE_FILE = pathlib.Path('PIPELINE_STATE.md')

STAGES = [
    'Stage 0 – Adler',
    'Stage 1 – Parallel Extract',
    'Stage 2 – RIA-Plus',
    'Stage 3 – Type Flow',
    'Stage 4 – Cangjie Optimizer',
    'Stage 5 – TVM Compilation',
    'Stage 6 – Runtime Deployment',
]

def load_or_create_state():
    """Create initial state file if missing."""
    if not STATE_FILE.exists():
        lines = [f'- [ ] {name}' for name in STAGES]
        STATE_FILE.write_text('\n'.join(lines))
    return STATE_FILE.read_text()

def parse_state(text):
    """Return dict mapping stage name to completion boolean."""
    pattern = r'- \[(.)\] (.+)'
    return {name: status == 'x' 
            for status, name in re.findall(pattern, text)}

def mark_complete(stage_name):
    """Update the checkpoint for a finished stage."""
    text = STATE_FILE.read_text()
    updated = re.sub(
        rf'(\- \[ \] {re.escape(stage_name)})',
        rf'- [x] {stage_name}',
        text
    )
    STATE_FILE.write_text(updated)

# Main resume-aware execution

state_text = load_or_create_state()
completion = parse_state(state_text)

for stage in STAGES:
    if not completion.get(stage, False):
        run_stage(stage)  # User's stage implementation

        mark_complete(stage)

Manual Intervention and Debugging

Because PIPELINE_STATE.md is plain text, developers can:

• Inspect progress with cat or any editor
• Force re-execution of a specific stage by changing [x] to [ ]
• Reset the entire pipeline by deleting the file or unchecking all stages
• Version control state alongside code to reproduce historical runs

This visibility is particularly valuable when debugging stage failures—simply revert the checkbox and re-run without modifying the driver script.

Key Files in the Checkpoint System

File	Purpose
PIPELINE_STATE.md	Checkpoint storage (markdown checklist)
SKILL.md	Primary driver implementing read/execute/update loop
README.en.md	Pipeline architecture and stage descriptions
01-stage0-adler.md through 07-stage6-runtime.md	Per-stage implementation documentation

Summary

• Checkpoint mechanism: Writes [x] to PIPELINE_STATE.md immediately after each stage succeeds
• Resume mechanism: Parses the markdown checklist on startup, skips completed stages automatically
• State format: Plain markdown task list enabling manual inspection and editing
• Driver location: Resume logic implemented in SKILL.md with simple regex-based file operations
• Recovery model: Crash-safe atomic writes; no external database dependencies

Frequently Asked Questions

What happens if I delete PIPELINE_STATE.md?

The pipeline will recreate the file with all stages unchecked on the next run, effectively resetting progress to the beginning. This is the standard way to force a full re-execution.

Can multiple pipelines run concurrently with the same state file?

No—concurrent execution is not supported. Each pipeline instance expects exclusive control of PIPELINE_STATE.md. Parallel runs would cause race conditions during the read-modify-write cycle.

How do I restart from a specific middle stage without re-running earlier stages?

Edit PIPELINE_STATE.md directly and change the desired restart stage from [x] to [ ]. Leave all prior stages checked. The next run will skip checked stages and begin execution at your target stage.

Is there a maximum pipeline size for this checkpoint approach?

The regex-based parsing scales linearly with stage count. The seven-stage RIA-TV++ pipeline processes instantly; hundreds of stages would remain performant given typical file sizes under 10KB.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}