How the Codex Plugin Persists Background Jobs: File-Based State Management

The Codex plugin persists background jobs as JSON files in the workspace's .codex/jobs directory, using synchronous file operations to guarantee immediate durability of running, completed, and failed job states.

The openai/codex-plugin-cc repository implements durable background job processing by writing every state change directly to the filesystem. When you initiate a tracked job, the system creates a dedicated JSON file containing metadata and a separate log file for console output, ensuring that job progress survives process crashes or unexpected terminations.

Job Storage Architecture and File Locations

Background job state is stored in the workspace’s .codex/jobs directory. The exact file paths are resolved by helper functions defined in plugins/codex/scripts/lib/state.mjs.

Each job receives two files:

• A job file (JSON) that stores metadata including status, timestamps, IDs, and result payloads
• A log file (text) that captures the job’s console output

This separation allows the system to append streaming output without rewriting the entire metadata structure on every line of text.

Creating and Updating Running Records

Initializing Job State

When a job starts, the runTrackedJob function (located in plugins/codex/scripts/lib/tracked-jobs.mjs) creates a running record containing the initial job metadata. This record is immediately written to disk via two operations:

1. writeJobFile persists the job-specific JSON file
2. upsertJob updates the global job index that tracks all jobs in the workspace

The system also checks for the CODEX_COMPANION_SESSION_ID environment variable (exposed by SESSION_ID_ENV) and optionally tags the job with the creating session’s ID for audit trails.

Real-Time Progress Updates

Long-running jobs report incremental progress through the createJobProgressUpdater factory. This returns a function that accepts normalized event objects containing fields like phase, threadId, and turnId.

When called, the updater performs a patch operation:

• It reads the existing job file
• Updates only the fields that have changed
• Writes the patched data back to both the job file and the global index

This minimizes disk I/O while keeping the persisted state current.

Finalizing Job States

Successful Completion

When a job runner returns successfully, runTrackedJob constructs a completed record. This JSON object includes:

• Final exitStatus (typically 0)
• The result payload
• rendered output string
• Completion timestamps

The final output block is appended to the log file via appendLogBlock, ensuring the complete console history is preserved alongside the metadata.

Error Handling and Failed Records

If the job runner throws an error, the catch block in runTrackedJob reads the existing job file (or falls back to the in-memory running record if the file is unavailable). It then writes a failed record containing:

• The error message
• Failure timestamps
• The final state transition

The global job index is updated synchronously to reflect the terminal failure state, ensuring that querying tools (such as those in plugins/codex/scripts/lib/job-control.mjs) see the correct status immediately.

Durability Guarantees Through Synchronous I/O

All file operations use synchronous Node.js APIs: fs.writeFileSync for JSON metadata and fs.appendFileSync for log output. This design choice guarantees that the kernel has flushed the data to disk before the function returns, eliminating the window of vulnerability that exists with asynchronous writes where a sudden process crash could lose the last state update.

While synchronous I/O blocks the event loop momentarily, the trade-off is justified for background jobs where state consistency is more critical than microsecond latency.

Code Examples

Starting a Tracked Job

import { runTrackedJob } from "./lib/tracked-jobs.mjs";

const job = {
  id: "my-job-123",
  workspaceRoot: "/path/to/workspace",
  // optional: title, logFile, etc.
};

async function myRunner() {
  // Do some work...
  return {
    exitStatus: 0,
    payload: { result: "ok" },
    rendered: "Job output",
    threadId: "t1",
    turnId: "turn42",
    summary: "Finished successfully"
  };
}

// Automatically persists running, then completed, records
await runTrackedJob(job, myRunner);

Updating Progress Mid-Execution

import { createJobProgressUpdater } from "./lib/tracked-jobs.mjs";

const progress = createJobProgressUpdater("/path/to/workspace", "my-job-123");

// Later in the job logic…
progress({ 
  phase: "processing", 
  threadId: "t1", 
  turnId: "turn5", 
  message: "Halfway there" 
});

Summary

• File-based persistence: Job metadata lives in .codex/jobs as JSON files, while console output streams to separate log files.
• Synchronous durability: The plugin uses fs.writeFileSync and fs.appendFileSync to ensure state is committed to disk immediately, surviving process crashes.
• State lifecycle: Jobs transition through running, completed, and failed records, each written atomically to the job file and global index.
• Progress tracking: The createJobProgressUpdater function patches only changed fields (phase, threadId, turnId) to minimize I/O while maintaining accuracy.
• Session correlation: The CODEX_COMPANION_SESSION_ID environment variable optionally tags jobs with their creating session for audit purposes.

Frequently Asked Questions

Where does the Codex plugin store background job data?

The plugin stores job data in the workspace’s .codex/jobs directory. Each job receives a JSON file for metadata (status, timestamps, IDs) and a separate log file for console output, with paths resolved by plugins/codex/scripts/lib/state.mjs.

What happens if a job fails or the process crashes?

Because the plugin uses synchronous file operations (fs.writeFileSync), the last state write is guaranteed to be on disk before the process can crash. If the job throws an error, the catch block writes a failed record with the error message and updates the global index, preserving the failure state for later inspection.

How does the plugin handle real-time progress updates?

Real-time updates are handled by the createJobProgressUpdater function in tracked-jobs.mjs. It accepts normalized events containing fields like phase, threadId, and turnId, then patches only those specific fields into the existing job file and global index rather than rewriting the entire document.

Can I identify which session created a specific job?

Yes. The plugin checks for the CODEX_COMPANION_SESSION_ID environment variable (defined as SESSION_ID_ENV) when creating jobs. If present, this value is stored in the job metadata, allowing you to correlate background jobs with the specific agent or user session that initiated them.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}