How Chunk Execution Isolates Video Rendering in Separate Processes in OpenMAIC

OpenMAIC isolates video rendering by splitting large render jobs into discrete chunks and executing each chunk in a detached Node.js child process with independent memory space and PID, preventing crashes or state corruption in one chunk from affecting the coordinator or other chunks.

The THU-MAIC/OpenMAIC repository implements a fault-tolerant video-export pipeline that leverages chunk execution to handle complex rendering workloads safely. By decomposing video generation into isolated OS processes, the system ensures robust parallelism while maintaining strict validation and tamper-proof assembly of the final output.

The Chunk Execution Architecture

OpenMAIC's chunk execution system operates through a coordinator-worker pattern where the parent process orchestrates rendering while child processes handle actual frame capture in sandboxed environments.

Configuration and Activation

The render service checks the RENDER_CHUNK_EXECUTION environment variable in render-service/src/config.ts (lines 60-66) to toggle process isolation. When set to 'true', the service bypasses the single-process pipeline and delegates execution to the chunk executor.

// render-service/src/config.ts
export const RENDER_CHUNK_EXECUTION = process.env.RENDER_CHUNK_EXECUTION === 'true';
export const MAX_PARALLEL_CHUNKS = parseInt(process.env.RENDER_MAX_PARALLEL_CHUNKS || '4', 10);

Immutable Render Planning

Before spawning workers, createRenderPlan in render-service/src/chunk-executor.ts (lines 60-78) constructs an immutable execution plan stored on disk. This plan contains frame ranges, output paths, and a cryptographic hash to detect tampering. Each chunk receives a deterministic identifier and validation checksum.

Process Isolation via Forking

The core isolation mechanism resides in renderChunkInTerminatedProcess within render-service/src/chunk-executor.ts (lines 16-24). For each chunk requiring rendering, the coordinator forks a new Node process with detached: true:

// Conceptual implementation from chunk-executor.ts
const child = fork(path.join(__dirname, 'chunk-worker.ts'), [], {
  detached: true,
  stdio: ['pipe', 'pipe', 'pipe', 'ipc']
});

The child process executes render-service/src/chunk-worker.ts (lines 9-15), which calls the HyperFrames producer's renderChunk API. Because the child is detached, it possesses:

• A unique Process ID (PID)
• Independent memory space
• Isolated signal handling

This architecture allows the parent to terminate individual workers via process.kill() without affecting other concurrent chunks or the coordinator's stability.

IPC and Result Validation

Upon completion, the child serializes a ChunkResult object containing the output path, frame count, and SHA-256 hash, transmitting it to the parent via process.send. The coordinator validates incoming messages against the immutable plan in render-service/src/chunk-executor.ts (lines 65-68), verifying hash consistency, frame counts, and capture modes before accepting results.

Fault Tolerance and Retry Logic

If a child crashes, the parent detects the exit event and consults the AbortController signal. The system implements a retry mechanism (up to two attempts) for transient failures, categorizing errors through ChunkExecutorError with specific codes including chunk_execution_failed and mismatched_chunk. When the request signal triggers an abort, the coordinator immediately terminates all child processes and halts the job.

Concurrency Control

The coordinator enforces resource limits defined by the selected resource profile through environment variables parsed in render-service/src/config.ts (lines 67-71):

• maxParallelChunks: Caps the number of concurrently running OS processes
• chunkWorkers: Controls capture worker threads inside each child process

These limits prevent resource exhaustion while maximizing throughput.

Deterministic Assembly

After all chunks complete, the coordinator verifies each segment by reading *.perf.json sidecar files in render-service/src/chunk-executor.ts (lines 26-44). This validation confirms SHA-256 hashes and plan integrity before invoking the producer's assemble step to concatenate chunk files into the final video.

Implementation Examples

Enabling Chunk Execution

Configure environment variables and invoke the high-level API:

// Enable process isolation
process.env.RENDER_CHUNK_EXECUTION = 'true';
process.env.RENDER_MAX_PARALLEL_CHUNKS = '4';

import { executeRenderChunks } from '@openmaic/render-service/src/chunk-executor.js';

const request = {
  projectDir: '/tmp/my-project',
  outputPath: '/tmp/output/video.mp4',
  options: { fps: 30, format: 'mp4', quality: 80 },
  chunkCount: 4,
  chunkWorkers: 2,
  maxParallelChunks: 4,
  onProgress: (p) => console.log(`Progress: ${(p.progress * 100).toFixed(1)}%`),
};

executeRenderChunks(request)
  .then(res => console.log('Output:', res.assembly.outputPath))
  .catch(err => console.error('Render failed:', err));

Manual Chunk Spawning

For low-level control, spawn individual chunks directly:

import { renderChunkInTerminatedProcess } from '@openmaic/render-service/src/chunk-executor.js';

const planDir = '/tmp/render-plan-abc';
const chunkIdx = 0;
const outPath = '/tmp/chunk-0.mp4';
const abortCtrl = new AbortController();

renderChunkInTerminatedProcess(planDir, chunkIdx, outPath, abortCtrl.signal)
  .then(result => console.log('Chunk rendered:', result))
  .catch(err => console.error('Chunk failed:', err));

Key Source Files

• render-service/src/chunk-executor.ts: Core coordinator implementing createRenderPlan, renderChunkInTerminatedProcess, and verifyChunkOutput. Manages child process lifecycle, validation, and video assembly.
• render-service/src/chunk-worker.ts: Worker entry point executed in detached child processes; interfaces with the HyperFrames producer to render individual chunks.
• render-service/src/config.ts: Parses RENDER_CHUNK_EXECUTION, maxParallelChunks, and resource profile limits from environment variables.
• render-service/src/main.ts: HTTP service entry point that routes render requests to the chunk executor when process isolation is enabled.

Summary

• Process Isolation: Each chunk runs in a detached Node.js child process with independent PID and memory space, preventing cross-chunk contamination.
• Immutable Plans: The createRenderPlan function generates cryptographically signed execution plans that detect tampering during validation.
• Robust Validation: The coordinator verifies SHA-256 hashes and frame counts via *.perf.json sidecars before assembly.
• Fault Tolerance: Automatic retry logic (up to two attempts) and specific error codes (chunk_execution_failed, mismatched_chunk) enable graceful degradation.
• Resource Management: maxParallelChunks and chunkWorkers environment variables cap concurrency to prevent system overload.

Frequently Asked Questions

What is the purpose of the RENDER_CHUNK_EXECUTION environment variable?

The RENDER_CHUNK_EXECUTION flag in render-service/src/config.ts controls whether the render service uses isolated child processes for chunk rendering. When set to true, the system activates the chunk executor architecture; when false (default), rendering occurs in a single process without isolation.

How does OpenMAIC prevent tampering with render chunks?

OpenMAIC generates an immutable render plan via createRenderPlan that includes a cryptographic hash of chunk parameters. When a child process completes, the coordinator validates the returned ChunkResult against this hash, frame count, and capture mode, rejecting any chunks that deviate from the original specification.

What happens if a chunk worker process crashes during rendering?

If a child process exits unexpectedly, the coordinator detects the exit event in render-service/src/chunk-executor.ts and checks the AbortController signal. Depending on configuration, it either retries the chunk (up to two times) or aborts the entire job, logging a ChunkExecutorError with the specific failure code to facilitate debugging.

How does the coordinator manage parallel chunk execution?

The coordinator respects maxParallelChunks (defined in the resource profile and parsed from environment variables) to limit the number of simultaneously forked child processes. Additional concurrency control via chunkWorkers manages internal capture threads within each isolated process, ensuring system resources remain within safe operational bounds.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}