How JS and Python Process Extensions Are Executed Differently in Modly's ProcessRunner

Modly's ProcessRunner uses persistent Node worker threads for JavaScript extensions and spawns fresh Python subprocesses for each Python extension run, with distinct message protocols and lifecycle management for each language.

In the lightningpixel/modly codebase, the ProcessRunner architecture provides a unified interface for executing extension code while handling the fundamental differences between JavaScript and Python runtimes. Understanding these execution differences is critical for extension developers optimizing performance or debugging cross-language integrations.

Architecture Overview: IProcessRunner Interface

Both JavaScript and Python extensions implement a common IProcessRunner interface, registered in a singleton Map<string, IProcessRunner> registry. Callers retrieve runners via getProcessRunner() for JavaScript or getPythonProcessRunner() for Python, ensuring each extension receives a dedicated runner instance.

The key distinction lies in the concrete implementations:

• JavaScript: ProcessRunner class—wraps a persistent Node worker thread
• Python: PythonProcessRunner class—spawns a fresh subprocess per execution

JavaScript Extension Execution: Persistent Worker Threads

Worker Creation and Code Loading

JavaScript extensions run inside Node worker threads defined by the WORKER_CODE string in electron/main/process-runner.ts (lines 8-45). The worker establishes a custom require function pointing to the extension's node_modules:

// From process-runner.ts lines 8-45 - WORKER_CODE template
const WORKER_CODE = `
const { parentPort, workerData } = require('worker_threads');
const Module = require('module');
const path = require('path');

// Create custom require for extension's node_modules
const require_ext = Module.createRequire(
  path.join(workerData.extDir, 'package.json')
);

// Load and execute the extension's entry file
const processor = require_ext(path.join(workerData.extDir, workerData.entry));
// ...
`;

The ensureReady() method (lines 89-118) creates this worker on first call and reuses it for subsequent run() invocations:

// From process-runner.ts lines 89-118
private async ensureReady(): Promise<void> {
  if (this.worker) return;
  
  this.worker = new Worker(WORKER_CODE, {
    workerData: {
      extDir: this.extDir,
      entry: this.entry,
      workspaceDir: this.workspaceDir,
      tmpDir: this.tmpDir
    },
    eval: true
  });
  
  // Wait for 'ready' message from worker
  await new Promise<void>((resolve, reject) => {
    this.worker!.once('message', (msg) => {
      if (msg.type === 'ready') resolve();
      else reject(new Error(`Unexpected init message: ${msg.type}`));
    });
  });
}

Message Protocol

JavaScript extensions communicate via worker_threads messaging:

Message Type	Direction	Purpose
ready	Worker → Parent	Signals initialization complete
log	Worker → Parent	Extension log output
progress	Worker → Parent	Progress updates (pct, label)
done	Worker → Parent	Execution completed with result
error	Worker → Parent	Execution failed with error

The run() method (lines 120-147) handles these messages, resolving or rejecting the returned Promise based on done or error responses.

Python Extension Execution: Fresh Subprocess Per Run

Process Spawning and Input Protocol

Python extensions cannot execute in Node threads, so PythonProcessRunner launches a separate Python interpreter for each run() call. As documented in lines 56-60 of process-runner.ts:

// Python protocol: single JSON line on stdin, line-delimited JSON on stdout
// Messages: {type: 'progress'|'log'|'done'|'error', ...}
// Non-JSON lines are treated as log output

The run() implementation (lines 81-95) spawns the process and writes input data:

// From process-runner.ts lines 81-95
async run(
  processInput: ProcessInput,
  params: Record<string, unknown>,
  onProgress: (pct: number, label: string) => void,
  onLog: (message: string) => void
): Promise<unknown> {
  const pythonProcess = spawn(this.pythonExe, [
    path.join(this.extDir, this.entry)
  ], {
    cwd: this.workspaceDir,
    env: { ...process.env, MODLY_TMP_DIR: this.tmpDir }
  });

  // Send input as single JSON line on stdin
  pythonProcess.stdin.write(JSON.stringify({ input: processInput, params }) + '\n');
  pythonProcess.stdin.end();
  
  // ... stdout parsing follows
}

Output Parsing and Message Handling

The Python runner parses line-delimited JSON from stdout (lines 99-124):

// From process-runner.ts lines 99-124
const rl = createInterface({ input: pythonProcess.stdout });

for await (const line of rl) {
  let msg: PythonMessage;
  try {
    msg = JSON.parse(line);
  } catch {
    // Non-JSON lines treated as log output
    onLog(line);
    continue;
  }

  switch (msg.type) {
    case 'progress':
      onProgress(msg.pct, msg.label);
      break;
    case 'log':
      onLog(msg.message);
      break;
    case 'done':
      resolve(msg.result);
      break;
    case 'error':
      reject(new Error(msg.error));
      break;
  }
}

Key Execution Differences Summary

Aspect	JavaScript (ProcessRunner)	Python (PythonProcessRunner)
Underlying mechanism	Node worker_threads	child_process.spawn()
Process lifecycle	Single persistent worker, reused across calls	New process spawned per run() call
Startup cost	One-time worker creation; amortized over calls	Paid on every execution
Termination behavior	terminate() stops worker thread	terminate() is no-op (process already exited)
Module isolation	Custom require with extension's node_modules	Standard Python import path
Input channel	workerData + structured messages	JSON line on stdin
Output channel	parentPort.postMessage()	Line-delimited JSON on stdout
Error sources	Worker message or uncaught exception	JSON error message or non-zero exit code

Practical Usage Examples

Running a JavaScript Extension

import { getProcessRunner } from './process-runner';

// Acquire or create runner (worker created lazily on first run)
const jsRunner = getProcessRunner(
  'image-processor',           // extension ID
  '/extensions/image-proc',    // extDir
  'dist/processor.js',         // entry file
  '/workspace/project',        // workspace directory
  '/tmp/modly-run-123'         // temporary directory
);

// Execute (reuses existing worker after first call)
const result = await jsRunner.run(
  { imagePath: '/workspace/input.png' },
  { quality: 85, format: 'webp' },
  (pct, label) => console.log(`${pct}%: ${label}`),
  (msg) => console.log(`[EXT] ${msg}`)
);

Running a Python Extension

import { getPythonProcessRunner, getExtPythonExe } from './process-runner';

// Locate Python in extension's virtual environment
const pythonExe = getExtPythonExe('/extensions/ml-model');
if (!pythonExe) throw new Error('Python not found in extension');

// Acquire runner (no persistent state between runs)
const pyRunner = getPythonProcessRunner(
  'ml-inference',
  pythonExe,
  '/extensions/ml-model',
  'inference.py',
  '/workspace/project',
  '/tmp/modly-run-456'
);

// Each run spawns fresh Python process
const result = await pyRunner.run(
  { data: [0.1, 0.2, 0.3] },
  { model: 'v2', batchSize: 32 },
  (pct, label) => console.log(`${pct}%: ${label}`),
  (msg) => console.log(`[PY] ${msg}`)
);

Resource Cleanup

import { 
  terminateProcessRunner, 
  terminateAllProcessRunners 
} from './process-runner';

// Release specific JS worker (frees memory, terminates thread)
terminateProcessRunner('image-processor');

// Bulk cleanup on application shutdown
terminateAllProcessRunners();

Source File Reference

File	Purpose
electron/main/process-runner.ts	Core ProcessRunner and PythonProcessRunner implementations with worker code, registry, and lifecycle management
electron/main/ipc-handlers.ts	IPC layer integration showing runner consumption patterns
electron/main/python-bridge.ts	getVenvPythonExe() and other Python environment utilities
electron/main/extension-path-guard.ts	Extension path validation before runner instantiation

Summary

• JavaScript extensions execute in persistent Node worker threads that are created once and reused, minimizing startup overhead through the ProcessRunner class
• Python extensions spawn a fresh subprocess on every run() call via PythonProcessRunner, with input via stdin JSON and output via line-delimited stdout JSON
• The message protocols differ architecturally: JavaScript uses structured worker_threads messages; Python uses stream-based JSON lines
• Termination behavior reflects lifecycle differences: JavaScript workers respond to terminate(); Python processes self-terminate and ignore cleanup calls
• Both implementations share the IProcessRunner interface and registry pattern in electron/main/process-runner.ts, providing unified caller semantics despite divergent internals

Frequently Asked Questions

Why doesn't Modly reuse Python processes like JavaScript workers?

Python's Global Interpreter Lock (GIL) and lack of native Node.js integration make persistent workers impractical. Fresh subprocesses prevent cross-run state contamination and align with Python's standard process model. The overhead is acceptable given typical Python extension use cases (heavy computation, ML inference) where execution time dominates startup cost.

How does error handling differ between JS and Python extensions?

JavaScript errors surface through worker.on('error') events or error messages from the worker thread, captured in run()'s Promise rejection. Python errors may appear as JSON error messages on stdout, non-zero exit codes, or process spawn failures—all converted to rejected Promises in PythonProcessRunner.

Can I control which Python executable runs my extension?

Yes. The getExtPythonExe() helper in python-bridge.ts locates Python within extension virtual environments. Pass an explicit pythonExe path to getPythonProcessRunner(), or rely on automatic detection that checks venv/bin/python (Unix) or venv\Scripts\python.exe (Windows) relative to the extension directory.

What happens if a JavaScript extension crashes?

Uncaught exceptions in the worker thread trigger the error event handler in ProcessRunner, rejecting the active run() Promise. The worker remains in a failed state; subsequent calls to run() will attempt to recreate the worker via ensureReady(). Use terminateProcessRunner() followed by fresh getProcessRunner() to force complete reset.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}