# How JS and Python Process Extensions Are Executed Differently in Modly's ProcessRunner > Discover how Modly's ProcessRunner executes JS and Python extensions differently. Learn about persistent Node workers for JS and fresh subprocesses for Python, with distinct protocols and lifecycles. - Repository: [lightningpixel/modly](https://github.com/lightningpixel/modly) - Tags: internals - Published: 2026-08-15 --- **Modly's ProcessRunner uses persistent Node worker threads for JavaScript extensions and spawns fresh Python subprocesses for each Python extension run, with distinct message protocols and lifecycle management for each language.** In the [lightningpixel/modly](https://github.com/lightningpixel/modly) codebase, the `ProcessRunner` architecture provides a unified interface for executing extension code while handling the fundamental differences between JavaScript and Python runtimes. Understanding these execution differences is critical for extension developers optimizing performance or debugging cross-language integrations. ## Architecture Overview: IProcessRunner Interface Both JavaScript and Python extensions implement a common `IProcessRunner` interface, registered in a singleton `Map` registry. Callers retrieve runners via `getProcessRunner()` for JavaScript or `getPythonProcessRunner()` for Python, ensuring each extension receives a dedicated runner instance. The key distinction lies in the concrete implementations: - **JavaScript**: `ProcessRunner` class—wraps a **persistent Node worker thread** - **Python**: `PythonProcessRunner` class—spawns a **fresh subprocess per execution** ## JavaScript Extension Execution: Persistent Worker Threads ### Worker Creation and Code Loading JavaScript extensions run inside Node worker threads defined by the `WORKER_CODE` string in [`electron/main/process-runner.ts`](https://github.com/lightningpixel/modly/blob/main/electron/main/process-runner.ts) (lines 8-45). The worker establishes a custom `require` function pointing to the extension's `node_modules`: ```typescript // From process-runner.ts lines 8-45 - WORKER_CODE template const WORKER_CODE = ` const { parentPort, workerData } = require('worker_threads'); const Module = require('module'); const path = require('path'); // Create custom require for extension's node_modules const require_ext = Module.createRequire( path.join(workerData.extDir, 'package.json') ); // Load and execute the extension's entry file const processor = require_ext(path.join(workerData.extDir, workerData.entry)); // ... `; ``` The `ensureReady()` method (lines 89-118) creates this worker on first call and reuses it for subsequent `run()` invocations: ```typescript // From process-runner.ts lines 89-118 private async ensureReady(): Promise { if (this.worker) return; this.worker = new Worker(WORKER_CODE, { workerData: { extDir: this.extDir, entry: this.entry, workspaceDir: this.workspaceDir, tmpDir: this.tmpDir }, eval: true }); // Wait for 'ready' message from worker await new Promise((resolve, reject) => { this.worker!.once('message', (msg) => { if (msg.type === 'ready') resolve(); else reject(new Error(`Unexpected init message: ${msg.type}`)); }); }); } ``` ### Message Protocol JavaScript extensions communicate via `worker_threads` messaging: | Message Type | Direction | Purpose | |-------------|-----------|---------| | `ready` | Worker → Parent | Signals initialization complete | | `log` | Worker → Parent | Extension log output | | `progress` | Worker → Parent | Progress updates (`pct`, `label`) | | `done` | Worker → Parent | Execution completed with `result` | | `error` | Worker → Parent | Execution failed with `error` | The `run()` method (lines 120-147) handles these messages, resolving or rejecting the returned Promise based on `done` or `error` responses. ## Python Extension Execution: Fresh Subprocess Per Run ### Process Spawning and Input Protocol Python extensions cannot execute in Node threads, so `PythonProcessRunner` launches a separate Python interpreter for each `run()` call. As documented in lines 56-60 of [`process-runner.ts`](https://github.com/lightningpixel/modly/blob/main/process-runner.ts): ```typescript // Python protocol: single JSON line on stdin, line-delimited JSON on stdout // Messages: {type: 'progress'|'log'|'done'|'error', ...} // Non-JSON lines are treated as log output ``` The `run()` implementation (lines 81-95) spawns the process and writes input data: ```typescript // From process-runner.ts lines 81-95 async run( processInput: ProcessInput, params: Record, onProgress: (pct: number, label: string) => void, onLog: (message: string) => void ): Promise { const pythonProcess = spawn(this.pythonExe, [ path.join(this.extDir, this.entry) ], { cwd: this.workspaceDir, env: { ...process.env, MODLY_TMP_DIR: this.tmpDir } }); // Send input as single JSON line on stdin pythonProcess.stdin.write(JSON.stringify({ input: processInput, params }) + '\n'); pythonProcess.stdin.end(); // ... stdout parsing follows } ``` ### Output Parsing and Message Handling The Python runner parses line-delimited JSON from stdout (lines 99-124): ```typescript // From process-runner.ts lines 99-124 const rl = createInterface({ input: pythonProcess.stdout }); for await (const line of rl) { let msg: PythonMessage; try { msg = JSON.parse(line); } catch { // Non-JSON lines treated as log output onLog(line); continue; } switch (msg.type) { case 'progress': onProgress(msg.pct, msg.label); break; case 'log': onLog(msg.message); break; case 'done': resolve(msg.result); break; case 'error': reject(new Error(msg.error)); break; } } ``` ## Key Execution Differences Summary | Aspect | JavaScript (`ProcessRunner`) | Python (`PythonProcessRunner`) | |--------|------------------------------|--------------------------------| | **Underlying mechanism** | Node `worker_threads` | `child_process.spawn()` | | **Process lifecycle** | Single persistent worker, reused across calls | New process spawned per `run()` call | | **Startup cost** | One-time worker creation; amortized over calls | Paid on every execution | | **Termination behavior** | `terminate()` stops worker thread | `terminate()` is no-op (process already exited) | | **Module isolation** | Custom `require` with extension's `node_modules` | Standard Python import path | | **Input channel** | `workerData` + structured messages | JSON line on `stdin` | | **Output channel** | `parentPort.postMessage()` | Line-delimited JSON on `stdout` | | **Error sources** | Worker message or uncaught exception | JSON error message or non-zero exit code | ## Practical Usage Examples ### Running a JavaScript Extension ```typescript import { getProcessRunner } from './process-runner'; // Acquire or create runner (worker created lazily on first run) const jsRunner = getProcessRunner( 'image-processor', // extension ID '/extensions/image-proc', // extDir 'dist/processor.js', // entry file '/workspace/project', // workspace directory '/tmp/modly-run-123' // temporary directory ); // Execute (reuses existing worker after first call) const result = await jsRunner.run( { imagePath: '/workspace/input.png' }, { quality: 85, format: 'webp' }, (pct, label) => console.log(`${pct}%: ${label}`), (msg) => console.log(`[EXT] ${msg}`) ); ``` ### Running a Python Extension ```typescript import { getPythonProcessRunner, getExtPythonExe } from './process-runner'; // Locate Python in extension's virtual environment const pythonExe = getExtPythonExe('/extensions/ml-model'); if (!pythonExe) throw new Error('Python not found in extension'); // Acquire runner (no persistent state between runs) const pyRunner = getPythonProcessRunner( 'ml-inference', pythonExe, '/extensions/ml-model', 'inference.py', '/workspace/project', '/tmp/modly-run-456' ); // Each run spawns fresh Python process const result = await pyRunner.run( { data: [0.1, 0.2, 0.3] }, { model: 'v2', batchSize: 32 }, (pct, label) => console.log(`${pct}%: ${label}`), (msg) => console.log(`[PY] ${msg}`) ); ``` ### Resource Cleanup ```typescript import { terminateProcessRunner, terminateAllProcessRunners } from './process-runner'; // Release specific JS worker (frees memory, terminates thread) terminateProcessRunner('image-processor'); // Bulk cleanup on application shutdown terminateAllProcessRunners(); ``` ## Source File Reference | File | Purpose | |------|---------| | [`electron/main/process-runner.ts`](https://github.com/lightningpixel/modly/blob/main/electron/main/process-runner.ts) | Core `ProcessRunner` and `PythonProcessRunner` implementations with worker code, registry, and lifecycle management | | [`electron/main/ipc-handlers.ts`](https://github.com/lightningpixel/modly/blob/main/electron/main/ipc-handlers.ts) | IPC layer integration showing runner consumption patterns | | [`electron/main/python-bridge.ts`](https://github.com/lightningpixel/modly/blob/main/electron/main/python-bridge.ts) | `getVenvPythonExe()` and other Python environment utilities | | [`electron/main/extension-path-guard.ts`](https://github.com/lightningpixel/modly/blob/main/electron/main/extension-path-guard.ts) | Extension path validation before runner instantiation | ## Summary - **JavaScript extensions** execute in **persistent Node worker threads** that are created once and reused, minimizing startup overhead through the `ProcessRunner` class - **Python extensions** spawn a **fresh subprocess on every `run()` call** via `PythonProcessRunner`, with input via `stdin` JSON and output via line-delimited `stdout` JSON - The **message protocols differ architecturally**: JavaScript uses structured `worker_threads` messages; Python uses stream-based JSON lines - **Termination behavior reflects lifecycle differences**: JavaScript workers respond to `terminate()`; Python processes self-terminate and ignore cleanup calls - Both implementations share the `IProcessRunner` interface and registry pattern in [`electron/main/process-runner.ts`](https://github.com/lightningpixel/modly/blob/main/electron/main/process-runner.ts), providing unified caller semantics despite divergent internals ## Frequently Asked Questions ### Why doesn't Modly reuse Python processes like JavaScript workers? Python's Global Interpreter Lock (GIL) and lack of native Node.js integration make persistent workers impractical. Fresh subprocesses prevent cross-run state contamination and align with Python's standard process model. The overhead is acceptable given typical Python extension use cases (heavy computation, ML inference) where execution time dominates startup cost. ### How does error handling differ between JS and Python extensions? JavaScript errors surface through `worker.on('error')` events or `error` messages from the worker thread, captured in `run()`'s Promise rejection. Python errors may appear as JSON `error` messages on stdout, non-zero exit codes, or process spawn failures—all converted to rejected Promises in `PythonProcessRunner`. ### Can I control which Python executable runs my extension? Yes. The `getExtPythonExe()` helper in [`python-bridge.ts`](https://github.com/lightningpixel/modly/blob/main/python-bridge.ts) locates Python within extension virtual environments. Pass an explicit `pythonExe` path to `getPythonProcessRunner()`, or rely on automatic detection that checks `venv/bin/python` (Unix) or `venv\Scripts\python.exe` (Windows) relative to the extension directory. ### What happens if a JavaScript extension crashes? Uncaught exceptions in the worker thread trigger the `error` event handler in `ProcessRunner`, rejecting the active `run()` Promise. The worker remains in a failed state; subsequent calls to `run()` will attempt to recreate the worker via `ensureReady()`. Use `terminateProcessRunner()` followed by fresh `getProcessRunner()` to force complete reset.