How Process Extensions Communicate Progress and Logs to the Modly Main App

Modly process extensions communicate back to the main app through a context object exposing log() and progress() functions, which serialize messages to the ProcessRunner via worker-thread IPC or stdout JSON lines.

Modly's extension system isolates user code for security and stability. JavaScript extensions run in Node.js worker threads, Python extensions in subprocesses. Both use the same message schema to stream progress and logs to the main Electron process. This article examines the implementation in electron/main/process-runner.ts and shows how extension authors and main-process developers coordinate this communication.

The Extension Context: log() and progress()

Every process extension receives a context object containing two callable functions. Extension authors use these to emit status updates without worrying about underlying transport mechanisms.

Function	Signature	Emitted message
log(message)	log(string)	{ type: 'log', message: <string> }
progress(percent, label)	progress(number, string)	{ type: 'progress', percent: <number>, label: <string> }

These functions abstract away whether the extension runs in a worker thread or subprocess. The main app receives identically-structured messages regardless of runtime.

JavaScript Extensions: Worker Thread IPC

Context Creation and Message Posting

JavaScript extensions execute inside a dedicated worker thread. The worker constructs the context object at lines 30-38 of electron/main/process-runner.ts:

// Inside the worker thread (extracted from process-runner.ts)
const context = {
  log: (msg: string) => {
    parentPort.postMessage({ type: 'log', message: msg });
  },
  progress: (pct: number, label: string) => {
    parentPort.postMessage({ type: 'progress', percent: pct, label });
  }
};

Calling parentPort.postMessage serializes the message and transfers it to the parent thread through Node.js's worker_threads MessageChannel.

Parent Thread Message Handling

The ProcessRunner.run method (lines 31-44 in the same file) attaches an onmessage handler to receive these updates:

// ProcessRunner.run implementation pattern
worker.on('message', (msg) => {
  if (msg.type === 'log') {
    onLog?.(msg.message);
  } else if (msg.type === 'progress') {
    onProgress?.(msg.percent, msg.label);
  }
});

The optional callbacks onLog and onProgress are supplied by the caller—typically a store or UI controller.

Python Extensions: Stdout JSON Lines

Python extensions follow a line-delimited JSON protocol written to stdout. The same message schema applies, but transport occurs through process I/O rather than IPC.

Emitting Messages from Python

A Python extension prints JSON objects to stdout, flushing after each line:

import json
import sys

def run():
    for i in range(0, 101, 25):
        # Progress update

        print(json.dumps({
            "type": "progress",
            "percent": i,
            "label": f"Stage {i//25 + 1}"
        }))
        sys.stdout.flush()
    
    # Log message

    print(json.dumps({
        "type": "log",
        "message": "Processing complete"
    }))
    sys.stdout.flush()
    
    # Completion signal with result

    print(json.dumps({
        "type": "done",
        "result": {"text": "Done"}
    }))
    sys.stdout.flush()

The type: 'done' message is Python-specific—it signals completion and carries the return value, since Python lacks the equivalent of a worker's return statement.

Parsing in PythonProcessRunner

Around lines 98-108 of electron/main/process-runner.ts, the PythonProcessRunner reads stdout line-by-line:

// Simplified from the stdout handler
pythonProcess.stdout.on('data', (data: Buffer) => {
  const lines = data.toString().split('\n').filter(Boolean);
  for (const line of lines) {
    try {
      const msg = JSON.parse(line);
      if (msg.type === 'log') onLog?.(msg.message);
      if (msg.type === 'progress') onProgress?.(msg.percent, msg.label);
      if (msg.type === 'done') resolve(msg.result);
    } catch (err) {
      // Non-JSON lines treated as raw logs
      onLog?.(line);
    }
  }
});

Critical detail: The parser falls back to treating unparsable lines as raw log output, allowing print() debugging without breaking the protocol.

Complete JavaScript Extension Example

// extensions/image-resize/processor.js
const sharp = require('sharp');

module.exports = async (input, params, { log, progress }) => {
  log(`Loading image from ${input.path}`);
  
  const steps = ['read', 'resize', 'sharpen', 'encode', 'write'];
  const pipeline = sharp(input.path)
    .resize(params.width, params.height)
    .sharpen();
  
  for (let i = 0; i < steps.length; i++) {
    progress(
      Math.round((i / steps.length) * 100),
      `Step ${i + 1}/${steps.length}: ${steps[i]}`
    );
    // Actual processing happens here...
    await pipeline.toFile(`${input.path}.resized.jpg`);
  }
  
  progress(100, 'Complete');
  log('Image processing finished successfully');
  
  return { outputPath: `${input.path}.resized.jpg` };
};

The module.exports function receives three arguments: input (the node input), params (user-configured parameters), and the context object destructured to { log, progress }.

Registering Callbacks from the Main Process

The ProcessRunner class accepts callback functions that bridge to UI state. Here's a complete main-process integration:

import { ProcessRunner } from '@/electron/main/process-runner';
import { useAppStore } from '@/shared/stores/appStore';
import { useWorkflowRunStore } from '@/areas/workflows/workflowRunStore';

async function executeExtension(extDir: string, nodeConfig: any) {
  const runner = new ProcessRunner(
    extDir,
    'processor.js',           // or 'processor.py'
    workspaceDir,
    tempDir
  );

  const appStore = useAppStore.getState();
  
  try {
    const result = await runner.run(
      nodeConfig.input,
      nodeConfig.params,
      // onProgress: updates both local node state and global app progress
      (percent, label) => {
        useWorkflowRunStore.getState().updateNodeProgress(
          nodeConfig.nodeId,
          percent,
          label
        );
        appStore.updateCurrentJob({ progress: percent, step: label });
      },
      // onLog: streams to developer console and UI log panel
      (message) => {
        console.log(`[${nodeConfig.nodeId}]`, message);
        appStore.appendToJobLog(message);
      }
    );
    
    return result;
  } finally {
    runner.cleanup();
  }
}

Key integration points:

• workflowRunStore tracks per-node progress within a multi-node workflow
• appStore maintains the global "current job" state for UI display
• Both stores update reactive UI components through Zustand subscriptions

Message Flow Architecture

┌─────────────────┐     ┌──────────────────┐     ┌─────────────────┐
│  JS Extension   │     │   Python Ext.    │     │   Main Process  │
│  (worker thread)│     │   (subprocess)   │     │  ProcessRunner  │
└────────┬────────┘     └────────┬─────────┘     └────────┬────────┘
         │                       │                        │
    log("msg")              print(json.dumps({        on('message')
    progress(50, "half")        "type": "log"...   ───►│  stdout.on('data')
         │                       │                        │
         ▼                       ▼                        ▼
   parentPort.             process.stdout           parse & route to
   postMessage()           (line-delimited)         onLog / onProgress
         │                       │                        │
         └───────────────────────┴────────────────────────┘
                              │
                              ▼
                    ┌─────────────────┐
                    │  AppStore /     │
                    │  WorkflowRunStore│
                    │  (UI state)     │
                    └─────────────────┘

This unified design lets Modly treat both runtimes identically at the UI layer while respecting their underlying constraints.

Error Handling and Edge Cases

Buffering and Partial Lines

The Python stdout handler accumulates partial lines when data events contain incomplete JSON. The implementation in process-runner.ts uses an internal buffer to handle split UTF-8 sequences and truncated JSON objects.

Malformed Messages

As noted above, JSON parse failures fall back to raw log output. This prevents a single malformed line from crashing the extension runner while preserving visibility into what went wrong.

Worker Termination

If a JavaScript extension throws or calls process.exit, the worker's 'error' and 'exit' events trigger cleanup. The ProcessRunner wrapper rejects the run promise with diagnostic information.

Summary

• Two runtimes, one protocol: JavaScript worker threads and Python subprocesses both use log() and progress() context functions with identical message schemas
• Transport differs: JS uses parentPort.postMessage(); Python uses stdout JSON lines
• Central dispatch: electron/main/process-runner.ts contains all message routing logic for both runtimes
• Callback-based UI updates: The run() method accepts onLog and onProgress callbacks that feed into AppStore and WorkflowRunStore
• Graceful degradation: Python extensions can use plain print(); the parser treats non-JSON lines as logs

Frequently Asked Questions

What happens if a Python extension forgets to flush stdout?

The PythonProcessRunner may receive incomplete or delayed messages. Always call sys.stdout.flush() after each print() to ensure timely delivery. The example code in the Modly documentation includes this pattern as a requirement.

Can extensions send custom message types beyond log and progress?

The base ProcessRunner only recognizes log, progress, and done (Python). For custom protocols, extensions can overload the input object with request-response patterns or write to files in the tempDir passed to the constructor.

How does Modly handle concurrent extension runs?

Each ProcessRunner instance creates an isolated worker or subprocess. Multiple runners can operate simultaneously, with their callbacks routing to different node IDs in WorkflowRunStore. No global locks coordinate between extensions—state management is the caller's responsibility.

Is there a size limit for log messages?

Worker threads serialize through the structured clone algorithm, which handles strings up to memory limits. Python's stdout pipe has typical shell buffer constraints (~64KB on many systems). For large payloads, extensions should write to the filesystem and emit a path reference.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}