How Modly's Extension System Architecture Works: Model vs Process Extensions Explained

Modly's extension system uses a unified AnyExtension union type that distinguishes between ModelExtension (AI generation) and ProcessExtension (custom executable scripts) through a type discriminator field, with each executing through fundamentally different paths—model extensions via HTTP backend calls and process extensions via local Electron IPC.

Modly is an open-source generative AI platform that provides a flexible plugin architecture through its extension system. Understanding how model extensions differ from process extensions is essential for developers building custom capabilities. Both types share common metadata fields but diverge sharply in execution semantics, security boundaries, and runtime behavior.

The Core Type Architecture

Modly defines its extension types in src/shared/types/electron.d.ts, establishing a discriminated union pattern that TypeScript consumers use throughout the application.

ModelExtension Structure

A model extension wraps a generative AI model—such as Stable Diffusion for images or LLMs for text—that receives inputs and produces new generated assets.

Key fields include:

• type: 'model' — the discriminator constant
• nodes — UI node definitions the model exposes for workflow composition
• trusted, builtin, source, localPath — metadata and provenance fields
• Optional manifestError flags for validation failures

// src/shared/types/electron.d.ts#L29-L45
interface ModelExtension {
  type: 'model';
  id: string;
  name: string;
  nodes: ExtensionNode[];
  trusted: boolean;
  builtin: boolean;
  source: ExtensionSource;
  localPath: string;
  manifestError?: string;
  paramsSchema?: JSONSchema;
}

ProcessExtension Structure

A process extension wraps an arbitrary executable—typically a Python script or compiled binary—that transforms existing assets rather than generating new ones from scratch.

Key fields include:

• type: 'process' — the discriminator constant
• entry — filesystem path to the executable script
• Same metadata fields as model extensions for consistency

// src/shared/types/electron.d.ts#L63-L80
interface ProcessExtension {
  type: 'process';
  id: string;
  name: string;
  entry: string;
  trusted: boolean;
  builtin: boolean;
  source: ExtensionSource;
  localPath: string;
  manifestError?: string;
  paramsSchema?: JSONSchema;
}

Both types unite under AnyExtension = ModelExtension | ProcessExtension, enabling polymorphic handling in stores and UI components while preserving type-specific behavior at execution time.

Extension Discovery and State Management

The extensionsStore.ts module maintains separate reactive arrays for each extension type after loading from the Electron preload API.

Loading and Filtering

When extensionsStore.loadExtensions() invokes window.electron.extensions.list(), it immediately partitions the unified list by type:

// src/shared/stores/extensionsStore.ts#L49-L53
const list = await window.electron.extensions.list();
set({
  modelExtensions:   list.filter((e) => e.type === 'model'),
  processExtensions: list.filter((e) => e.type === 'process'),
});

This separation persists throughout the store's lifecycle, allowing UI components to render appropriate pickers and configuration panels for each category.

Installation Routing

The generic installExtension helper directs newly installed extensions to the correct array based on runtime type inspection:

// src/shared/stores/extensionsStore.ts#L28-L35
const installExtension = (ext: AnyExtension) => {
  set((state) => {
    if (ext.type === 'model') {
      return { modelExtensions: [...state.modelExtensions, ext] };
    } else {
      return { processExtensions: [...state.processExtensions, ext] };
    }
  });
};

Execution Pathways: Where the Architectures Diverge

The workflowRunStore.ts module contains the critical branching logic in its executeExtensionNode function. Despite both extension types appearing as nodes in the same workflow graph, their execution implementations share no code path.

Model Extension Execution: Backend Generation Jobs

When a workflow node references a model extension (ext.type === 'model'), the client constructs a multipart HTTP POST to the Modly backend's generation API:

// src/areas/workflows/workflowRunStore.ts#L40-L88
if (ext?.type === 'model') {
  const fd = new FormData();
  fd.append('image', blob, fname);
  fd.append('model_id', node.data.extensionId ?? '');
  fd.append('params', JSON.stringify(effectiveParams));
  
  const { data } = await client.post('/generate/from-image', fd);
  const jobId = data.job_id;
  
  // Poll until completion
  const status = await pollJobStatus(jobId);
  updateNodeOutput(nodeId, status.output_url);
}

This path characteristics:

• Stateful: returns a jobId for asynchronous polling
• Server-side execution: the model runs on GPU-equipped backend infrastructure
• Network-transported assets: images and parameters serialize over HTTP
• Latency-tolerant: assumes multi-second generation times

Process Extension Execution: Local IPC Invocation

When a node references a process extension (ext.type === 'process'), execution remains entirely client-side through Electron's IPC bridge:

// src/areas/workflows/workflowRunStore.ts#L26-L33
if (ext?.type === 'process') {
  const result = await window.electron.extensions.runProcess(
    extId,
    { filePath: nodeInputPath, text: nodeInputText, texts: nodeInputTexts, nodeId: nid },
    liveParams,
  );
  
  if (!result.success) throw new Error(result.error);
  updateNodeOutput(nodeId, result.result.filePath ?? result.result.text);
}

This path characteristics:

• Synchronous from client's perspective: returns immediate result or error
• Local execution: spawns Python process via src/electron/main/ipc-handlers.ts#L1398-L1408
• Filesystem-local: operates on paths accessible to the Electron main process
• Lower latency: suitable for preprocessing, postprocessing, and utility transforms

Security and Trust Implications

Both extension types respect Modly's trust system, but the attack surface differs materially.

Concern	Model Extension	Process Extension
Code execution venue	Remote backend (containerized)	Local user machine
Filesystem access	None (API-mediated)	Full access to Electron's permissions
Network capabilities	Isolated backend network	Inherits Electron main process reach
Supply chain risk	Model weights tampering	Script injection, dependency confusion

The trusted and builtin boolean flags in both extension types gate automatic execution. Untrusted extensions require explicit user confirmation before workflow nodes activate.

UI Integration and Parameter Schemas

Despite execution differences, both extension types expose paramsSchema (JSON Schema) for workflow node configuration. The UI renders parameter editors generically, though model extensions typically expose generation-oriented controls (seed, steps, CFG scale) while process extensions expose transformation parameters (thresholds, format options, boolean flags).

Neither extension type appears visually distinct to end users in node pickers—discrimination happens at the data layer through the type field, maintaining conceptual simplicity while enabling powerful specialization.

Summary

• Modly's AnyExtension union type unifies model and process extensions under common metadata while preserving execution-specific fields
• Model extensions (type: 'model') execute remotely via HTTP /generate/* endpoints, return job IDs, and run AI inference on backend infrastructure
• Process extensions (type: 'process') execute locally via window.electron.extensions.runProcess, return immediate results, and spawn Python scripts through Electron IPC
• The extensionsStore maintains separate filtered arrays after loading from window.electron.extensions.list(), routing installations appropriately
• Workflow execution branches at executeExtensionNode—no shared runtime path between the two extension categories
• Both types respect trust flags and expose JSON Schema parameters, but process extensions carry elevated local security privileges

Frequently Asked Questions

Can a single extension be both a model and a process extension?

No. The type discriminator field in electron.d.ts ('model' | 'process') forces mutual exclusivity. An extension package must declare one execution mode. If you need both generation and transformation capabilities, ship two extensions or implement the transformation as a post-processing step within a model extension's node graph.

How does Modly prevent malicious process extensions from harming my system?

Process extensions inherit Electron main process privileges, creating inherent risk. Modly mitigates this through: (1) mandatory trusted flag verification—untrusted extensions require explicit user activation; (2) builtin extensions from verified publishers execute by default; (3) source attribution tracking via the source field for auditability. However, process extensions fundamentally require more trust than model extensions due to local execution.

Why don't model extensions run locally like process extensions?

Generative AI models typically require GPU acceleration and substantial memory unavailable in standard Electron contexts. By routing model execution to a backend service, Modly: enables hardware flexibility (cloud GPUs, local server variants), centralizes model weight management, and isolates computationally intensive inference from UI responsiveness. Process extensions handle lighter-weight CPU-bound transformations suitable for client hardware.

Can I convert between extension types after installation?

No conversion mechanism exists. The type field is immutable after extension loading. To change execution modes, modify the extension's manifest and reinstall. The extensionsStore re-filters on every loadExtensions() call, so type changes reflect immediately upon reload if you manually edit extension metadata—though this breaks the extension contract and will likely cause runtime failures.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}