# How Modly's Extension System Architecture Works: Model vs Process Extensions Explained

> Understand Modly's extension system architecture. Learn how Model Extensions for AI generation and Process Extensions for custom scripts work differently via HTTP and local IPC.

- Repository: [lightningpixel/modly](https://github.com/lightningpixel/modly)
- Tags: architecture
- Published: 2026-08-15

---

**Modly's extension system uses a unified `AnyExtension` union type that distinguishes between `ModelExtension` (AI generation) and `ProcessExtension` (custom executable scripts) through a `type` discriminator field, with each executing through fundamentally different paths—model extensions via HTTP backend calls and process extensions via local Electron IPC.**

Modly is an open-source generative AI platform that provides a flexible plugin architecture through its extension system. Understanding how **model extensions** differ from **process extensions** is essential for developers building custom capabilities. Both types share common metadata fields but diverge sharply in execution semantics, security boundaries, and runtime behavior.

## The Core Type Architecture

Modly defines its extension types in **[`src/shared/types/electron.d.ts`](https://github.com/lightningpixel/modly/blob/main/src/shared/types/electron.d.ts)**, establishing a discriminated union pattern that TypeScript consumers use throughout the application.

### ModelExtension Structure

A **model extension** wraps a generative AI model—such as Stable Diffusion for images or LLMs for text—that receives inputs and produces new generated assets.

Key fields include:

- **`type: 'model'`** — the discriminator constant
- **`nodes`** — UI node definitions the model exposes for workflow composition
- **`trusted`**, **`builtin`**, **`source`**, **`localPath`** — metadata and provenance fields
- Optional **`manifestError`** flags for validation failures

```typescript
// src/shared/types/electron.d.ts#L29-L45
interface ModelExtension {
  type: 'model';
  id: string;
  name: string;
  nodes: ExtensionNode[];
  trusted: boolean;
  builtin: boolean;
  source: ExtensionSource;
  localPath: string;
  manifestError?: string;
  paramsSchema?: JSONSchema;
}

```

### ProcessExtension Structure

A **process extension** wraps an arbitrary executable—typically a Python script or compiled binary—that transforms existing assets rather than generating new ones from scratch.

Key fields include:

- **`type: 'process'`** — the discriminator constant
- **`entry`** — filesystem path to the executable script
- Same metadata fields as model extensions for consistency

```typescript
// src/shared/types/electron.d.ts#L63-L80
interface ProcessExtension {
  type: 'process';
  id: string;
  name: string;
  entry: string;
  trusted: boolean;
  builtin: boolean;
  source: ExtensionSource;
  localPath: string;
  manifestError?: string;
  paramsSchema?: JSONSchema;
}

```

Both types unite under **`AnyExtension = ModelExtension | ProcessExtension`**, enabling polymorphic handling in stores and UI components while preserving type-specific behavior at execution time.

## Extension Discovery and State Management

The **[`extensionsStore.ts`](https://github.com/lightningpixel/modly/blob/main/extensionsStore.ts)** module maintains separate reactive arrays for each extension type after loading from the Electron preload API.

### Loading and Filtering

When `extensionsStore.loadExtensions()` invokes **`window.electron.extensions.list()`**, it immediately partitions the unified list by type:

```tsx
// src/shared/stores/extensionsStore.ts#L49-L53
const list = await window.electron.extensions.list();
set({
  modelExtensions:   list.filter((e) => e.type === 'model'),
  processExtensions: list.filter((e) => e.type === 'process'),
});

```

This separation persists throughout the store's lifecycle, allowing UI components to render appropriate pickers and configuration panels for each category.

### Installation Routing

The generic `installExtension` helper directs newly installed extensions to the correct array based on runtime type inspection:

```tsx
// src/shared/stores/extensionsStore.ts#L28-L35
const installExtension = (ext: AnyExtension) => {
  set((state) => {
    if (ext.type === 'model') {
      return { modelExtensions: [...state.modelExtensions, ext] };
    } else {
      return { processExtensions: [...state.processExtensions, ext] };
    }
  });
};

```

## Execution Pathways: Where the Architectures Diverge

The **[`workflowRunStore.ts`](https://github.com/lightningpixel/modly/blob/main/workflowRunStore.ts)** module contains the critical branching logic in its `executeExtensionNode` function. Despite both extension types appearing as nodes in the same workflow graph, their execution implementations share no code path.

### Model Extension Execution: Backend Generation Jobs

When a workflow node references a model extension (`ext.type === 'model'`), the client constructs a multipart HTTP POST to the Modly backend's generation API:

```tsx
// src/areas/workflows/workflowRunStore.ts#L40-L88
if (ext?.type === 'model') {
  const fd = new FormData();
  fd.append('image', blob, fname);
  fd.append('model_id', node.data.extensionId ?? '');
  fd.append('params', JSON.stringify(effectiveParams));
  
  const { data } = await client.post('/generate/from-image', fd);
  const jobId = data.job_id;
  
  // Poll until completion
  const status = await pollJobStatus(jobId);
  updateNodeOutput(nodeId, status.output_url);
}

```

This path characteristics:

- **Stateful**: returns a `jobId` for asynchronous polling
- **Server-side execution**: the model runs on GPU-equipped backend infrastructure
- **Network-transported assets**: images and parameters serialize over HTTP
- **Latency-tolerant**: assumes multi-second generation times

### Process Extension Execution: Local IPC Invocation

When a node references a process extension (`ext.type === 'process'`), execution remains entirely client-side through Electron's IPC bridge:

```tsx
// src/areas/workflows/workflowRunStore.ts#L26-L33
if (ext?.type === 'process') {
  const result = await window.electron.extensions.runProcess(
    extId,
    { filePath: nodeInputPath, text: nodeInputText, texts: nodeInputTexts, nodeId: nid },
    liveParams,
  );
  
  if (!result.success) throw new Error(result.error);
  updateNodeOutput(nodeId, result.result.filePath ?? result.result.text);
}

```

This path characteristics:

- **Synchronous from client's perspective**: returns immediate result or error
- **Local execution**: spawns Python process via **`src/electron/main/ipc-handlers.ts#L1398-L1408`**
- **Filesystem-local**: operates on paths accessible to the Electron main process
- **Lower latency**: suitable for preprocessing, postprocessing, and utility transforms

## Security and Trust Implications

Both extension types respect Modly's **trust system**, but the attack surface differs materially.

| Concern | Model Extension | Process Extension |
|--------|-----------------|-------------------|
| Code execution venue | Remote backend (containerized) | Local user machine |
| Filesystem access | None (API-mediated) | Full access to Electron's permissions |
| Network capabilities | Isolated backend network | Inherits Electron main process reach |
| Supply chain risk | Model weights tampering | Script injection, dependency confusion |

The **`trusted`** and **`builtin`** boolean flags in both extension types gate automatic execution. Untrusted extensions require explicit user confirmation before workflow nodes activate.

## UI Integration and Parameter Schemas

Despite execution differences, both extension types expose **`paramsSchema`** (JSON Schema) for workflow node configuration. The UI renders parameter editors generically, though model extensions typically expose generation-oriented controls (seed, steps, CFG scale) while process extensions expose transformation parameters (thresholds, format options, boolean flags).

Neither extension type appears visually distinct to end users in node pickers—discrimination happens at the data layer through the `type` field, maintaining conceptual simplicity while enabling powerful specialization.

## Summary

- Modly's **`AnyExtension`** union type unifies **model** and **process** extensions under common metadata while preserving execution-specific fields
- **Model extensions** (type: `'model'`) execute remotely via HTTP `/generate/*` endpoints, return job IDs, and run AI inference on backend infrastructure
- **Process extensions** (type: `'process'`) execute locally via **`window.electron.extensions.runProcess`**, return immediate results, and spawn Python scripts through Electron IPC
- The **`extensionsStore`** maintains separate filtered arrays after loading from **`window.electron.extensions.list()`**, routing installations appropriately
- **Workflow execution** branches at `executeExtensionNode`—no shared runtime path between the two extension categories
- Both types respect trust flags and expose JSON Schema parameters, but process extensions carry elevated local security privileges

## Frequently Asked Questions

### Can a single extension be both a model and a process extension?

No. The `type` discriminator field in [`electron.d.ts`](https://github.com/lightningpixel/modly/blob/main/electron.d.ts) (`'model' | 'process'`) forces mutual exclusivity. An extension package must declare one execution mode. If you need both generation and transformation capabilities, ship two extensions or implement the transformation as a post-processing step within a model extension's node graph.

### How does Modly prevent malicious process extensions from harming my system?

Process extensions inherit Electron main process privileges, creating inherent risk. Modly mitigates this through: (1) mandatory **`trusted`** flag verification—untrusted extensions require explicit user activation; (2) **`builtin`** extensions from verified publishers execute by default; (3) source attribution tracking via the `source` field for auditability. However, process extensions fundamentally require more trust than model extensions due to local execution.

### Why don't model extensions run locally like process extensions?

Generative AI models typically require GPU acceleration and substantial memory unavailable in standard Electron contexts. By routing model execution to a backend service, Modly: enables hardware flexibility (cloud GPUs, local server variants), centralizes model weight management, and isolates computationally intensive inference from UI responsiveness. Process extensions handle lighter-weight CPU-bound transformations suitable for client hardware.

### Can I convert between extension types after installation?

No conversion mechanism exists. The `type` field is immutable after extension loading. To change execution modes, modify the extension's manifest and reinstall. The `extensionsStore` re-filters on every `loadExtensions()` call, so type changes reflect immediately upon reload if you manually edit extension metadata—though this breaks the extension contract and will likely cause runtime failures.