# How Modly's Extension System Supports External AI Models: A Complete Technical Guide

> Discover how Modly's extension system supports external AI models by treating them as extensions. Learn about its discovery-and-loading pipeline for seamless integration.

- Repository: [lightningpixel/modly](https://github.com/lightningpixel/modly)
- Tags: technical-guide
- Published: 2026-08-20

---

**Modly treats every external AI model as a model extension, using a discovery-and-loading pipeline that scans extension directories, validates manifests, and exposes models as workflow nodes.**

Modly is an open-source AI workflow platform that makes external model integration seamless through its extension system. Whether you're adding HuggingFace transformers, custom PyTorch models, or proprietary inference engines, Modly's architecture treats them uniformly as **model extensions**. This article examines the complete lifecycle—from discovery to execution—based on the source code in `lightningpixel/modly`.

## Discovery and Loading of Model Extensions

When the Modly UI initializes, the extension discovery process begins through the `useExtensionsStore` module.

The store calls the Electron IPC method `extensions:list`, which triggers a filesystem scan of the extensions directory. The handler in [[`electron/main/ipc-handlers.ts`](https://github.com/lightningpixel/modly/blob/main/electron/main/ipc-handlers.ts)](https://github.com/lightningpixel/modly/blob/main/electron/main/ipc-handlers.ts) reads each subdirectory, parses the [`modly.json`](https://github.com/lightningpixel/modly/blob/main/modly.json) manifest, and returns a normalized list of extension objects.

In [[`src/shared/stores/extensionsStore.ts`](https://github.com/lightningpixel/modly/blob/main/src/shared/stores/extensionsStore.ts)](https://github.com/lightningpixel/modly/blob/main/src/shared/stores/extensionsStore.ts) (lines 46-53), the received list splits into two categories:

```typescript
// From extensionsStore.ts
const modelExtensions = extensions.filter(e => e.type === 'model')
const processExtensions = extensions.filter(e => e.type === 'process')

```

This categorization determines how each extension appears in the UI and what execution path it follows.

## Installing External AI Models

Users can install model extensions from two sources: GitHub repositories or local folders. The store exposes corresponding methods that delegate to IPC handlers.

### Installation Methods

| Method | Source | IPC Handler |
|--------|--------|-------------|
| `installFromGitHub(repoUrl)` | Remote Git repository | `extensions:install-from-github` |
| `installFromLocal(folderPath)` | Local filesystem | `extensions:install-from-local` |

Both methods:

1. Copy extension files into the user extensions directory
2. Write a `.modly-local` sentinel file to mark non-git-managed extensions
3. Trigger the reload sequence

After installation, `reload()` is invoked (lines 60-75 in [`extensionsStore.ts`](https://github.com/lightningpixel/modly/blob/main/extensionsStore.ts)) to make the new model available without restarting the application.

## Validation and Runtime Registration

The `extensions:reload` IPC handler performs a critical synchronization step. Per the source in [[`electron/main/ipc-handlers.ts`](https://github.com/lightningpixel/modly/blob/main/electron/main/ipc-handlers.ts)](https://github.com/lightningpixel/modly/blob/main/electron/main/ipc-handlers.ts) (around lines 945-959):

1. The Python backend re-scans its model registry
2. New model weights are verified and cached
3. The UI refreshes its extension list via `extensions:list`

This ensures that model metadata, capability declarations, and entry points are validated before the model appears in the visual editor.

## Executing Models in Workflows

When a workflow contains a model node, [[`src/areas/workflows/workflowRunStore.ts`](https://github.com/lightningpixel/modly/blob/main/src/areas/workflows/workflowRunStore.ts)](https://github.com/lightningpixel/modly/blob/main/src/areas/workflows/workflowRunStore.ts) (lines 338-388) handles execution:

```typescript
// Runtime check for model extension nodes
if (extension?.type === 'model') {
  const formData = new FormData()
  formData.append('model_id', node.extensionId)
  // Append input assets (images, text, etc.)
  
  const result = await fetch('/api/model/run', {
    method: 'POST',
    body: formData
  })
  return result.json()
}

```

The backend receives the `model_id`, locates the corresponding extension files, and invokes the Python entry point (specified in `entry` field of manifest) with the provided inputs.

## Model-Specific IPC Operations

Modly exposes a thin wrapper at `window.electron.model` defined in [[`electron/preload/electron-api.ts`](https://github.com/lightningpixel/modly/blob/main/electron/preload/electron-api.ts)](https://github.com/lightningpixel/modly/blob/main/electron/preload/electron-api.ts) (lines 111-123). This API surface includes:

- `download(modelId)` — Fetch model weights from remote
- `delete(modelId)` — Remove local model files
- `showInFolder(modelId)` — Open file manager to model location
- `getStatus(modelId)` — Check download/ready state

These wrappers forward to Electron main process, which coordinates with the Python backend hosting actual model binaries.

## Extension Manifest Schema

For an external AI model to integrate, it must provide a [`modly.json`](https://github.com/lightningpixel/modly/blob/main/modly.json) manifest:

```json
{
  "id": "stable-diffusion-xl",
  "type": "model",
  "name": "Stable Diffusion XL",
  "description": "High-resolution text-to-image generation",
  "capabilities": ["text2img", "img2img"],
  "entry": "inference.py",
  "requirements": ["torch>=2.0", "diffusers", "accelerate"]
}

```

Required fields:

- `id` — Unique identifier (kebab-case recommended)
- `type` — Must be `"model"` for AI model extensions
- `capabilities` — Array of operation modes the model supports
- `entry` — Python script executed for inference

Optional fields include `icon`, `tags`, `author`, and `license`.

## Practical Code Examples

### Installing a Model from GitHub

```typescript
import { useExtensionsStore } from '@shared/stores/extensionsStore'

async function installHuggingFaceModel(repoUrl: string) {
  const store = useExtensionsStore.getState()
  
  const { success, error } = await store.installFromGitHub(repoUrl)
  
  if (success) {
    await store.reload()
    console.log('Model registered and ready for workflows')
  } else {
    throw new Error(`Installation failed: ${error}`)
  }
}

```

### Querying Available Models by Capability

```typescript
import { useExtensionsStore } from '@shared/stores/extensionsStore'

function findTextToImageModels() {
  const { modelExtensions } = useExtensionsStore.getState()
  
  return modelExtensions.filter(m => 
    m.capabilities.includes('text2img')
  ).map(m => ({
    id: m.id,
    name: m.name,
    ready: m.installed && m.downloaded
  }))
}

```

### Programmatic Model Execution

```typescript
async function generateImage(
  modelId: string, 
  prompt: string, 
  width: number, 
  height: number
) {
  const form = new FormData()
  form.append('model_id', modelId)
  form.append('prompt', prompt)
  form.append('width', String(width))
  form.append('height', String(height))

  const response = await fetch('/api/model/run', {
    method: 'POST',
    body: form
  })
  
  if (!response.ok) {
    const error = await response.text()
    throw new Error(`Inference failed: ${error}`)
  }
  
  return response.blob() // Generated image
}

```

## Summary

Modly's extension system enables external AI model support through five core mechanisms:

- **Unified discovery** — `useExtensionsStore` loads and categorizes extensions by `type` field
- **Flexible installation** — GitHub and local sources supported with automatic directory management
- **Hot reloading** — `extensions:reload` validates and registers models without restart
- **Workflow integration** — Model nodes execute via multipart POST to `/api/model/run`
- **IPC abstraction** — `window.electron.model` provides lifecycle operations for model management

The manifest-driven architecture means any conforming Python-based model can plug into Modly's visual workflow system with minimal boilerplate.

## Frequently Asked Questions

### What Python frameworks are compatible with Modly model extensions?

Any framework that exposes a Python callable can be used. The `entry` script receives parsed arguments and input files via standard interfaces. Common choices include PyTorch, TensorFlow, ONNX Runtime, and HuggingFace Diffusers/Transformers. The `requirements` array in the manifest specifies dependencies for automatic installation.

### Can multiple versions of the same model exist simultaneously?

Yes, but each requires a unique `id` in its manifest. Modly uses the `id` as the primary key for extension registration. If you need versioned models, include version strings in the identifier (e.g., `"sdxl-1-0"` vs `"sdxl-refiner-1-0"`).

### How does Modly handle model file storage and caching?

Model weights are stored in a subdirectory named after the extension `id` within the user extensions folder. The `window.electron.model.download()` IPC method fetches weights from configured URLs, and the status is tracked per-extension. The Python backend maintains its own registry cache that maps `model_id` to filesystem paths for inference.

### Is it possible to extend Modly with non-Python models?

Currently, the inference execution requires a Python entry point. However, the entry script can shell out to other runtimes or use inter-process communication. For pure non-Python models, a wrapper Python script that invokes the external runtime is the recommended pattern until native multi-runtime support is implemented.