How Modly's Extension System Supports External AI Models: A Complete Technical Guide

Modly treats every external AI model as a model extension, using a discovery-and-loading pipeline that scans extension directories, validates manifests, and exposes models as workflow nodes.

Modly is an open-source AI workflow platform that makes external model integration seamless through its extension system. Whether you're adding HuggingFace transformers, custom PyTorch models, or proprietary inference engines, Modly's architecture treats them uniformly as model extensions. This article examines the complete lifecycle—from discovery to execution—based on the source code in lightningpixel/modly.

Discovery and Loading of Model Extensions

When the Modly UI initializes, the extension discovery process begins through the useExtensionsStore module.

The store calls the Electron IPC method extensions:list, which triggers a filesystem scan of the extensions directory. The handler in [electron/main/ipc-handlers.ts](https://github.com/lightningpixel/modly/blob/main/electron/main/ipc-handlers.ts) reads each subdirectory, parses the modly.json manifest, and returns a normalized list of extension objects.

In [src/shared/stores/extensionsStore.ts](https://github.com/lightningpixel/modly/blob/main/src/shared/stores/extensionsStore.ts) (lines 46-53), the received list splits into two categories:

// From extensionsStore.ts
const modelExtensions = extensions.filter(e => e.type === 'model')
const processExtensions = extensions.filter(e => e.type === 'process')

This categorization determines how each extension appears in the UI and what execution path it follows.

Installing External AI Models

Users can install model extensions from two sources: GitHub repositories or local folders. The store exposes corresponding methods that delegate to IPC handlers.

Installation Methods

Method	Source	IPC Handler
installFromGitHub(repoUrl)	Remote Git repository	extensions:install-from-github
installFromLocal(folderPath)	Local filesystem	extensions:install-from-local

Both methods:

1. Copy extension files into the user extensions directory
2. Write a .modly-local sentinel file to mark non-git-managed extensions
3. Trigger the reload sequence

After installation, reload() is invoked (lines 60-75 in extensionsStore.ts) to make the new model available without restarting the application.

Validation and Runtime Registration

The extensions:reload IPC handler performs a critical synchronization step. Per the source in [electron/main/ipc-handlers.ts](https://github.com/lightningpixel/modly/blob/main/electron/main/ipc-handlers.ts) (around lines 945-959):

1. The Python backend re-scans its model registry
2. New model weights are verified and cached
3. The UI refreshes its extension list via extensions:list

This ensures that model metadata, capability declarations, and entry points are validated before the model appears in the visual editor.

Executing Models in Workflows

When a workflow contains a model node, [src/areas/workflows/workflowRunStore.ts](https://github.com/lightningpixel/modly/blob/main/src/areas/workflows/workflowRunStore.ts) (lines 338-388) handles execution:

// Runtime check for model extension nodes
if (extension?.type === 'model') {
  const formData = new FormData()
  formData.append('model_id', node.extensionId)
  // Append input assets (images, text, etc.)
  
  const result = await fetch('/api/model/run', {
    method: 'POST',
    body: formData
  })
  return result.json()
}

The backend receives the model_id, locates the corresponding extension files, and invokes the Python entry point (specified in entry field of manifest) with the provided inputs.

Model-Specific IPC Operations

Modly exposes a thin wrapper at window.electron.model defined in [electron/preload/electron-api.ts](https://github.com/lightningpixel/modly/blob/main/electron/preload/electron-api.ts) (lines 111-123). This API surface includes:

• download(modelId) — Fetch model weights from remote
• delete(modelId) — Remove local model files
• showInFolder(modelId) — Open file manager to model location
• getStatus(modelId) — Check download/ready state

These wrappers forward to Electron main process, which coordinates with the Python backend hosting actual model binaries.

Extension Manifest Schema

For an external AI model to integrate, it must provide a modly.json manifest:

{
  "id": "stable-diffusion-xl",
  "type": "model",
  "name": "Stable Diffusion XL",
  "description": "High-resolution text-to-image generation",
  "capabilities": ["text2img", "img2img"],
  "entry": "inference.py",
  "requirements": ["torch>=2.0", "diffusers", "accelerate"]
}

Required fields:

• id — Unique identifier (kebab-case recommended)
• type — Must be "model" for AI model extensions
• capabilities — Array of operation modes the model supports
• entry — Python script executed for inference

Optional fields include icon, tags, author, and license.

Practical Code Examples

Installing a Model from GitHub

import { useExtensionsStore } from '@shared/stores/extensionsStore'

async function installHuggingFaceModel(repoUrl: string) {
  const store = useExtensionsStore.getState()
  
  const { success, error } = await store.installFromGitHub(repoUrl)
  
  if (success) {
    await store.reload()
    console.log('Model registered and ready for workflows')
  } else {
    throw new Error(`Installation failed: ${error}`)
  }
}

Querying Available Models by Capability

import { useExtensionsStore } from '@shared/stores/extensionsStore'

function findTextToImageModels() {
  const { modelExtensions } = useExtensionsStore.getState()
  
  return modelExtensions.filter(m => 
    m.capabilities.includes('text2img')
  ).map(m => ({
    id: m.id,
    name: m.name,
    ready: m.installed && m.downloaded
  }))
}

Programmatic Model Execution

async function generateImage(
  modelId: string, 
  prompt: string, 
  width: number, 
  height: number
) {
  const form = new FormData()
  form.append('model_id', modelId)
  form.append('prompt', prompt)
  form.append('width', String(width))
  form.append('height', String(height))

  const response = await fetch('/api/model/run', {
    method: 'POST',
    body: form
  })
  
  if (!response.ok) {
    const error = await response.text()
    throw new Error(`Inference failed: ${error}`)
  }
  
  return response.blob() // Generated image
}

Summary

Modly's extension system enables external AI model support through five core mechanisms:

• Unified discovery — useExtensionsStore loads and categorizes extensions by type field
• Flexible installation — GitHub and local sources supported with automatic directory management
• Hot reloading — extensions:reload validates and registers models without restart
• Workflow integration — Model nodes execute via multipart POST to /api/model/run
• IPC abstraction — window.electron.model provides lifecycle operations for model management

The manifest-driven architecture means any conforming Python-based model can plug into Modly's visual workflow system with minimal boilerplate.

Frequently Asked Questions

What Python frameworks are compatible with Modly model extensions?

Any framework that exposes a Python callable can be used. The entry script receives parsed arguments and input files via standard interfaces. Common choices include PyTorch, TensorFlow, ONNX Runtime, and HuggingFace Diffusers/Transformers. The requirements array in the manifest specifies dependencies for automatic installation.

Can multiple versions of the same model exist simultaneously?

Yes, but each requires a unique id in its manifest. Modly uses the id as the primary key for extension registration. If you need versioned models, include version strings in the identifier (e.g., "sdxl-1-0" vs "sdxl-refiner-1-0").

How does Modly handle model file storage and caching?

Model weights are stored in a subdirectory named after the extension id within the user extensions folder. The window.electron.model.download() IPC method fetches weights from configured URLs, and the status is tracked per-extension. The Python backend maintains its own registry cache that maps model_id to filesystem paths for inference.

Is it possible to extend Modly with non-Python models?

Currently, the inference execution requires a Python entry point. However, the entry script can shell out to other runtimes or use inter-process communication. For pure non-Python models, a wrapper Python script that invokes the external runtime is the recommended pattern until native multi-runtime support is implemented.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}