How Modly Installs Extensions from GitHub and Handles Backup/Restore

Modly installs extensions from GitHub by cloning repositories into a temporary dot-prefixed directory, marking them with a .modly-incomplete file during setup, and renaming them upon successful completion, while using dot-prefixed folders to create lightweight filesystem backups that can be restored by renaming.

The lightningpixel/modly repository implements a robust extension management system that treats each model or processing extension as a self-contained GitHub repository. Understanding how extensions are installed from GitHub and the underlying backup/restore process is essential for developers managing custom AI model integrations.

The GitHub Extension Installation Workflow

Modly manages extensions through the EXTENSIONS_DIR environment variable, which defaults to ~/.modly/extensions. When a user initiates an installation from GitHub, the system follows a specific staging protocol to ensure atomic operations and prevent partial installations.

Staging in Dot-Prefixed Directories

The installation process begins by cloning the remote repository into a temporary directory prefixed with a dot (.). For example, an extension named my-extension clones into .my-extension within EXTENSIONS_DIR. This convention serves two purposes: it hides the staging folder from the registry scanner, and it provides a namespace for backup operations.

According to the source code in api/services/generator_registry.py, the registry explicitly ignores any directory starting with a dot during the discovery phase:

# api/services/generator_registry.py – _discover_extensions method

if ext_dir.name.startswith("."):
    continue  # Skip hidden/staging directories

The .modly-incomplete Marker

While the extension is being cloned and configured, Modly creates a marker file named .modly-incomplete inside the temporary directory. This file acts as a semaphore to prevent the registry from loading an extension that has not finished its setup sequence.

The registry checks for this marker before attempting to load any extension:

if (ext_dir / ".modly-incomplete").exists():
    print(f"[Registry] Skipping '{ext_dir.name}': install has not completed")
    continue

Only after the setup.py script executes successfully (if present) does the installer rename the directory to its final name (removing the dot prefix) and delete the .modly-incomplete marker.

Setup and Virtual Environment Creation

Extensions requiring isolation run in subprocess mode, which necessitates creating an isolated virtual environment. TheFastAPI endpoint /setup/{ext_id} in api/routers/extensions.py handles this:

# api/routers/extensions.py

@router.post("/setup/{ext_id}")
async def setup_extension(ext_id: str):
    """
    Creates the isolated venv for an extension by running its setup.py.
    Called automatically after installing an extension from GitHub.
    """
    ext_dir = EXTENSIONS_DIR / ext_id
    setup_py = ext_dir / "setup.py"
    # Spawns subprocess using Modly’s embedded Python

Extensions without a setup.py are treated as legacy extensions and load directly via generator.py.

Registry Discovery and Validation

The GeneratorRegistry scans the extensions directory during startup to build the available model catalog. Located in api/services/generator_registry.py, the _discover_extensions method enforces strict validation rules:

1. Skip dot-prefixed directories (staging/backups)
2. Skip directories containing .modly-incomplete
3. Require manifest.json and generator.py for valid extensions

Valid extensions populate the registry and become available through Modly's API. The registry distinguishes between direct mode (loading generator.py directly) and subprocess mode (managing an isolated process via api/services/extension_process.py).

The Backup and Restore Mechanism

Modly implements backup and restore through filesystem operations rather than database snapshots or API calls, leveraging the same dot-prefix convention used for staging.

Creating Backups During Updates

When updating an existing extension, Modly first copies the current extension folder to a dot-prefixed backup directory (e.g., .my-extension-backup). The new version clones into the original directory name. If the new installation fails or produces errors, the system can revert by renaming the backup folder back to the active extension name.

This approach ensures that a working version remains available on disk until the new version completes successfully.

Restoring Previous Versions

Restoration requires no specialized API endpoints. Users or automated scripts can restore a previous version by manipulating directory names:

import os
import shutil

EXT_DIR = os.path.expanduser("~/.modly/extensions")
ext_id = "my-extension"

backup_dir = os.path.join(EXT_DIR, f".{ext_id}-backup")
original_dir = os.path.join(EXT_DIR, ext_id)

# Remove faulty installation

if os.path.isdir(original_dir):
    shutil.rmtree(original_dir)

# Restore backup

os.rename(backup_dir, original_dir)

After restoration, calling the reload endpoint refreshes the registry without restarting the application:

import requests
requests.post("http://localhost:8000/extensions/reload")

Implementation Example: Complete Installation Flow

The client-side installation logic, typically handled in tools like tools/modly-cli/agent.py, orchestrates the clone, setup, and activation sequence:

import os
import subprocess
import requests

EXT_DIR = os.getenv("EXTENSIONS_DIR", os.path.expanduser("~/.modly/extensions"))
ext_id = "my-cool-extension"
git_url = "https://github.com/user/my-cool-extension"

# Step 1: Clone into temporary dot-folder

tmp_dir = os.path.join(EXT_DIR, f".{ext_id}")
subprocess.run(["git", "clone", git_url, tmp_dir], check=True)

# Step 2: Mark as incomplete

incomplete_marker = os.path.join(tmp_dir, ".modly-incomplete")
open(incomplete_marker, "w").close()

# Step 3: Run setup via API

requests.post(f"http://localhost:8000/extensions/setup/{ext_id}")

# Step 4: Activate by renaming and removing marker

final_dir = os.path.join(EXT_DIR, ext_id)
os.rename(tmp_dir, final_dir)
os.remove(os.path.join(final_dir, ".modly-incomplete"))

Summary

• Staging: Extensions clone into dot-prefixed directories (e.g., .my-extension) to remain hidden from the registry until ready.
• Safety: The .modly-incomplete marker prevents partial installations from loading, with checks implemented in api/services/generator_registry.py.
• Setup: The /setup/{ext_id} endpoint in api/routers/extensions.py creates isolated virtual environments for subprocess-mode extensions.
• Backup: Dot-prefixed folders serve as atomic backups during updates, allowing instant rollback by renaming directories.
• Restore: Recovery is filesystem-based—simply rename the backup folder and call /extensions/reload to refresh the registry.

Frequently Asked Questions

What happens if an extension installation is interrupted?

If the installation process terminates before completion, the .modly-incomplete marker file remains in the dot-prefixed directory. During the next registry scan in api/services/generator_registry.py, the system detects this marker and skips the directory, preventing Modly from attempting to load a corrupted or incomplete extension. The user can safely delete the temporary folder and retry the installation.

Can I manually restore an extension without using the API?

Yes. The backup mechanism relies entirely on filesystem operations. If you have a dot-prefixed backup folder (such as .my-extension-backup), you can restore it by deleting the current extension folder and renaming the backup to remove the dot prefix. After renaming, send a POST request to /extensions/reload or restart Modly to refresh the registry.

Where does Modly store extensions by default?

By default, Modly stores extensions in ~/.modly/extensions as defined by the EXTENSIONS_DIR environment variable. Within this directory, active extensions appear as normal folders (e.g., my-extension), while staging directories and backups use dot prefixes (e.g., .my-extension or .my-extension-backup) to remain invisible to the registry scanner.

Does every extension need a setup.py file?

No. Extensions only require a setup.py if they intend to run in subprocess mode with an isolated virtual environment. Extensions without setup.py are treated as legacy extensions and load directly via their generator.py file in the main process. However, all valid extensions must contain both a manifest.json and a generator.py to be recognized by the GeneratorRegistry.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}