How Modly's API Layer Is Structured and How It Interfaces With the FastAPI Backend

Modly's API layer uses a FastAPI application with modular routers, a central GeneratorRegistry for model discovery, and abstract BaseGenerator subclasses that handle inference either in-process or via isolated subprocesses.

Modly is an open-source AI inference framework for 3D generation. Its API layer architecture cleanly separates HTTP concerns from model-specific logic, enabling extensibility through a plugin-based extension system. This article examines the core components and their interactions based on the actual source code in the lightningpixel/modly repository.

Core Components of the API Layer

api/main.py: FastAPI Application Entry Point

The root of the API layer resides in api/main.py. This file instantiates the FastAPI app, configures CORS middleware, registers all routers, and defines an asynchronous lifespan context that manages the full application lifecycle.

# api/main.py

app = FastAPI(
    title="Modly API",
    version="0.4.1",
    lifespan=lifespan,
)

The lifespan context manager is critical: it calls GeneratorRegistry.initialize() on startup to discover and load extensions, then performs graceful shutdown when the application terminates.

Routers are included with optional URL prefixes:

app.include_router(generation.router, prefix="/generate")
app.include_router(status.router)

# … additional routers …

The same file also implements serve_workspace_file, which streams generated assets from WORKSPACE_DIR back to clients.

Modular Routers in api/routers/

Endpoints are organized by domain into separate APIRouter modules:

Router File	Purpose
api/routers/generation.py	Image-to-mesh generation, job tracking, cancellation, progress callbacks
api/routers/status.py	Health checks for the Electron frontend
api/routers/models.py	Model management operations
api/routers/settings.py	Configuration endpoints
api/routers/extensions.py	Extension discovery and metadata
api/routers/export.py	Export format handling
api/routers/workflow_runs.py	Workflow execution
api/routers/agent.py	Agent-related operations

Each router encapsulates related functionality and is imported into api/main.py for registration.

Generator Registry: The Bridge to AI Inference

services/generator_registry.GeneratorRegistry

Located in api/services/generator_registry.py, the GeneratorRegistry is the central coordination point between the HTTP layer and model implementations. It performs three core responsibilities:

• Discovery: Scans EXTENSIONS_DIR for valid extensions (folders containing manifest.json)
• Instantiation: Creates either direct BaseGenerator instances or ExtensionProcess wrappers based on extension configuration
• Lifecycle management: Handles model switching, loading, unloading, and status tracking

Environment variables configure critical paths (lines 24-31 in generator_registry.py):

EXTENSIONS_DIR  # Where user extensions are located

WORKSPACE_DIR   # Output directory for generated files

MODELS_DIR      # Cache for downloaded model checkpoints

services.generators.base.BaseGenerator

All model adapters must inherit from the abstract base class defined in api/services/generators/base.py. The contract requires implementation of:

• generate() – Execute inference and write output to workspace
• load() – Initialize model weights and move to target device
• unload() – Free GPU memory and resources
• is_loaded() – Check load state
• is_downloaded() – Verify model weights are cached locally

services.extension_process.ExtensionProcess

For extensions shipping with their own virtual environment, api/services/extension_process.py provides a subprocess wrapper that implements the same public interface as BaseGenerator. This allows the rest of the API to treat both execution modes uniformly—whether running in-process or isolated.

Request-Response Flow: End-to-End Interaction

The interface between FastAPI backend and model inference follows a structured pipeline:

1. Application startup – FastAPI invokes the lifespan context; GeneratorRegistry.initialize() discovers extensions and prepares the workspace

2. Request routing – HTTP requests route to appropriate APIRouter endpoints; for example, POST /generate/from-image hits api/routers/generation.py

3. Model selection – The endpoint validates model_id and calls generator_registry.switch_model(model_id) to activate the requested generator

4. Model loading – generator_registry.get_active() triggers on-demand download (if needed) and loads the model in a background thread with progress callbacks—keeping the route non-blocking

5. Generation execution – A background task (_run_generation) invokes the generator's generate method with image bytes, parameters, progress callback, and optional cancellation event; output writes to WORKSPACE_DIR/<collection>

6. Result serving – The endpoint returns a job ID for polling; completed results are accessible via /workspace/<relative-path> served by serve_workspace_file

Adding Custom Model Extensions

The architecture enables hot-pluggable models without modifying core API code:

1. Create extension folder: /path/to/extensions/my_model/

2. Add manifest.json:

{
  "id": "my_model",
  "name": "My Model",
  "generator_class": "MyGenerator",
  "type": "model",
  "nodes": []
}

3. Implement generator.py subclassing BaseGenerator

4. Set EXTENSIONS_DIR=/path/to/extensions and restart Modly

The GeneratorRegistry automatically discovers and registers the extension on startup.

Calling the Generation API

POST /generate/from-image HTTP/1.1
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary

------WebKitFormBoundary
Content-Disposition: form-data; name="image"; filename="example.png"
Content-Type: image/png

<binary PNG data>
------WebKitFormBoundary
Content-Disposition: form-data; name="model_id"

sf3d
------WebKitFormBoundary
Content-Disposition: form-data; name="params"

{"prompt":"A futuristic cityscape"}
------WebKitFormBoundary--

The response contains a job_id. Poll GET /generate/status/{job_id} until status is "done", then retrieve the result at the provided output_url.

Summary

• api/main.py instantiates FastAPI, configures middleware, registers routers, and manages application lifespan
• APIRouter modules in api/routers/ group endpoints by domain with optional URL prefixes
• GeneratorRegistry discovers extensions, manages model lifecycle, and abstracts execution mode (direct or subprocess)
• BaseGenerator defines the contract all model adapters must implement
• ExtensionProcess provides isolation for extensions with custom virtual environments
• Background task execution keeps HTTP routes responsive during model loading and generation
• Plugin-based extension system allows adding new models by dropping folders into EXTENSIONS_DIR

Frequently Asked Questions

What file serves as the FastAPI application entry point in Modly?

api/main.py serves as the entry point. It creates the FastAPI instance with lifespan context management, adds CORS middleware, includes all routers with their prefixes, and implements workspace file serving.

How does Modly isolate extensions that require different Python dependencies?

Modly uses ExtensionProcess in api/services/extension_process.py to wrap such extensions. This class runs the generator in a separate subprocess with its own virtual environment while exposing the same interface as direct BaseGenerator instances, allowing uniform treatment by the registry and routers.

Can new AI models be added without modifying Modly's source code?

Yes. Place a folder containing manifest.json and a generator.py implementing BaseGenerator into the directory specified by EXTENSIONS_DIR. The GeneratorRegistry automatically discovers and loads it on the next startup.

How does Modly prevent model loading from blocking HTTP requests?

Both model loading and generation execute in background threads managed through FastAPI's BackgroundTask or asyncio mechanisms. Progress callbacks update job status while the initial HTTP response returns immediately with a job ID for polling.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}