# How Modly Uses Local AI Models for Image-to-3D Mesh Generation

> Discover how Modly uses local AI models for image to 3D mesh generation. Run inference pipelines entirely on-device and export GLB files without cloud APIs.

- Repository: [lightningpixel/modly](https://github.com/lightningpixel/modly)
- Tags: how-to-guide
- Published: 2026-08-21

---

**Modly generates 3D meshes from 2D images entirely on-device by loading local AI model weights from Hugging Face into specialized generator adapters that run inference pipelines and export GLB files without requiring cloud APIs.**

Modly is an open-source tool that enables fully offline image-to-3D mesh generation by orchestrating local AI models through a modular Python architecture. Unlike cloud-dependent services, Modly downloads model weights once to your local machine and executes inference pipelines entirely on your hardware, ensuring privacy and eliminating API latency.

## Local Model Architecture and the Generator Registry

When Modly initializes, the **generator registry** discovers all available model adapters under `api/services/generators/`. Each adapter subclasses `BaseGenerator` (defined in [`api/services/generators/base.py`](https://github.com/lightningpixel/modly/blob/main/api/services/generators/base.py)) and declares three critical attributes: a unique `MODEL_ID`, a downloadable **Hugging Face** repository (`hf_repo`), and an abstract `generate` method that accepts raw image bytes and returns a mesh file path.

The registry activates models via `generator_registry.switch_model()`, which verifies that weight files exist in `MODELS_DIR/<model_id>`. If weights are missing, `BaseGenerator._auto_download` (lines 44-69 in [`api/services/generators/base.py`](https://github.com/lightningpixel/modly/blob/main/api/services/generators/base.py)) fetches them from the specified Hugging Face repository. Users can switch active models via the `/model/switch` API endpoint or the `--model` CLI flag. Once downloaded, all operations run offline with no external API calls.

## The Image-to-Mesh Inference Pipeline

Generation requests enter through two interfaces: the HTTP API at `/generate/from-image` or the `modly generate` CLI command. The request handler in [`api/routers/generation.py`](https://github.com/lightningpixel/modly/blob/main/api/routers/generation.py) (lines 39-61) extracts the uploaded image, resolves the active `model_id` from the registry, and delegates to the generator's `generate` method.

Inside a concrete generator implementation (such as the **sf3d** adapter), the `generate` method decodes the input image bytes, executes the neural network forward pass using locally stored weights, and writes the raw mesh to a temporary file. The method returns the path to a **GLB** file (converted from intermediate `.ply` or `.obj` formats), which the router wraps in a JSON response containing progress metadata and file locations.

## Post-Processing Workflow for Mesh Optimization

After the AI generator produces the raw mesh, Modly applies a configurable chain of **workflow nodes** to improve geometric quality:

- **Mesh Remesher**: Located at [`src/areas/workflows/nodes/mesh-remesher/processor.py`](https://github.com/lightningpixel/modly/blob/main/src/areas/workflows/nodes/mesh-remesher/processor.py), this node restructures mesh topology using PyMeshLab with options for quad, triangle, or no remeshing.

- **Mesh Repair**: The processor at [`src/areas/workflows/nodes/mesh-repair/processor.py`](https://github.com/lightningpixel/modly/blob/main/src/areas/workflows/nodes/mesh-repair/processor.py) fixes non-manifold edges, removes duplicate vertices, and closes holes in the geometry.

- **Mesh Smoother**: Implemented in [`src/areas/workflows/nodes/mesh-smoother/processor.py`](https://github.com/lightningpixel/modly/blob/main/src/areas/workflows/nodes/mesh-smoother/processor.py), this applies Taubin or Laplacian smoothing algorithms to create cleaner surfaces.

These processors execute sequentially after generation completes, controlled by the CLI command logic in [`tools/modly-cli/agent.py`](https://github.com/lightningpixel/modly/blob/main/tools/modly-cli/agent.py) (lines 869-898).

## Running Modly Locally

The following commands demonstrate the complete local workflow:

```bash

# 1. Select and download the model (run once)

modly model switch --model sf3d

# 2. Generate mesh with default quad remeshing

modly generate --image ./photo.png --output ./photo.glb

# 3. Debug mode - generate without post-processing

modly generate --image ./photo.png --no-export

# 4. Direct HTTP API usage

curl -X POST "http://localhost:8000/generate/from-image?model_id=sf3d" \
     -F "image=@photo.png" \
     -F "params={}" \
     -H "Accept: application/json"

```

All model files persist in `MODELS_DIR`; subsequent generations require no internet connectivity, enabling complete offline operation.

## Summary

- Modly uses a **generator registry** pattern to discover and manage local AI model adapters under `api/services/generators/`.
- The **BaseGenerator** abstract class in [`api/services/generators/base.py`](https://github.com/lightningpixel/modly/blob/main/api/services/generators/base.py) defines the contract for downloading weights from Hugging Face and running inference.
- Image uploads route through [`api/routers/generation.py`](https://github.com/lightningpixel/modly/blob/main/api/routers/generation.py), which delegates to the active generator's `generate` method for local processing.
- Post-processing nodes (remesher, repair, smoother) refine raw AI output into production-ready meshes.
- The entire pipeline operates offline after initial model download, storing weights in `MODELS_DIR/<model_id>`.

## Frequently Asked Questions

### Does Modly require an internet connection to generate 3D meshes?

No. Modly only requires internet connectivity for the initial download of model weights from Hugging Face. Once `BaseGenerator._auto_download` caches the files in `MODELS_DIR`, all image-to-3D conversion happens locally without external API calls.

### What file formats does Modly support for output?

Modly primarily exports to **GLB** (GL Transmission Format Binary) for final delivery. During processing, generators may produce intermediate formats like `.ply` or `.obj` before conversion. The export format can be configured via CLI flags or API parameters in the generation router.

### How do I switch between different AI models in Modly?

You can switch models using the CLI command `modly model switch --model <model_id>` or by calling the `/model/switch` HTTP endpoint. The generator registry updates the active model reference and verifies that weights exist in the local `MODELS_DIR` before allowing inference.

### Can I skip the post-processing steps to get raw mesh output?

Yes. Use the `--no-export` flag with the `modly generate` command to bypass the remeshing, repair, and smoothing workflow nodes. This returns the raw generator output directly, which is useful for debugging or when you need unprocessed geometry.