How Modly Uses Local AI Models for Image-to-3D Mesh Generation

Modly generates 3D meshes from 2D images entirely on-device by loading local AI model weights from Hugging Face into specialized generator adapters that run inference pipelines and export GLB files without requiring cloud APIs.

Modly is an open-source tool that enables fully offline image-to-3D mesh generation by orchestrating local AI models through a modular Python architecture. Unlike cloud-dependent services, Modly downloads model weights once to your local machine and executes inference pipelines entirely on your hardware, ensuring privacy and eliminating API latency.

Local Model Architecture and the Generator Registry

When Modly initializes, the generator registry discovers all available model adapters under api/services/generators/. Each adapter subclasses BaseGenerator (defined in api/services/generators/base.py) and declares three critical attributes: a unique MODEL_ID, a downloadable Hugging Face repository (hf_repo), and an abstract generate method that accepts raw image bytes and returns a mesh file path.

The registry activates models via generator_registry.switch_model(), which verifies that weight files exist in MODELS_DIR/<model_id>. If weights are missing, BaseGenerator._auto_download (lines 44-69 in api/services/generators/base.py) fetches them from the specified Hugging Face repository. Users can switch active models via the /model/switch API endpoint or the --model CLI flag. Once downloaded, all operations run offline with no external API calls.

The Image-to-Mesh Inference Pipeline

Generation requests enter through two interfaces: the HTTP API at /generate/from-image or the modly generate CLI command. The request handler in api/routers/generation.py (lines 39-61) extracts the uploaded image, resolves the active model_id from the registry, and delegates to the generator's generate method.

Inside a concrete generator implementation (such as the sf3d adapter), the generate method decodes the input image bytes, executes the neural network forward pass using locally stored weights, and writes the raw mesh to a temporary file. The method returns the path to a GLB file (converted from intermediate .ply or .obj formats), which the router wraps in a JSON response containing progress metadata and file locations.

Post-Processing Workflow for Mesh Optimization

After the AI generator produces the raw mesh, Modly applies a configurable chain of workflow nodes to improve geometric quality:

• Mesh Remesher: Located at src/areas/workflows/nodes/mesh-remesher/processor.py, this node restructures mesh topology using PyMeshLab with options for quad, triangle, or no remeshing.

• Mesh Repair: The processor at src/areas/workflows/nodes/mesh-repair/processor.py fixes non-manifold edges, removes duplicate vertices, and closes holes in the geometry.

• Mesh Smoother: Implemented in src/areas/workflows/nodes/mesh-smoother/processor.py, this applies Taubin or Laplacian smoothing algorithms to create cleaner surfaces.

These processors execute sequentially after generation completes, controlled by the CLI command logic in tools/modly-cli/agent.py (lines 869-898).

Running Modly Locally

The following commands demonstrate the complete local workflow:

# 1. Select and download the model (run once)

modly model switch --model sf3d

# 2. Generate mesh with default quad remeshing

modly generate --image ./photo.png --output ./photo.glb

# 3. Debug mode - generate without post-processing

modly generate --image ./photo.png --no-export

# 4. Direct HTTP API usage

curl -X POST "http://localhost:8000/generate/from-image?model_id=sf3d" \
     -F "image=@photo.png" \
     -F "params={}" \
     -H "Accept: application/json"

All model files persist in MODELS_DIR; subsequent generations require no internet connectivity, enabling complete offline operation.

Summary

• Modly uses a generator registry pattern to discover and manage local AI model adapters under api/services/generators/.
• The BaseGenerator abstract class in api/services/generators/base.py defines the contract for downloading weights from Hugging Face and running inference.
• Image uploads route through api/routers/generation.py, which delegates to the active generator's generate method for local processing.
• Post-processing nodes (remesher, repair, smoother) refine raw AI output into production-ready meshes.
• The entire pipeline operates offline after initial model download, storing weights in MODELS_DIR/<model_id>.

Frequently Asked Questions

Does Modly require an internet connection to generate 3D meshes?

No. Modly only requires internet connectivity for the initial download of model weights from Hugging Face. Once BaseGenerator._auto_download caches the files in MODELS_DIR, all image-to-3D conversion happens locally without external API calls.

What file formats does Modly support for output?

Modly primarily exports to GLB (GL Transmission Format Binary) for final delivery. During processing, generators may produce intermediate formats like .ply or .obj before conversion. The export format can be configured via CLI flags or API parameters in the generation router.

How do I switch between different AI models in Modly?

You can switch models using the CLI command modly model switch --model <model_id> or by calling the /model/switch HTTP endpoint. The generator registry updates the active model reference and verifies that weights exist in the local MODELS_DIR before allowing inference.

Can I skip the post-processing steps to get raw mesh output?

Yes. Use the --no-export flag with the modly generate command to bypass the remeshing, repair, and smoothing workflow nodes. This returns the raw generator output directly, which is useful for debugging or when you need unprocessed geometry.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}