How to Add Custom Models to llmfit's Catalog Without Recompiling

Create a JSON overlay file at your platform's user data directory (~/.local/share/llmfit/custom_models.json on Linux) and llmfit will automatically merge your custom entries into the built-in catalog at runtime.

The llmfit tool ships with an embedded catalog of models, but you don't need to rebuild the Rust binary to extend it. The llmfit-core library provides a runtime overlay mechanism that reads user-defined models from a JSON file and merges them into the catalog. This approach lets you add proprietary, fine-tuned, or newly released models without modifying source code or waiting for upstream updates.

Where llmfit Looks for Custom Models

The function custom_models_file() in llmfit-core/src/models.rs (lines 1466-1470) returns the platform-specific path:

• Linux: ~/.local/share/llmfit/custom_models.json
• macOS: ~/Library/Application Support/llmfit/custom_models.json
• Windows: %APPDATA%\Roaming\llmfit\custom_models.json

This path is constructed using the same cache_dir() utility found in llmfit-core/src/update.rs, ensuring consistency with other user-level data storage.

How the Overlay Loading Works

When the model database initializes, load_custom_models_from() (lines 1477-1519) handles the merge logic:

• File missing: Silently ignored—no warning, no error
• Parse success: Custom entries replace any built-in model with the same slug (canonicalized name)
• Parse failure: Warning emitted, but llmfit continues with the embedded catalog

This replacement behavior means you can override existing entries to correct parameters or update quantization formats without touching the source.

Required JSON Schema for Custom Models

Each entry must match the LlmModel struct defined around line 961-970. The minimal required structure:

{
  "name": "my-custom-model",
  "provider": "local",
  "parameter_count": 7000000000,
  "min_ram_gb": 4.5,
  "recommended_ram_gb": 9.0,
  "min_vram_gb": 3.0,
  "quantization": "Q4_K_M",
  "context_length": 8192,
  "use_case": "general"
}

Key field definitions based on the source:

• name: Unique identifier used as the slug (canonicalized internally)
• provider: Typically "local" for custom GGUF files or alternative backends
• parameter_count: Integer count of model parameters
• min_ram_gb / recommended_ram_gb: System memory thresholds for analysis
• min_vram_gb: GPU memory floor for fit calculations
• quantization: Format string (e.g., "Q4_K_M", "Q5_K_S", "FP16")
• context_length: Maximum token context window
• use_case: Category string for filtering ("general", "coding", "chat", etc.)

Step-by-Step: Adding Your First Custom Model

1. Create the overlay directory and file

# Linux/macOS

mkdir -p ~/.local/share/llmfit
touch ~/.local/share/llmfit/custom_models.json

# Windows PowerShell

New-Item -ItemType Directory -Force "$env:APPDATA\llmfit"
New-Item -ItemType File "$env:APPDATA\llmfit\custom_models.json"

2. Populate custom_models.json with one or more models

[
  {
    "name": "my-custom-7b-coder",
    "provider": "local",
    "parameter_count": 7000000000,
    "min_ram_gb": 4.5,
    "recommended_ram_gb": 9.0,
    "min_vram_gb": 3.0,
    "quantization": "Q4_K_M",
    "context_length": 16384,
    "use_case": "coding"
  },
  {
    "name": "my-tiny-1.5b-model",
    "provider": "local",
    "parameter_count": 1500000000,
    "min_ram_gb": 1.2,
    "recommended_ram_gb": 2.5,
    "min_vram_gb": 0.8,
    "quantization": "Q4_K_M",
    "context_length": 4096,
    "use_case": "general"
  }
]

3. Verify custom models appear in the catalog

# List all models and filter for your custom entry

llmfit --list-models | grep my-custom

The output should show both my-custom-7b-coder and my-tiny-1.5b-model alongside llmfit's built-in HuggingFace models.

Overriding Built-In Models

To replace an existing catalog entry, use the same name field as the embedded model. For example, to override the quantization or RAM estimates for a popular model, check its slug in the source hf_models.json or via --list-models, then create a custom entry with matching name. The overlay takes precedence during the merge.

Why No Recompilation Is Required

The custom model overlay is pure runtime configuration:

• llmfit-core deserializes the JSON at database initialization
• No include_str! or compile-time embedding of custom data
• The binary contains only the logic to locate and parse the file

This design separates code from data, enabling rapid iteration on model parameters without Cargo, rustc, or CI pipelines.

Key Source Files Reference

File	Purpose	Link
llmfit-core/src/models.rs	LlmModel struct, custom_models_file(), load_custom_models_from()	models.rs
llmfit-core/src/update.rs	cache_dir() helper for platform paths	update.rs
llmfit-core/src/analysis.rs	Consumes the merged model list for fit analysis	analysis.rs

Summary

• Location: Platform-specific user data directory (~/.local/share/llmfit/ on Linux)
• Mechanism: JSON overlay file custom_models.json merged at runtime by load_custom_models_from()
• Behavior: Custom entries replace same-named built-in models; missing files ignored silently
• Schema: Match the LlmModel struct fields—name, provider, memory specs, quantization, context_length, use_case
• No rebuild: The binary reads the file dynamically; edit and restart llmfit to apply changes

Frequently Asked Questions

Can I use the overlay to remove or disable built-in models?

No. The overlay only adds or replaces entries—it cannot delete from the embedded catalog. To effectively hide a model, replace its entry with implausible parameters (e.g., min_ram_gb: 9999) so it fails fit criteria, or filter at the CLI level.

Does llmfit validate the custom JSON schema strictly?

Partially. load_custom_models_from() uses Serde deserialization; missing required fields or type mismatches trigger a parse warning and skip the file. Extra fields are generally ignored per Rust's typical deserialization behavior, but rely only on documented LlmModel fields for forward compatibility.

Will my custom models persist across llmfit updates?

Yes. The overlay lives in your user data directory, not the installation path. Updating the binary via cargo install or package managers leaves ~/.local/share/llmfit/ untouched.

Can I specify remote (API-based) models instead of local GGUF files?

The provider field supports different backends, but as of the current source, "local" is the primary tested path for custom entries. API-based providers like OpenAI or Anthropic models are typically built into the embedded catalog. Check models.rs for the latest Provider enum variants if experimenting with remote custom endpoints.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}