How to Add Custom Models to llmfit Without Recompiling

You can add custom models to llmfit without recompiling by creating a custom_models.json overlay file that the runtime loader merges with the embedded catalog, optionally overriding the path via the LLMFIT_CUSTOM_MODELS environment variable.

llmfit is a Rust-based CLI tool that matches Large Language Models (LLMs) to your hardware constraints. While the binary ships with embedded JSON catalogs for Hugging Face and ONNX models, the source code in llmfit-core/src/models.rs implements a runtime overlay mechanism that lets you extend or override these definitions without rebuilding the binary.

Understanding the Runtime Overlay Mechanism

The ModelDatabase::new() constructor in llmfit-core/src/models.rs (lines 1551-1574) implements a two-phase loading strategy. First, it deserializes the embedded hf_models.json and onnx_models.json catalogs compiled into the binary. Then it calls custom_models_file() to determine the overlay path and load_custom_models_from() to parse any user-defined entries.

If a custom entry shares the same canonical slug as an embedded model, the custom definition replaces the built-in one. Otherwise, the entry is appended to the list. This merge happens entirely at runtime, meaning the binary reads your modifications from disk every time it executes.

Locating the Custom Models File

By default, llmfit looks for the overlay file using the custom_models_file helper function (lines 1414-1422). On Linux, this resolves to ~/.local/share/llmfit/custom_models.json within the update cache directory. The path varies by platform following OS conventions for application data.

You can override this location by setting the LLMFIT_CUSTOM_MODELS environment variable to an absolute path before invoking the CLI:

export LLMFIT_CUSTOM_MODELS="/path/to/my_custom_models.json"
llmfit fit --perfect

The load_custom_models_from function (lines 1428-1439) handles the actual disk I/O. If the file does not exist, the loader silently continues with the embedded catalog only. If the file contains malformed JSON, the program emits a warning to stderr but does not crash, ensuring robustness against editing errors.

Creating a Valid custom_models.json

Your overlay must follow the same schema as the embedded hf_models.json. The JSON file should contain an array of model objects. Required fields include name, provider, parameter_count, min_ram_gb, recommended_ram_gb, min_vram_gb, quantization, context_length, and use_case.

Optional metadata such as capabilities, languages, license, and hardware flags like is_moe improve filtering in the TUI but can be omitted.

Create the file with a structure like this:

[
  {
    "name": "my-org/MyCustom-7B",
    "provider": "my-org",
    "parameter_count": "7B",
    "min_ram_gb": 12.5,
    "recommended_ram_gb": 24.0,
    "min_vram_gb": 8.5,
    "quantization": "Q4_K_M",
    "context_length": 4096,
    "use_case": "general",
    "is_moe": false,
    "capabilities": ["chat", "completion"],
    "languages": ["en"],
    "format": "gguf",
    "hf_downloads": 0,
    "hf_likes": 0,
    "license": "MIT"
  }
]

Save this to the default location or the path specified by LLMFIT_CUSTOM_MODELS.

Step-by-Step: Adding Your First Custom Model

Follow these steps to register a new model without touching the Rust source:

1. Create the cache directory if it does not exist:

   mkdir -p ~/.local/share/llmfit

2. Write the overlay file using the schema above. Ensure valid JSON syntax; the parser will reject trailing commas or unquoted keys.

3. Verify the merge by listing models matching your entry:

   llmfit list "MyCustom"

   The output should display your custom entry with the specified VRAM and RAM requirements.

4. Override existing models (optional) by using the same name slug as a built-in model. Your custom values for min_vram_gb or quantization will take precedence in the llmfit-core/src/fit.rs calculations.

Summary

• Runtime overlay: llmfit merges custom_models.json with embedded catalogs at startup via ModelDatabase::new().
• File location: Default is the platform-specific cache directory; override with LLMFIT_CUSTOM_MODELS.
• Schema compliance: Match the hf_models.json structure with required fields like name, provider, and min_ram_gb.
• Safe loading: Missing files are ignored; malformed JSON triggers warnings but not crashes.
• Override capability: Custom entries with matching slugs replace built-in definitions.

Frequently Asked Questions

What happens if my custom_models.json contains syntax errors?

The load_custom_models_from function catches parse errors and logs a warning to stderr, then continues execution using only the embedded model catalog. The program will not crash, but your custom models will not appear in the list until the JSON is valid.

Can I override existing Hugging Face models with custom parameters?

Yes. If your custom entry uses the same name (canonical slug) as an embedded model from hf_models.json, your version replaces the built-in definition entirely. This is useful for adjusting VRAM estimates or quantization defaults for specific hardware configurations.

Where is the default custom_models.json path on Windows and macOS?

The custom_models_file function resolves platform-specific cache directories. On macOS, this typically resides in ~/Library/Application Support/llmfit/custom_models.json, while Windows uses the %LOCALAPPDATA%\llmfit\custom_models.json path. You can always override these defaults using the LLMFIT_CUSTOM_MODELS environment variable.

Do I need to restart llmfit after editing the overlay file?

No recompilation is necessary, but you must restart the llmfit process. The JSON file is read once during ModelDatabase initialization at startup. Changes made while the TUI is running will not appear until the next invocation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}