Using llmfit with MLX on Apple Silicon: Setup and Configuration Guide

llmfit supports MLX inference exclusively on Apple Silicon devices that expose the Metal GPU backend and unified memory architecture, automatically detecting compatible hardware in llmfit-core/src/hardware.rs and filtering out MLX-only models on incompatible platforms.

The llmfit repository provides a Rust-based CLI tool for benchmarking large language model inference performance across different hardware configurations. For developers working on Apple Silicon Macs, llmfit integrates natively with Apple's MLX machine learning framework to leverage GPU acceleration, but this functionality is gated by strict hardware detection logic to ensure compatibility.

Hardware Requirements and Auto-Detection

MLX support in llmfit is not merely a configuration option—it is a hardware-gated feature. According to the source code in llmfit-core/src/hardware.rs, the system initializes the MLX runtime only when two specific conditions are met: the presence of GpuBackend::Metal and the availability of unified memory. These checks occur during the is_apple_silicon flag computation in llmfit-tui/src/main.rs.

When the tool detects an incompatible environment (Linux, Windows, or Intel-based Macs without Metal support), it explicitly filters out MLX-quantized models from the results. This prevents users from attempting to load models that require the MLX runtime on unsupported hardware. The detection logic ensures that MLX-only models never appear in the fit results on non-Apple Silicon systems.

How llmfit Discovers MLX Models

The model discovery process relies on a specific naming convention and local cache inspection. As implemented in llmfit-core/src/providers.rs, the MlxProvider scans the local Hugging Face cache directory for repositories containing the "-MLX" suffix in their names (e.g., Qwen3-8B-MLX-4bit).

When the provider identifies these models, it sets the is_mlx_only flag defined in llmfit-core/src/models.rs and assigns the InferenceRuntime::Mlx variant to their runtime field. This metadata tagging ensures that the fit engine routes these specific models through the MLX inference path while maintaining compatibility with standard PyTorch or GGUF models in the same workspace.

Configuring the MLX Runtime

By default, llmfit communicates with a local MLX server running at http://localhost:8080. You can override this endpoint using the MLX_LM_HOST environment variable, which the MlxProvider::default() constructor reads during initialization (lines 45-55 in llmfit-core/src/providers.rs).

The runtime selection respects explicit user overrides through the CLI. When you specify --force-runtime mlx, the argument parser in llmfit-tui/src/main.rs (lines 13-28) maps the string "mlx" to the InferenceRuntime::Mlx enum variant, bypassing auto-detection and forcing the fit engine to use the MlxProvider implementation regardless of other available backends.

CLI Commands for MLX Workflows

Force MLX Runtime on Apple Silicon

To explicitly run a performance fit using the MLX backend on a compatible Mac:

cargo run -- fit --force-runtime mlx

This command triggers the MlxProvider::detect_with_installed method (lines 26-34 in llmfit-core/src/providers.rs) to establish a connection to the local MLX server and compute throughput estimates using Metal GPU acceleration.

Filter for MLX-Compatible Models Only

When browsing available models, restrict the output to those supporting MLX quantization:

cargo run -- list --runtime mlx

The runtime filter logic in llmfit-tui/src/main.rs (lines 60-66) retains only entries where the runtime field equals InferenceRuntime::Mlx, hiding GGUF or PyTorch variants that cannot execute on the MLX framework.

Override the MLX Server Endpoint

If your MLX server runs on a non-standard port or remote host within your network:

export MLX_LM_HOST=http://127.0.0.1:8001
cargo run -- fit --force-runtime mlx

The provider falls back to http://localhost:8080 only if the MLX_LM_HOST variable is unset or contains a malformed URL.

Platform Restrictions and Filtering

The enforcement of Apple Silicon exclusivity happens at the application layer, not just the provider level. In llmfit-tui/src/main.rs, the application computes the is_apple_silicon boolean flag early in the execution lifecycle. If this flag evaluates to false, the system applies a pre-filter to the model registry that removes all entries marked with Model::is_mlx_only before displaying them to the user.

This design prevents runtime errors and confusing failures that would occur if a user attempted to load MLX-quantized weights on CUDA or CPU-only environments. The UI components in llmfit-tui/src/tui_ui.rs display the MLX detection status to confirm when the accelerated path is active.

Summary

• Apple Silicon Required: llmfit enables MLX support only on macOS systems with Metal GPU backend and unified memory, as verified in llmfit-core/src/hardware.rs.
• Model Tagging: MLX models are identified by the "-MLX" suffix in their Hugging Face repository names and flagged with is_mlx_only in llmfit-core/src/models.rs.
• Server Communication: The MlxProvider queries http://localhost:8080 by default, configurable via the MLX_LM_HOST environment variable.
• CLI Overrides: Use --force-runtime mlx to bypass auto-detection, or --runtime mlx to filter model lists.
• Cross-Platform Safety: Non-Apple Silicon systems automatically hide MLX-only models to prevent incompatible execution attempts.

Frequently Asked Questions

Can I use llmfit with MLX on Windows or Linux?

No. According to the source code in llmfit-tui/src/main.rs, llmfit explicitly filters out MLX-only models on non-Apple Silicon platforms. The MLX runtime requires the Metal GPU backend (GpuBackend::Metal) and unified memory architecture found exclusively in Apple Silicon Macs. On Windows or Linux, llmfit will not display or load models with the "-MLX" suffix, even if you attempt to force the runtime manually.

What naming convention do MLX models need in llmfit?

MLX-compatible models must include "-MLX" in their Hugging Face repository name (for example, Qwen3-8B-MLX-4bit). The MlxProvider implementation in llmfit-core/src/providers.rs scans the local cache for this substring to identify compatible checkpoints. Models without this suffix are treated as standard PyTorch or GGUF formats and routed to their respective inference runtimes.

Does llmfit require a running MLX server before execution?

Yes. The MlxProvider expects a local MLX server accessible at http://localhost:8080 by default. You can verify connectivity or specify an alternative endpoint using the MLX_LM_HOST environment variable. If the server is unreachable, the detect_with_installed method will fail to compute throughput estimates, and the fit operation will exit with a connection error rather than falling back to CPU execution.

How does llmfit handle mixed hardware environments?

On systems that meet the Apple Silicon criteria (Metal backend + unified memory), llmfit displays both MLX and non-MLX models, allowing you to benchmark across different quantization schemes. However, the application computes the is_apple_silicon flag at startup and applies a filter in the TUI layer that removes MLX-only entries from the model list on incompatible hardware, ensuring that users cannot accidentally attempt to load Metal-specific weights on CUDA or CPU-only configurations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}