Why MLX Runtime Produces Faster Estimates Than llama.cpp on Apple Silicon in LLMFIT

LLMFIT’s roof-line performance model assigns higher effective bandwidth values to the MLX runtime because it interfaces directly with Apple’s Metal GPU driver, while llama.cpp relies on a generic GGUF/Metal abstraction that reports conservative bandwidth limits, resulting in significantly higher predicted tokens-per-second for MLX on Apple Silicon.

The LLMFIT project implements a hardware-aware performance estimation framework that calculates theoretical throughput for large language model inference using runtime-specific roof-line models. When comparing inference backends on Apple Silicon devices, the system consistently generates higher token-per-second (TPS) estimates for the MLX runtime compared to llama.cpp. This prediction gap derives from fundamental architectural differences in how each runtime exposes Apple’s unified memory bandwidth and GPU driver capabilities to the calculator.

How LLMFIT Calculates Throughput Using Roof-Line Analysis

In llmfit-core/src/fit.rs, the estimate_tps function implements a roof-line performance model that multiplies a runtime’s computational ceiling by the hardware’s effective memory bandwidth. The critical logic resides in the gpu_bandwidth_roofline branch between lines 998-1015, where the system selects bandwidth coefficients based on the specific InferenceRuntime variant.

The function evaluates the InferenceRuntime enum defined in llmfit-core/src/providers.rs, which distinguishes between Mlx and LlamaCpp variants to apply runtime-specific bandwidth constants. Because MLX can query the true memory bandwidth of the Apple GPU through the unified memory system, the model assigns it a higher bandwidth value than llama.cpp, which is constrained by the generic Metal abstraction layer used by the GGUF format.

Architectural Advantages of MLX on Apple Silicon

Three technical factors explain why LLMFIT assigns higher performance coefficients to MLX when estimating throughput on Apple Silicon:

Direct Metal Driver Integration

The MLX runtime communicates directly with Apple’s Metal GPU driver, enabling accurate queries of the GPU’s true memory bandwidth for the roof-line calculation. This direct access allows the performance model to utilize the full bandwidth potential of the unified memory architecture. In contrast, llama.cpp operates through the GGUF format and a generic OpenCL/Metal abstraction layer that masks actual hardware capabilities, forcing the estimator to apply conservative bandwidth limits that reduce the predicted TPS.

Unified Memory Architecture Utilization

MLX treats VRAM as system RAM within Apple’s unified memory architecture, eliminating redundant data copies between CPU and GPU buffers. According to the implementation notes in docs/platform-support.md, this avoids the extra data movement overhead inherent in llama.cpp’s separate GPU/CPU memory paths. LLMFIT’s performance model accounts for this efficiency gain by assigning higher effective throughput values to the MLX memory model.

Runtime-Specific Quantization Optimization

MLX employs custom mlx-8bit and mlx-4bit quantization hierarchies specifically tuned for Metal performance. The llama.cpp runtime utilizes GGUF quantization formats that prioritize cross-platform compatibility over Metal-specific optimizations. Because quantization efficiency directly impacts memory bandwidth requirements in the roof-line model, MLX’s optimized formats yield higher estimated tokens-per-second calculations.

The Speed-Up Detection Logic in fit.rs

The codebase explicitly compares MLX and llama.cpp performance when generating estimates for Apple Silicon platforms. In llmfit-core/src/fit.rs, the implementation calculates the percentage speed-up only when the MLX runtime is selected:

if runtime == InferenceRuntime::Mlx {
    let llamacpp_tps = estimate_tps(..., InferenceRuntime::LlamaCpp, &config);
    // If llama.cpp is slower, report the speed‑up percentage
    if llamacpp_tps > 0.1 {
        let speedup = ((estimated_tps / llamacpp_tps - 1.0) * 100.0).round();
        if speedup > 0.0 {
            notes.push(format!(
                "MLX runtime: ~{:.0}% faster than llama.cpp ({:.1} vs {:.1} tok/s)",
                speedup, estimated_tps, llamacpp_tps
            ));
        }
    }
}

This conditional logic generates the "~X% faster than llama.cpp" annotations visible in CLI output and API responses.

Observing the Performance Difference

Developers can view these comparative estimates through both command-line interaction and programmatic API access.

CLI Usage

Running a fit operation for an MLX-quantized model triggers the automatic comparison logic:

# Run a fit for a model available for both runtimes

cargo run -- fit --model Qwen2-0.5B-MLX-4bit

The resulting output displays the calculated architectural advantage:

▸ Model: Qwen2-0.5B-MLX-4bit
▸ Runtime: MLX
▸ Estimated speed: 120.5 tok/s
▸ MLX runtime: ~23% faster than llama.cpp (120.5 vs 98.1 tok/s)

Programmatic Access

Applications using the LLMFIT core library can inspect the notes field of model fit results to retrieve the comparison data:

let model_fits = llmfit_core::analysis::build_model_fits(&specs, &db, &config)?;
for fit in model_fits {
    if fit.runtime == InferenceRuntime::Mlx {
        for note in &fit.notes {
            println!("{}", note);
        }
    }
}

Summary

• Roof-line bandwidth selection: The estimate_tps function in llmfit-core/src/fit.rs assigns higher effective bandwidth values to MLX than llama.cpp due to direct Metal driver access versus generic abstraction layers.
• Unified memory efficiency: MLX eliminates GPU/CPU data copy overhead by treating VRAM as system RAM, while llama.cpp maintains separate memory paths that reduce effective bandwidth.
• Quantization optimizations: MLX-specific 4-bit and 8-bit formats are tuned for Metal performance compared to llama.cpp’s cross-platform GGUF quantizations.
• Automated comparison reporting: The system automatically calculates and displays percentage speed-ups when generating estimates for MLX models on Apple Silicon hardware.

Frequently Asked Questions

Why does LLMFIT estimate different speeds for the same model on identical Apple Silicon hardware?

LLMFIT applies runtime-specific bandwidth coefficients in its roof-line calculations defined in llmfit-core/src/fit.rs. Because MLX interfaces directly with the Metal driver while llama.cpp uses a generic abstraction layer that cannot expose full hardware capabilities, the system assigns higher effective memory bandwidth to MLX, resulting in elevated predicted token-per-second rates for the same physical device.

Is the MLX runtime actually faster in production, or only in LLMFIT estimates?

The estimates reflect real architectural advantages documented in docs/platform-support.md. MLX’s unified memory integration and Metal-specific optimizations generally translate to measurable performance gains during actual inference on Apple Silicon, though real-world throughput varies based on specific model quantization, thermal constraints, and system load.

Can I configure llama.cpp to use the same bandwidth estimates as MLX in LLMFIT?

No. The bandwidth values are intrinsic to each runtime’s architectural implementation within the gpu_bandwidth_roofline logic. Llama.cpp’s reliance on the GGUF format and generic GPU backend prevents it from exposing the same Metal driver capabilities that enable MLX’s higher bandwidth utilization, regardless of configuration changes.

Where does the runtime selection logic reside in the codebase?

The InferenceRuntime enum, which includes both Mlx and LlamaCpp variants alongside their platform constraints, is defined in llmfit-core/src/providers.rs. This enum drives the conditional estimation logic in llmfit-core/src/fit.rs and the provider documentation in docs/providers.md that specifies MLX as an Apple-Silicon-exclusive runtime.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}