How llmfit Estimates LLM Inference Speed: A Memory-Bandwidth Roofline Model

llmfit estimates LLM inference speed using a memory-bandwidth roofline model that divides available hardware memory bandwidth by the quantized model size, then applies efficiency and run-mode correction factors to predict tokens-per-second throughput.

llmfit, an open-source deployment planning tool for large language models, calculates inference performance using a deterministic hardware-aware formula rather than empirical benchmarking. The estimation engine resides in llmfit-core/src/fit.rs, where it combines system specifications from hardware.rs with quantization metadata from models.rs to generate reproducible speed predictions. This approach assumes that modern transformer inference is memory-bound—limited by how quickly weights can be read from VRAM or system RAM rather than compute throughput.

The Roofline Foundation: Memory Bandwidth as the Bottleneck

The estimator operates on the principle that token generation is memory-bandwidth bound. During autoregressive decoding, each forward pass must load the entire model weight matrix from memory to compute a single next token. Consequently, the theoretical maximum throughput depends on how fast the hardware can stream weights divided by the total bytes required per parameter.

According to the AlexsJones/llmfit source code, the core formula is implemented in the estimate_tps function:

raw_tps = memory_bandwidth_GB/s / model_size_GB

Where memory_bandwidth_GB/s is retrieved from hardware.rs lookup tables (gpu_memory_bandwidth_gbps or ddr_bandwidth_gbps), and model_size_GB is calculated from the model's parameter count multiplied by models::quant_bytes_per_param for the selected quantization scheme.

Step-by-Step Speed Calculation in fit.rs

The ModelFit::analyze_inner method orchestrates the estimation pipeline, progressing from hardware detection to final throughput prediction.

1. Execution Path and Quantization Selection

First, analyze_inner determines the optimal run-mode (GPU, MoE-offload, CPU-only, or Tensor-Parallel) based on available VRAM. It simultaneously selects the best quantization that fits within the detected memory constraints. These decisions feed directly into the speed calculator:

• GPU mode: Uses VRAM bandwidth
• MoE-offload: Uses system DDR bandwidth for expert loading
• CPU-only: Uses DDR bandwidth with distinct scaling factors

2. Raw Throughput Calculation

The estimate_tps function (lines 1081-1128 in llmfit-core/src/fit.rs) computes the baseline tokens-per-second:

// Pseudocode based on fit.rs implementation
let raw_tps = hardware_bandwidth / model_size_gb;
let estimated = raw_tps * efficiency * run_mode_factor;

The model_size_gb value accounts for the specific bytes-per-parameter of the chosen quantization format (e.g., Q4_K_M = 4.5 bits/parameter, FP16 = 2 bytes/parameter).

3. Efficiency and Run-Mode Adjustments

The raw throughput undergoes two corrective multiplications defined in CalcConfig:

• Efficiency factor: Default 0.55 to account for kernel launch overhead, KV-cache reads, and PCIe transfers (CalcConfig::efficiency)
• Run-mode multiplier: Hardware-specific scaling from CalcConfig::run_mode_factors:
  • GPU: 1.0
  • Tensor-Parallel: 0.9
  • MoE-offload: 0.8

4. MoE-Specific Bandwidth Handling

For Mixture-of-Experts (MoE) models, estimate_tps implements specialized logic (lines 1231-1249) to handle expert loading:

• MoE-offload mode: Only active experts are read from system RAM, so the calculation uses ddr_bandwidth_gbps with the smaller active-expert memory footprint
• GPU mode: Conservatively assumes the full model size must be read from VRAM, as most inference runtimes load all layers uniformly regardless of sparsity

Using the Estimator: Code Examples

Access the estimator programmatically through the llmfit-core library:

// Using the library directly
use llmfit_core::{hardware::SystemSpecs, models::LlmModel, fit::ModelFit};

let system = SystemSpecs::detect();          // auto-detect CPU/GPU specs
let model = LlmModel::from_hf_repo("meta-llama/Llama-2-7b-chat"); // load metadata

let fit = ModelFit::analyze(&model, &system);
println!("Estimated throughput: {:.1} tok/s", fit.estimated_tps);
println!("Run mode: {}", fit.run_mode_text());
println!("Best quantization: {}", fit.best_quant);

Or invoke via the CLI to see the estimation results in the TUI table:

# CLI usage

$ llmfit fit --model meta-llama/Llama-2-7b-chat

# Output excerpt

│ Model                     │ 7B │ 8k → 8k │ GPU │ 45.2 tok/s │ Perfect │

The computed TPS is stored in ModelFit.estimated_tps and rendered through llmfit-tui/src/display.rs for visual feedback.

Configuring the Estimate via CalcConfig

Users can tune the estimation model through the CalcConfig struct to match specific hardware behaviors:

• Adjust efficiency (0.0–1.0) for custom kernel overhead profiles
• Modify run_mode_factors to account for specific multi-GPU topologies
• Override bandwidth values in hardware.rs for unreleased or proprietary hardware

Summary

• llmfit uses a memory-bandwidth roofline model where speed equals bandwidth divided by model size
• The estimate_tps function in llmfit-core/src/fit.rs implements the core calculation at lines 1081-1128
• Raw throughput is scaled by an efficiency factor (default 0.55) and run-mode multipliers (GPU=1.0, Tensor-Parallel=0.9, MoE-offload=0.8)
• MoE models trigger bandwidth-specific logic that distinguishes between DDR-limited offloading and VRAM-limited GPU execution
• Results are exposed via ModelFit.estimated_tps and the EstimateBasis struct for transparency

Frequently Asked Questions

What hardware specifications does llmfit detect for speed estimation?

llmfit queries the system via SystemSpecs::detect() in hardware.rs to determine GPU model, VRAM capacity, and CPU memory bandwidth. These values map to the gpu_memory_bandwidth_gbps lookup table to determine the memory ceiling used in the roofline formula.

How does quantization affect the inference speed estimate?

Quantization directly reduces model_size_GB by lowering bytes-per-parameter (e.g., Q4_K_M uses ~4.5 bits vs FP16's 16 bits). The models::quant_bytes_per_param function translates quantization schemes into memory footprints, allowing estimate_tps to calculate higher throughput for more aggressive compression formats.

Why does MoE GPU mode assume the full model size instead of active experts?

The implementation conservatively assumes full model loading (lines 1231-1249) because most production inference engines—including vLLM and TensorRT-LLM—load all expert layers into VRAM during model initialization to avoid runtime loading latency. While only a subset of experts activate per forward pass, the memory bandwidth cost reflects the total resident model size.

Can I adjust the efficiency factor for my specific hardware setup?

Yes. The CalcConfig struct exposes the efficiency field (default 0.55) which you can modify when calling ModelFit::analyze_with_config(). Increase the value for optimized kernels or high-bandwidth memory (HBM) architectures; decrease it for systems with significant PCIe bottlenecks or thermal throttling.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}