# How llmfit Estimates LLM Inference Speed: A Memory-Bandwidth Roofline Model

> Discover how llmfit estimates LLM inference speed using a memory-bandwidth roofline model. Learn its prediction methodology for tokens-per-second throughput.

- Repository: [Alex Jones/llmfit](https://github.com/AlexsJones/llmfit)
- Tags: deep-dive
- Published: 2026-08-19

---

**llmfit estimates LLM inference speed using a memory-bandwidth roofline model that divides available hardware memory bandwidth by the quantized model size, then applies efficiency and run-mode correction factors to predict tokens-per-second throughput.**

llmfit, an open-source deployment planning tool for large language models, calculates inference performance using a deterministic hardware-aware formula rather than empirical benchmarking. The estimation engine resides in [`llmfit-core/src/fit.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/fit.rs), where it combines system specifications from [`hardware.rs`](https://github.com/AlexsJones/llmfit/blob/main/hardware.rs) with quantization metadata from [`models.rs`](https://github.com/AlexsJones/llmfit/blob/main/models.rs) to generate reproducible speed predictions. This approach assumes that modern transformer inference is memory-bound—limited by how quickly weights can be read from VRAM or system RAM rather than compute throughput.

## The Roofline Foundation: Memory Bandwidth as the Bottleneck

The estimator operates on the principle that **token generation is memory-bandwidth bound**. During autoregressive decoding, each forward pass must load the entire model weight matrix from memory to compute a single next token. Consequently, the theoretical maximum throughput depends on how fast the hardware can stream weights divided by the total bytes required per parameter.

According to the AlexsJones/llmfit source code, the core formula is implemented in the `estimate_tps` function:

```

raw_tps = memory_bandwidth_GB/s / model_size_GB

```

Where `memory_bandwidth_GB/s` is retrieved from [`hardware.rs`](https://github.com/AlexsJones/llmfit/blob/main/hardware.rs) lookup tables (`gpu_memory_bandwidth_gbps` or `ddr_bandwidth_gbps`), and `model_size_GB` is calculated from the model's parameter count multiplied by `models::quant_bytes_per_param` for the selected quantization scheme.

## Step-by-Step Speed Calculation in [`fit.rs`](https://github.com/AlexsJones/llmfit/blob/main/fit.rs)

The `ModelFit::analyze_inner` method orchestrates the estimation pipeline, progressing from hardware detection to final throughput prediction.

### 1. Execution Path and Quantization Selection

First, `analyze_inner` determines the optimal **run-mode** (GPU, MoE-offload, CPU-only, or Tensor-Parallel) based on available VRAM. It simultaneously selects the **best quantization** that fits within the detected memory constraints. These decisions feed directly into the speed calculator:

- **GPU mode**: Uses VRAM bandwidth
- **MoE-offload**: Uses system DDR bandwidth for expert loading
- **CPU-only**: Uses DDR bandwidth with distinct scaling factors

### 2. Raw Throughput Calculation

The `estimate_tps` function (lines 1081-1128 in [`llmfit-core/src/fit.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/fit.rs)) computes the baseline tokens-per-second:

```rust
// Pseudocode based on fit.rs implementation
let raw_tps = hardware_bandwidth / model_size_gb;
let estimated = raw_tps * efficiency * run_mode_factor;

```

The `model_size_gb` value accounts for the specific bytes-per-parameter of the chosen quantization format (e.g., Q4_K_M = 4.5 bits/parameter, FP16 = 2 bytes/parameter).

### 3. Efficiency and Run-Mode Adjustments

The raw throughput undergoes two corrective multiplications defined in `CalcConfig`:

- **Efficiency factor**: Default `0.55` to account for kernel launch overhead, KV-cache reads, and PCIe transfers (`CalcConfig::efficiency`)
- **Run-mode multiplier**: Hardware-specific scaling from `CalcConfig::run_mode_factors`:
  - GPU: `1.0`
  - Tensor-Parallel: `0.9`
  - MoE-offload: `0.8`

### 4. MoE-Specific Bandwidth Handling

For Mixture-of-Experts (MoE) models, `estimate_tps` implements specialized logic (lines 1231-1249) to handle expert loading:

- **MoE-offload mode**: Only active experts are read from system RAM, so the calculation uses `ddr_bandwidth_gbps` with the smaller active-expert memory footprint
- **GPU mode**: Conservatively assumes the **full model size** must be read from VRAM, as most inference runtimes load all layers uniformly regardless of sparsity

## Using the Estimator: Code Examples

Access the estimator programmatically through the `llmfit-core` library:

```rust
// Using the library directly
use llmfit_core::{hardware::SystemSpecs, models::LlmModel, fit::ModelFit};

let system = SystemSpecs::detect();          // auto-detect CPU/GPU specs
let model = LlmModel::from_hf_repo("meta-llama/Llama-2-7b-chat"); // load metadata

let fit = ModelFit::analyze(&model, &system);
println!("Estimated throughput: {:.1} tok/s", fit.estimated_tps);
println!("Run mode: {}", fit.run_mode_text());
println!("Best quantization: {}", fit.best_quant);

```

Or invoke via the CLI to see the estimation results in the TUI table:

```bash

# CLI usage

$ llmfit fit --model meta-llama/Llama-2-7b-chat

# Output excerpt

│ Model                     │ 7B │ 8k → 8k │ GPU │ 45.2 tok/s │ Perfect │

```

The computed TPS is stored in `ModelFit.estimated_tps` and rendered through [`llmfit-tui/src/display.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-tui/src/display.rs) for visual feedback.

## Configuring the Estimate via `CalcConfig`

Users can tune the estimation model through the `CalcConfig` struct to match specific hardware behaviors:

- Adjust `efficiency` (0.0–1.0) for custom kernel overhead profiles
- Modify `run_mode_factors` to account for specific multi-GPU topologies
- Override bandwidth values in [`hardware.rs`](https://github.com/AlexsJones/llmfit/blob/main/hardware.rs) for unreleased or proprietary hardware

## Summary

- llmfit uses a **memory-bandwidth roofline model** where speed equals bandwidth divided by model size
- The `estimate_tps` function in [`llmfit-core/src/fit.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/fit.rs) implements the core calculation at lines 1081-1128
- **Raw throughput** is scaled by an efficiency factor (default 0.55) and run-mode multipliers (GPU=1.0, Tensor-Parallel=0.9, MoE-offload=0.8)
- **MoE models** trigger bandwidth-specific logic that distinguishes between DDR-limited offloading and VRAM-limited GPU execution
- Results are exposed via `ModelFit.estimated_tps` and the `EstimateBasis` struct for transparency

## Frequently Asked Questions

### What hardware specifications does llmfit detect for speed estimation?

llmfit queries the system via `SystemSpecs::detect()` in [`hardware.rs`](https://github.com/AlexsJones/llmfit/blob/main/hardware.rs) to determine GPU model, VRAM capacity, and CPU memory bandwidth. These values map to the `gpu_memory_bandwidth_gbps` lookup table to determine the memory ceiling used in the roofline formula.

### How does quantization affect the inference speed estimate?

Quantization directly reduces `model_size_GB` by lowering bytes-per-parameter (e.g., Q4_K_M uses ~4.5 bits vs FP16's 16 bits). The `models::quant_bytes_per_param` function translates quantization schemes into memory footprints, allowing `estimate_tps` to calculate higher throughput for more aggressive compression formats.

### Why does MoE GPU mode assume the full model size instead of active experts?

The implementation conservatively assumes full model loading (lines 1231-1249) because most production inference engines—including vLLM and TensorRT-LLM—load all expert layers into VRAM during model initialization to avoid runtime loading latency. While only a subset of experts activate per forward pass, the memory bandwidth cost reflects the total resident model size.

### Can I adjust the efficiency factor for my specific hardware setup?

Yes. The `CalcConfig` struct exposes the `efficiency` field (default 0.55) which you can modify when calling `ModelFit::analyze_with_config()`. Increase the value for optimized kernels or high-bandwidth memory (HBM) architectures; decrease it for systems with significant PCIe bottlenecks or thermal throttling.