# How llmfit's Memory-Bandwidth TPS Estimation Model Works: A Technical Deep Dive

> Explore llmfit's memory-bandwidth TPS estimation model. Learn how it treats inference as a memory-bound problem to calculate token throughput by dividing bandwidth by per-token memory traffic.

- Repository: [Alex Jones/llmfit](https://github.com/AlexsJones/llmfit)
- Tags: deep-dive
- Published: 2026-08-20

---

**llmfit estimates token throughput (TPS) by treating inference as a memory-bandwidth-bound problem, dividing available memory bandwidth by the per-token memory traffic required to read model weights.**

The `llmfit` project's core purpose is to determine if a given LLM can run efficiently on your hardware. Its TPS estimator assumes that modern transformers are bottlenecked not by compute, but by how fast weights can stream from memory to compute units. This article breaks down exactly how the Rust implementation models this relationship across GPUs, CPUs, and MoE architectures.

## The Core Assumption: Memory-Bandwidth-Bound Inference

llmfit's model rests on a single simplifying assumption: **each generated token requires reading the entire model weight matrix once**. This holds true for autoregressive decoding, where matrix multiplications process one token at a time while the full parameter set remains resident (or streamed) in memory.

This assumption means TPS depends entirely on two variables:

- **Available memory bandwidth** (GB/s) — how fast weights can move
- **Model size in gigabytes** — how much weight data must move per token

The estimator ignores FLOPS, KV-cache overhead, and attention compute—trading precision for portability across diverse hardware.

## Stage 1: Measuring or Lookup of Memory Bandwidth

The estimator first determines the effective memory bandwidth for the target device. In [`llmfit-core/src/hardware.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/hardware.rs), two distinct paths handle this:

### GPU Bandwidth Lookup

For GPUs, `gpu_memory_bandwidth_gbps(name)` (lines 2630–2680) maps device names to hard-coded bandwidth values:

```rust
// From hardware.rs - simplified conceptual structure
pub fn gpu_memory_bandwidth_gbps(name: &str) -> Option<f64> {
    match name {
        "NVIDIA A100 80GB" => Some(2039.0),  // HBM2e
        "NVIDIA H100 80GB" => Some(3350.0),  // HBM3
        "AMD MI300X" => Some(5300.0),        // HBM3
        // ... extensive table ...
        _ => None, // Unknown GPU triggers fallback
    }
}

```

When the GPU model is unrecognized, the function returns `None`. The planner then applies a **conservative constant of ~200 GB/s** for CUDA/ROCm devices rather than failing entirely.

### CPU RAM Bandwidth Measurement

For CPU inference, `measured_ram_bandwidth_gbps()` runs a one-time multithreaded `memcpy` benchmark and caches the result:

```rust
// Conceptual usage pattern from hardware.rs
let ram_bw = hardware::measured_ram_bandwidth_gbps(); // Runs once, then cached

```

This empirical approach accounts for actual memory controller performance, NUMA topology, and system load—factors a static table cannot capture.

## Stage 2: Calculating Per-Token Memory Traffic

With bandwidth established, the estimator computes how much data moves per token. In [`llmfit-core/src/models.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/models.rs), model size derivation handles both standard dense transformers and MoE architectures.

### Standard Model Sizing

For dense models, lines 65–75 apply quantization-aware size calculation:

```rust
// From models.rs - Q4_K_M default sizing
let model_gb = self.params as f64 * 0.5 / 1024.0_f64.powi(3);

```

The `0.5` bytes/parameter factor corresponds to **Q4_K_M quantization** (4-bit weights with K-quantization). Other quant levels adjust this multiplier accordingly.

### MoE Bandwidth Decomposition

Mixture-of-Experts models complicate the model because not all parameters are active per token. The `moe_bandwidth_decomposition()` function (lines 816–835) splits traffic into:

| Component | Source | Bandwidth Pool |
|-----------|--------|---------------|
| Active expert weights | GPU HBM | GPU bandwidth |
| Inactive expert weights | System RAM | DDR bandwidth |

This decomposition enables realistic TPS estimates when experts are offloaded to CPU memory—accurately modeling that only the active expert benefits from GPU speed.

## Stage 3: Deriving TPS from the Bandwidth-to-Size Ratio

The final calculation occurs in [`llmfit-core/src/plan.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/plan.rs). The `estimate_model_plan_with_config()` function (lines 219–236) delegates to `estimate_tps_from_bandwidth()` (~lines 350–380), which applies the fundamental formula:

```rust
// From plan.rs - core estimation logic
let tps = bandwidth_gbps / model_gb;

```

### GPU Path

When `gpu_memory_bandwidth_gbps()` returns `Some(bw)`, that bandwidth applies directly:

```rust
if let Some(gpu_bw) = system.gpu_bandwidth_gbps {
    let tps = gpu_bw / model.model_size_gb();
    // ...
}

```

### CPU / Fallback Path

For CPU-only inference or unknown GPUs, the measured or default RAM bandwidth substitutes:

```rust
let effective_bw = system
    .gpu_bandwidth_gbps
    .unwrap_or(200.0); // Conservative CUDA/ROCm default

// Or for explicit CPU inference:
let ram_bw = system.measured_ram_bandwidth_gbps.unwrap_or(30.0);

```

### MoE-Specific Calculation

When MoE offloading is configured, the estimator computes **two throughput limits** and takes the minimum:

1. **GPU-bound TPS**: `gpu_bandwidth / active_expert_gb`
2. **DDR-bound TPS**: `ram_bandwidth / inactive_experts_gb`

The result reflects the actual bottleneck—often PCIe or system memory when expert counts grow large.

## Output and Consumption

The calculated `estimated_tps` field populates the `FitResult` struct in [`llmfit-core/src/fit.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/fit.rs) (lines 115–130):

```rust
pub struct FitResult {
    pub model: Model,
    pub memory_required_gb: f64,
    pub estimated_tps: f64,  // Core output of the bandwidth model
    pub fits_in_memory: bool,
    // ...
}

```

Downstream TUI layers in [`llmfit-tui/src/display.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-tui/src/display.rs) render this for human review, while the `--json` CLI flag emits it for programmatic consumption.

## Working with the Estimator: Code Examples

### CLI Invocation

```bash

# Fit a model and examine TPS estimate

cargo run -- fit qwen2.5-32b --quant Q4_K_M --json

# Sample output excerpt

{
  "model": "qwen2.5-32b",
  "quant": "Q4_K_M",
  "estimated_tps": 29.7,
  "memory_required_gb": 15.2,
  "fits_in_memory": true
}

```

### Programmatic API

```rust
use llmfit_core::{hardware, models, plan};

// Detect system capabilities
let system = hardware::SystemSpecs::detect().unwrap();

// Configure model with quantization
let model = models::Model::from_hf_id("qwen2.5-32b")?
    .with_quant("Q4_K_M");

// Generate execution plan with TPS estimate
let plan = plan::estimate_model_plan_with_config(
    &system,
    &model,
    4096,        // context window
    None,        // no target TPS constraint
)?;

println!("Estimated TPS: {:.2}", plan.estimated_tps);
// Output: Estimated TPS: 29.70

```

## Key Implementation Files

| File | Purpose |
|------|---------|
| [`llmfit-core/src/hardware.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/hardware.rs) | GPU bandwidth lookup table; RAM bandwidth measurement |
| [`llmfit-core/src/models.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/models.rs) | Model size calculation; MoE traffic decomposition |
| [`llmfit-core/src/plan.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/plan.rs) | Core TPS estimation: bandwidth ÷ size |
| [`llmfit-core/src/fit.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/fit.rs) | `FitResult` struct holding `estimated_tps` |
| [`llmfit-tui/src/display.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-tui/src/display.rs) | Human-readable TPS rendering |

## Summary

- **llmfit's TPS model assumes memory-bandwidth-bound inference**: each token requires one full read of model weights
- **Bandwidth discovery is hardware-specific**: GPU lookups use hard-coded tables; CPUs use measured benchmarks
- **Model sizing incorporates quantization**: Q4_K_M defaults to 0.5 bytes per parameter
- **MoE architectures are decomposed**: active vs. inactive experts may use different memory pools
- **Final TPS equals bandwidth divided by size**: simple, fast, and surprisingly accurate for throughput estimation

## Frequently Asked Questions

### What does TPS mean in llmfit's context?

**TPS stands for tokens per second**—the rate at which an LLM can generate new tokens during inference. LlMFit estimates this throughput based on how quickly your hardware can stream model weights from memory, rather than raw compute capability.

### Why does llmfit ignore FLOPS in its TPS calculation?

**Modern LLM inference is memory-bound, not compute-bound.** During autoregressive generation, matrix multiplications are so large that GPUs spend most time waiting for weight data to arrive. By modeling only memory bandwidth, llmfit achieves accurate estimates without per-GPU FLOPS tables that would complicate maintenance.

### How accurate is the TPS estimate for MoE models?

**Accuracy depends on expert loading patterns.** The estimator assumes perfect prefetching of the active expert and sequential streaming of inactive ones. Real-world performance varies with batch size, context length, and implementation details of the inference engine (vLLM, llama.cpp, etc.). The conservative minimum() approach generally underestimates rather than overestimates throughput.

### Can I override the bandwidth values llmfit uses?

**Not through public API, but the source is modifiable.** The `gpu_memory_bandwidth_gbps` table in [`hardware.rs`](https://github.com/AlexsJones/llmfit/blob/main/hardware.rs) can be extended, or you can patch `measured_ram_bandwidth_gbps` to inject custom values. For programmatic use, wrapping `SystemSpecs` to override `gpu_bandwidth_gbps` before calling `estimate_model_plan_with_config()` achieves the same effect.