How llmfit's Memory-Bandwidth TPS Estimation Model Works: A Technical Deep Dive

llmfit estimates token throughput (TPS) by treating inference as a memory-bandwidth-bound problem, dividing available memory bandwidth by the per-token memory traffic required to read model weights.

The llmfit project's core purpose is to determine if a given LLM can run efficiently on your hardware. Its TPS estimator assumes that modern transformers are bottlenecked not by compute, but by how fast weights can stream from memory to compute units. This article breaks down exactly how the Rust implementation models this relationship across GPUs, CPUs, and MoE architectures.

The Core Assumption: Memory-Bandwidth-Bound Inference

llmfit's model rests on a single simplifying assumption: each generated token requires reading the entire model weight matrix once. This holds true for autoregressive decoding, where matrix multiplications process one token at a time while the full parameter set remains resident (or streamed) in memory.

This assumption means TPS depends entirely on two variables:

• Available memory bandwidth (GB/s) — how fast weights can move
• Model size in gigabytes — how much weight data must move per token

The estimator ignores FLOPS, KV-cache overhead, and attention compute—trading precision for portability across diverse hardware.

Stage 1: Measuring or Lookup of Memory Bandwidth

The estimator first determines the effective memory bandwidth for the target device. In llmfit-core/src/hardware.rs, two distinct paths handle this:

GPU Bandwidth Lookup

For GPUs, gpu_memory_bandwidth_gbps(name) (lines 2630–2680) maps device names to hard-coded bandwidth values:

// From hardware.rs - simplified conceptual structure
pub fn gpu_memory_bandwidth_gbps(name: &str) -> Option<f64> {
    match name {
        "NVIDIA A100 80GB" => Some(2039.0),  // HBM2e
        "NVIDIA H100 80GB" => Some(3350.0),  // HBM3
        "AMD MI300X" => Some(5300.0),        // HBM3
        // ... extensive table ...
        _ => None, // Unknown GPU triggers fallback
    }
}

When the GPU model is unrecognized, the function returns None. The planner then applies a conservative constant of ~200 GB/s for CUDA/ROCm devices rather than failing entirely.

CPU RAM Bandwidth Measurement

For CPU inference, measured_ram_bandwidth_gbps() runs a one-time multithreaded memcpy benchmark and caches the result:

// Conceptual usage pattern from hardware.rs
let ram_bw = hardware::measured_ram_bandwidth_gbps(); // Runs once, then cached

This empirical approach accounts for actual memory controller performance, NUMA topology, and system load—factors a static table cannot capture.

Stage 2: Calculating Per-Token Memory Traffic

With bandwidth established, the estimator computes how much data moves per token. In llmfit-core/src/models.rs, model size derivation handles both standard dense transformers and MoE architectures.

Standard Model Sizing

For dense models, lines 65–75 apply quantization-aware size calculation:

// From models.rs - Q4_K_M default sizing
let model_gb = self.params as f64 * 0.5 / 1024.0_f64.powi(3);

The 0.5 bytes/parameter factor corresponds to Q4_K_M quantization (4-bit weights with K-quantization). Other quant levels adjust this multiplier accordingly.

MoE Bandwidth Decomposition

Mixture-of-Experts models complicate the model because not all parameters are active per token. The moe_bandwidth_decomposition() function (lines 816–835) splits traffic into:

Component	Source	Bandwidth Pool
Active expert weights	GPU HBM	GPU bandwidth
Inactive expert weights	System RAM	DDR bandwidth

This decomposition enables realistic TPS estimates when experts are offloaded to CPU memory—accurately modeling that only the active expert benefits from GPU speed.

Stage 3: Deriving TPS from the Bandwidth-to-Size Ratio

The final calculation occurs in llmfit-core/src/plan.rs. The estimate_model_plan_with_config() function (lines 219–236) delegates to estimate_tps_from_bandwidth() (~lines 350–380), which applies the fundamental formula:

// From plan.rs - core estimation logic
let tps = bandwidth_gbps / model_gb;

GPU Path

When gpu_memory_bandwidth_gbps() returns Some(bw), that bandwidth applies directly:

if let Some(gpu_bw) = system.gpu_bandwidth_gbps {
    let tps = gpu_bw / model.model_size_gb();
    // ...
}

CPU / Fallback Path

For CPU-only inference or unknown GPUs, the measured or default RAM bandwidth substitutes:

let effective_bw = system
    .gpu_bandwidth_gbps
    .unwrap_or(200.0); // Conservative CUDA/ROCm default

// Or for explicit CPU inference:
let ram_bw = system.measured_ram_bandwidth_gbps.unwrap_or(30.0);

MoE-Specific Calculation

When MoE offloading is configured, the estimator computes two throughput limits and takes the minimum:

1. GPU-bound TPS: gpu_bandwidth / active_expert_gb
2. DDR-bound TPS: ram_bandwidth / inactive_experts_gb

The result reflects the actual bottleneck—often PCIe or system memory when expert counts grow large.

Output and Consumption

The calculated estimated_tps field populates the FitResult struct in llmfit-core/src/fit.rs (lines 115–130):

pub struct FitResult {
    pub model: Model,
    pub memory_required_gb: f64,
    pub estimated_tps: f64,  // Core output of the bandwidth model
    pub fits_in_memory: bool,
    // ...
}

Downstream TUI layers in llmfit-tui/src/display.rs render this for human review, while the --json CLI flag emits it for programmatic consumption.

Working with the Estimator: Code Examples

CLI Invocation

# Fit a model and examine TPS estimate

cargo run -- fit qwen2.5-32b --quant Q4_K_M --json

# Sample output excerpt

{
  "model": "qwen2.5-32b",
  "quant": "Q4_K_M",
  "estimated_tps": 29.7,
  "memory_required_gb": 15.2,
  "fits_in_memory": true
}

Programmatic API

use llmfit_core::{hardware, models, plan};

// Detect system capabilities
let system = hardware::SystemSpecs::detect().unwrap();

// Configure model with quantization
let model = models::Model::from_hf_id("qwen2.5-32b")?
    .with_quant("Q4_K_M");

// Generate execution plan with TPS estimate
let plan = plan::estimate_model_plan_with_config(
    &system,
    &model,
    4096,        // context window
    None,        // no target TPS constraint
)?;

println!("Estimated TPS: {:.2}", plan.estimated_tps);
// Output: Estimated TPS: 29.70

Key Implementation Files

File	Purpose
llmfit-core/src/hardware.rs	GPU bandwidth lookup table; RAM bandwidth measurement
llmfit-core/src/models.rs	Model size calculation; MoE traffic decomposition
llmfit-core/src/plan.rs	Core TPS estimation: bandwidth ÷ size
llmfit-core/src/fit.rs	FitResult struct holding estimated_tps
llmfit-tui/src/display.rs	Human-readable TPS rendering

Summary

• llmfit's TPS model assumes memory-bandwidth-bound inference: each token requires one full read of model weights
• Bandwidth discovery is hardware-specific: GPU lookups use hard-coded tables; CPUs use measured benchmarks
• Model sizing incorporates quantization: Q4_K_M defaults to 0.5 bytes per parameter
• MoE architectures are decomposed: active vs. inactive experts may use different memory pools
• Final TPS equals bandwidth divided by size: simple, fast, and surprisingly accurate for throughput estimation

Frequently Asked Questions

What does TPS mean in llmfit's context?

TPS stands for tokens per second—the rate at which an LLM can generate new tokens during inference. LlMFit estimates this throughput based on how quickly your hardware can stream model weights from memory, rather than raw compute capability.

Why does llmfit ignore FLOPS in its TPS calculation?

Modern LLM inference is memory-bound, not compute-bound. During autoregressive generation, matrix multiplications are so large that GPUs spend most time waiting for weight data to arrive. By modeling only memory bandwidth, llmfit achieves accurate estimates without per-GPU FLOPS tables that would complicate maintenance.

How accurate is the TPS estimate for MoE models?

Accuracy depends on expert loading patterns. The estimator assumes perfect prefetching of the active expert and sequential streaming of inactive ones. Real-world performance varies with batch size, context length, and implementation details of the inference engine (vLLM, llama.cpp, etc.). The conservative minimum() approach generally underestimates rather than overestimates throughput.

Can I override the bandwidth values llmfit uses?

Not through public API, but the source is modifiable. The gpu_memory_bandwidth_gbps table in hardware.rs can be extended, or you can patch measured_ram_bandwidth_gbps to inject custom values. For programmatic use, wrapping SystemSpecs to override gpu_bandwidth_gbps before calling estimate_model_plan_with_config() achieves the same effect.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}