How LLMFIT_DDR_BANDWIDTH Configures MoE Expert Streaming Estimates in llmfit

The LLMFIT_DDR_BANDWIDTH environment variable overrides system RAM bandwidth calculations to control throughput predictions for Mixture-of-Experts (MoE) off-load inference in the llmfit engine.

When running large MoE models with limited GPU VRAM, the llmfit tool streams inactive expert weights from system RAM. The LLMFIT_DDR_BANDWIDTH setting directly determines how the AlexsJones/llmfit repository calculates tokens-per-second (TPS) estimates for these streaming workloads, ensuring predictions match your actual hardware capabilities.

The Role of RAM Bandwidth in MoE Off-load

Mixture-of-Experts architectures place unique demands on system memory. In the MoE off-load path implemented in llmfit-core/src/fit.rs, only active experts reside in GPU VRAM while inactive weights remain in system RAM and stream on demand.

This creates a memory-bound bottleneck where inference speed depends entirely on DDR throughput. The estimation engine calculates expert loading time using the formula:

// expert read time = active_gb / ddr_bandwidth_gbps
let expert_read_time = active_gb / ddr_bandwidth_gbps(config);

This calculation occurs specifically within RunMode::MoeOffload logic (lines 31-41 of fit.rs), making the bandwidth value critical for accurate TPS forecasting on MoE models like Mixtral 8x7b.

Configuration Priority Hierarchy

The ddr_bandwidth_gbps function in llmfit-core/src/fit.rs resolves the effective bandwidth through a strict priority chain. The system checks these sources in order:

1. TUI Advanced Config – The CalcConfig::ddr_bandwidth_gbps field set via the graphical interface takes highest precedence.

2. LLMFIT_DDR_BANDWIDTH Environment Variable – If present, the raw numeric value (in GB/s) overrides hardware detection. The parser validates positive floating-point values:

if let Some(bw) = std::env::var("LLMFIT_DDR_BANDWIDTH")
    .ok()
    .and_then(|v| v.parse::<f64>().ok())
    .filter(|b| *b > 0.0) { … }

This logic appears at lines 1158-1165 of fit.rs.

3. Hardware Measurement – When no override exists, hardware::measured_ram_bandwidth_gbps() probes the actual system capabilities.

4. Conservative Fallback – If measurement fails, the engine defaults to 50 GB/s, representing typical DDR4-3200 dual-channel bandwidth.

Practical Usage Examples

Override Bandwidth via Environment Variable

Set the variable before running the fit command to simulate or enforce specific RAM constraints:

# Force 90 GB/s DDR bandwidth estimate

export LLMFIT_DDR_BANDWIDTH=90
llmfit fit --model mixtral-8x7b

The TPS calculation will use 90 GB/s instead of measured or fallback values, reflecting high-end DDR5 or quad-channel configurations.

Programmatic Access in Rust

For custom tooling that replicates llmfit's logic:

use std::env;

fn get_ddr_bw() -> f64 {
    env::var("LLMFIT_DDR_BANDWIDTH")
        .ok()
        .and_then(|v| v.parse::<f64>().ok())
        .filter(|b| *b > 0.0)
        .unwrap_or(50.0) // matches llmfit fallback
}
println!("Using DDR bandwidth: {} GB/s", get_ddr_bw());

TUI Advanced Configuration

The graphical interface overrides both environment variables and hardware detection through CalcConfig:

let cfg = CalcConfig {
    ddr_bandwidth_gbps: Some(120.0), // 120 GB/s from UI
    ..Default::default()
};
let fit = ModelFit::analyze_with_config(&model, &system, cfg);
println!("Estimated TPS (MoE offload): {}", fit.estimated_tps);

This pattern appears in llmfit-tui/src/tui_ui.rs where the Advanced Configuration panel exposes direct bandwidth control.

Implementation Reference

The bandwidth override mechanism spans three core components:

• llmfit-core/src/fit.rs – Contains the ddr_bandwidth_gbps resolution logic and MoE-offload TPS estimation. The environment variable parsing occurs at lines 1158-1165.

• llmfit-core/src/hardware.rs – Provides the measured_ram_bandwidth_gbps fallback function when LLMFIT_DDR_BANDWIDTH remains unset.

• llmfit-tui/src/tui_ui.rs – Implements the TUI Advanced Configuration panel that writes to CalcConfig::ddr_bandwidth_gbps, superseding environment variables.

Additional architectural context appears in docs/how-it-works.md, which documents the MoE streaming estimation pipeline.

Summary

• LLMFIT_DDR_BANDWIDTH overrides system RAM speed detection for MoE throughput estimates, accepting values in GB/s.
• The variable sits at priority level two in the bandwidth resolution chain, below TUI config but above hardware measurement.
• It directly determines expert_read_time calculations when RunMode::MoeOffload is active, affecting tokens-per-second predictions.
• If omitted, llmfit either measures actual RAM bandwidth or falls back to 50 GB/s (DDR4-3200 dual-channel).
• Setting this variable is essential when targeting high-throughput DDR5 systems or when hardware detection produces inaccurate readings.

Frequently Asked Questions

What happens if I don't set LLMFIT_DDR_BANDWIDTH?

If the environment variable is unset and no TUI configuration exists, llmfit calls hardware::measured_ram_bandwidth_gbps() to probe your system's actual memory throughput. If measurement fails or returns invalid data, the engine conservatively assumes 50 GB/s, matching standard DDR4-3200 dual-channel performance. This ensures estimates remain realistic on typical consumer hardware.

Can I use this variable for non-MoE models?

No. The bandwidth override only affects the estimation path when RunMode::MoeOffload is active and the model architecture is detected as a Mixture-of-Experts. Dense models that load entirely into VRAM do not stream weights from system RAM, so LLMFIT_DDR_BANDWIDTH has no impact on their TPS calculations.

How does the TUI Advanced Config interact with the environment variable?

The TUI configuration takes absolute precedence. When CalcConfig::ddr_bandwidth_gbps contains Some(value), the ddr_bandwidth_gbps function in fit.rs returns that value immediately without checking environment variables or hardware measurements. To use LLMFIT_DDR_BANDWIDTH, ensure the TUI field is cleared or set to None.

What units should I use for LLMFIT_DDR_BANDWIDTH?

Specify the value in gigabytes per second (GB/s) as a floating-point number. For example, use 90 for 90 GB/s or 120.5 for 120.5 GB/s. The parser rejects non-numeric strings and values less than or equal to zero, falling back to hardware detection or the 50 GB/s default if validation fails.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}