llmfit Run Modes Explained: GPU, MoE Offload, CPU Offload, CPU Only, and Tensor Parallel

llmfit uses five distinct run modes to determine how a large language model executes on your hardware, ranging from full GPU acceleration to distributed cluster inference.

The RunMode enum in llmfit-core/src/fit.rs defines how the library balances memory constraints against inference speed. Understanding these modes helps you interpret fit reports and optimize deployment for your specific hardware configuration.

---

RunMode Enum Definition

The core enumeration appears at lines 88-94 of llmfit-core/src/fit.rs:

pub enum RunMode {
    Gpu,            // Fully loaded into VRAM – fast
    MoeOffload,     // MoE: active experts in VRAM, inactive offloaded to RAM
    CpuOffload,     // Partial GPU offload, spills to system RAM – mixed
    CpuOnly,        // Entirely in system RAM, no GPU – slow
    TensorParallel, // Distributed via NCCL across cluster nodes
}

Each variant represents a fundamentally different memory placement strategy with direct performance implications.

---

How llmfit Selects a Run Mode

The selection logic executes inside ModelFit::analyze_inner following a priority-based decision tree:

1. GPU availability check – If system.has_gpu is true, the analyzer first attempts RunMode::Gpu
2. MoE model validation – For Mixture-of-Experts models, it checks if active experts fit in VRAM before considering MoeOffload
3. VRAM sufficiency test – When the full model exceeds GPU memory, it falls back to CpuOffload
4. No-GPU fallback – Absent discrete GPU detection, CpuOnly is selected
5. Cluster override – system.cluster_mode forces TensorParallel regardless of single-node capacity

The MoE-specific offloading path is implemented in moe_offload_path, which calculates whether active experts can reside in VRAM while inactive experts spill to system RAM.

---

Detailed Comparison of llmfit Run Modes

Gpu Mode

Memory placement: Entire model weights reside in GPU VRAM

Performance characteristics: Fastest inference, bandwidth-bound by GPU memory throughput

Activation condition: Sufficient VRAM for complete model (or MoE active expert subset)

In score_fit, only GPU-based paths can achieve FitLevel::Perfect. The UI displays this mode as "GPU" via run_mode_text.

MoeOffload Mode

Memory placement: Active experts in VRAM, inactive experts in system RAM

Performance characteristics: Good throughput for very large MoE models; slower than pure GPU due to expert-swapping overhead, but superior to full CPU spill

Activation condition: MoE architecture detected where full model exceeds VRAM but active expert subset fits

The distinction between active and inactive experts comes from model fields is_moe and active_experts defined in llmfit-core/src/models.rs.

CpuOffload Mode

Memory placement: Model partially in VRAM, with automatic spilling to system RAM

Performance characteristics: Mixed speed—GPU compute cores remain utilized but PCIe transfers create bottlenecks

Activation condition: Model slightly exceeds available VRAM; CPU+GPU hybrid execution still beneficial

The UI labels this "CPU+GPU" to indicate the hybrid nature.

CpuOnly Mode

Memory placement: Entirely in system RAM, zero GPU involvement

Performance characteristics: Slowest inference path; CPU-optimized matrix operations without tensor core acceleration

Activation condition: No discrete GPU detected, or explicit override on unified-memory systems

Displays as "CPU" in fit reports.

TensorParallel Mode

Memory placement: Model weights sharded across multiple GPUs/nodes via NCCL

Performance characteristics: Scales linearly with cluster size; network latency and synchronization overhead reduce efficiency at small scales

Activation condition: system.cluster_mode enabled with vLLM backend

The UI abbreviates this mode as "TP" for Tensor Parallel.

---

Display and Scoring Implementation

Human-readable labels map to enum variants in run_mode_text:

pub fn run_mode_text(&self) -> &str {
    match self.run_mode {
        RunMode::Gpu => "GPU",
        RunMode::TensorParallel => "TP",
        RunMode::MoeOffload => "MoE",
        RunMode::CpuOffload => "CPU+GPU",
        RunMode::CpuOnly => "CPU",
    }
}

The score_fit function applies different quality thresholds per mode—GPU paths alone qualify for perfect fits, while offloaded modes receive degraded scores proportional to their memory-transfer penalties.

---

Inspecting Run Modes Programmatically

Retrieve the selected mode after hardware analysis:

use llmfit_core::fit::ModelFit;
use llmfit_core::hardware::SystemSpecs;
use llmfit_core::models::LlmModel;

fn inspect_run_mode() {
    let system = SystemSpecs::detect().unwrap();
    let model = LlmModel::from_name("Llama-2-7B-chat-hf").unwrap();

    let fit = ModelFit::analyze(&model, &system);

    println!("Run mode: {}", fit.run_mode_text());
    println!("Fit level: {}", fit.fit_text());
    println!("Estimated throughput: {:.1} tok/s", fit.estimated_tps);
}

Force a specific runtime to observe mode changes:

let forced_runtime = Some(llmfit_core::fit::InferenceRuntime::LlamaCpp);
let fit = ModelFit::analyze_with_forced_runtime(&model, &system, None, forced_runtime);
println!("With forced LlamaCpp runtime → {}", fit.run_mode_text());

---

Key Source Files

File	Purpose
llmfit-core/src/fit.rs	RunMode definition, selection logic, moe_offload_path, run_mode_text, score_fit
llmfit-core/src/hardware.rs	GPU detection, VRAM querying, unified memory identification
llmfit-core/src/models.rs	MoE fields (is_moe, active_experts, num_experts)
llmfit-tui/src/display.rs	Terminal rendering of run-mode labels

---

Summary

• Gpu: Full VRAM residency, fastest inference, qualifies for perfect fit scores
• MoeOffload: Expert-selective offloading for large MoE models that don't fit entirely in GPU memory
• CpuOffload: Hybrid execution when models slightly exceed VRAM capacity
• CpuOnly: RAM-only fallback when no discrete GPU is available
• TensorParallel: Distributed execution across cluster nodes via vLLM and NCCL

The llmfit run mode system automatically negotiates between hardware constraints and model architecture to select the highest-performance viable execution path.

---

Frequently Asked Questions

What triggers TensorParallel mode instead of single-GPU execution?

TensorParallel activates when system.cluster_mode is enabled in your configuration, forcing distributed inference via vLLM across multiple nodes regardless of whether the model would fit on a single GPU. This overrides the standard single-node decision tree entirely.

Why does my MoE model use CpuOffload instead of MoeOffload?

MoeOffload requires that the active expert subset fits in available VRAM. If your system cannot accommodate even the active experts, the analyzer skips to CpuOffload or CpuOnly. Check your model's active_experts count against system.vram_bytes in the fit analysis.

Can I force a specific run mode manually?

No direct run-mode forcing exists in the public API. You influence selection indirectly through: forcing a runtime backend (analyze_with_forced_runtime), modifying cluster mode settings, or using quantization parameters that reduce memory requirements. The library prioritizes automatic optimal selection over manual override.

How does CpuOffload performance compare to Gpu mode?

CpuOffload maintains GPU compute utilization but incurs PCIe transfer overhead for memory spills. Throughput typically degrades 30-70% depending on activation sparsity and PCIe bandwidth. The score_fit function reflects this with reduced fit levels—Gpu alone achieves FitLevel::Perfect.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}