# llmfit Run Modes Explained: GPU, MoE Offload, CPU Offload, CPU Only, and Tensor Parallel

> Explore llmfit run modes: GPU, MoE Offload, CPU Offload, CPU Only, and Tensor Parallel. Optimize LLM execution across your hardware for maximum performance and efficiency.

- Repository: [Alex Jones/llmfit](https://github.com/AlexsJones/llmfit)
- Tags: deep-dive
- Published: 2026-08-20

---

**llmfit uses five distinct run modes to determine how a large language model executes on your hardware, ranging from full GPU acceleration to distributed cluster inference.**

The `RunMode` enum in [`llmfit-core/src/fit.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/fit.rs) defines how the library balances memory constraints against inference speed. Understanding these modes helps you interpret fit reports and optimize deployment for your specific hardware configuration.

---

## RunMode Enum Definition

The core enumeration appears at [lines 88-94](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/fit.rs#L88-L94) of [`llmfit-core/src/fit.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/fit.rs):

```rust
pub enum RunMode {
    Gpu,            // Fully loaded into VRAM – fast
    MoeOffload,     // MoE: active experts in VRAM, inactive offloaded to RAM
    CpuOffload,     // Partial GPU offload, spills to system RAM – mixed
    CpuOnly,        // Entirely in system RAM, no GPU – slow
    TensorParallel, // Distributed via NCCL across cluster nodes
}

```

Each variant represents a fundamentally different memory placement strategy with direct performance implications.

---

## How llmfit Selects a Run Mode

The selection logic executes inside `ModelFit::analyze_inner` following a priority-based decision tree:

1. **GPU availability check** – If `system.has_gpu` is true, the analyzer first attempts `RunMode::Gpu`
2. **MoE model validation** – For Mixture-of-Experts models, it checks if active experts fit in VRAM before considering `MoeOffload`
3. **VRAM sufficiency test** – When the full model exceeds GPU memory, it falls back to `CpuOffload`
4. **No-GPU fallback** – Absent discrete GPU detection, `CpuOnly` is selected
5. **Cluster override** – `system.cluster_mode` forces `TensorParallel` regardless of single-node capacity

The MoE-specific offloading path is implemented in [`moe_offload_path`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/fit.rs#L823-L845), which calculates whether active experts can reside in VRAM while inactive experts spill to system RAM.

---

## Detailed Comparison of llmfit Run Modes

### Gpu Mode

**Memory placement:** Entire model weights reside in GPU VRAM

**Performance characteristics:** Fastest inference, bandwidth-bound by GPU memory throughput

**Activation condition:** Sufficient VRAM for complete model (or MoE active expert subset)

In `score_fit`, only GPU-based paths can achieve `FitLevel::Perfect`. The UI displays this mode as `"GPU"` via `run_mode_text`.

### MoeOffload Mode

**Memory placement:** Active experts in VRAM, inactive experts in system RAM

**Performance characteristics:** Good throughput for very large MoE models; slower than pure GPU due to expert-swapping overhead, but superior to full CPU spill

**Activation condition:** MoE architecture detected where full model exceeds VRAM but active expert subset fits

The distinction between active and inactive experts comes from model fields `is_moe` and `active_experts` defined in [`llmfit-core/src/models.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/models.rs).

### CpuOffload Mode

**Memory placement:** Model partially in VRAM, with automatic spilling to system RAM

**Performance characteristics:** Mixed speed—GPU compute cores remain utilized but PCIe transfers create bottlenecks

**Activation condition:** Model slightly exceeds available VRAM; CPU+GPU hybrid execution still beneficial

The UI labels this `"CPU+GPU"` to indicate the hybrid nature.

### CpuOnly Mode

**Memory placement:** Entirely in system RAM, zero GPU involvement

**Performance characteristics:** Slowest inference path; CPU-optimized matrix operations without tensor core acceleration

**Activation condition:** No discrete GPU detected, or explicit override on unified-memory systems

Displays as `"CPU"` in fit reports.

### TensorParallel Mode

**Memory placement:** Model weights sharded across multiple GPUs/nodes via NCCL

**Performance characteristics:** Scales linearly with cluster size; network latency and synchronization overhead reduce efficiency at small scales

**Activation condition:** `system.cluster_mode` enabled with vLLM backend

The UI abbreviates this mode as `"TP"` for Tensor Parallel.

---

## Display and Scoring Implementation

Human-readable labels map to enum variants in [`run_mode_text`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/fit.rs#L2929-L2936):

```rust
pub fn run_mode_text(&self) -> &str {
    match self.run_mode {
        RunMode::Gpu => "GPU",
        RunMode::TensorParallel => "TP",
        RunMode::MoeOffload => "MoE",
        RunMode::CpuOffload => "CPU+GPU",
        RunMode::CpuOnly => "CPU",
    }
}

```

The `score_fit` function applies different quality thresholds per mode—GPU paths alone qualify for perfect fits, while offloaded modes receive degraded scores proportional to their memory-transfer penalties.

---

## Inspecting Run Modes Programmatically

Retrieve the selected mode after hardware analysis:

```rust
use llmfit_core::fit::ModelFit;
use llmfit_core::hardware::SystemSpecs;
use llmfit_core::models::LlmModel;

fn inspect_run_mode() {
    let system = SystemSpecs::detect().unwrap();
    let model = LlmModel::from_name("Llama-2-7B-chat-hf").unwrap();

    let fit = ModelFit::analyze(&model, &system);

    println!("Run mode: {}", fit.run_mode_text());
    println!("Fit level: {}", fit.fit_text());
    println!("Estimated throughput: {:.1} tok/s", fit.estimated_tps);
}

```

Force a specific runtime to observe mode changes:

```rust
let forced_runtime = Some(llmfit_core::fit::InferenceRuntime::LlamaCpp);
let fit = ModelFit::analyze_with_forced_runtime(&model, &system, None, forced_runtime);
println!("With forced LlamaCpp runtime → {}", fit.run_mode_text());

```

---

## Key Source Files

| File | Purpose |
|------|---------|
| [`llmfit-core/src/fit.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/fit.rs) | `RunMode` definition, selection logic, `moe_offload_path`, `run_mode_text`, `score_fit` |
| [`llmfit-core/src/hardware.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/hardware.rs) | GPU detection, VRAM querying, unified memory identification |
| [`llmfit-core/src/models.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/models.rs) | MoE fields (`is_moe`, `active_experts`, `num_experts`) |
| [`llmfit-tui/src/display.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-tui/src/display.rs) | Terminal rendering of run-mode labels |

---

## Summary

- **Gpu**: Full VRAM residency, fastest inference, qualifies for perfect fit scores
- **MoeOffload**: Expert-selective offloading for large MoE models that don't fit entirely in GPU memory
- **CpuOffload**: Hybrid execution when models slightly exceed VRAM capacity
- **CpuOnly**: RAM-only fallback when no discrete GPU is available
- **TensorParallel**: Distributed execution across cluster nodes via vLLM and NCCL

The llmfit run mode system automatically negotiates between hardware constraints and model architecture to select the highest-performance viable execution path.

---

## Frequently Asked Questions

### What triggers TensorParallel mode instead of single-GPU execution?

TensorParallel activates when `system.cluster_mode` is enabled in your configuration, forcing distributed inference via vLLM across multiple nodes regardless of whether the model would fit on a single GPU. This overrides the standard single-node decision tree entirely.

### Why does my MoE model use CpuOffload instead of MoeOffload?

MoeOffload requires that the **active expert subset** fits in available VRAM. If your system cannot accommodate even the active experts, the analyzer skips to CpuOffload or CpuOnly. Check your model's `active_experts` count against `system.vram_bytes` in the fit analysis.

### Can I force a specific run mode manually?

No direct run-mode forcing exists in the public API. You influence selection indirectly through: forcing a runtime backend (`analyze_with_forced_runtime`), modifying cluster mode settings, or using quantization parameters that reduce memory requirements. The library prioritizes automatic optimal selection over manual override.

### How does CpuOffload performance compare to Gpu mode?

CpuOffload maintains GPU compute utilization but incurs PCIe transfer overhead for memory spills. Throughput typically degrades 30-70% depending on activation sparsity and PCIe bandwidth. The `score_fit` function reflects this with reduced fit levels—Gpu alone achieves `FitLevel::Perfect`.