# LLM Run Modes in llmfit: A Complete Guide to GPU, CPU, and Distributed Execution

> Explore llmfit's LLM run modes: Gpu, MoeOffload, CpuOffload, CpuOnly, and TensorParallel. Optimize inference by distributing model weights across GPU VRAM, RAM, and clusters for peak performance.

- Repository: [Alex Jones/llmfit](https://github.com/AlexsJones/llmfit)
- Tags: deep-dive
- Published: 2026-08-19

---

**llmfit defines five distinct LLM run modes—Gpu, MoeOffload, CpuOffload, CpuOnly, and TensorParallel—that determine exactly how model weights are distributed across GPU VRAM, system RAM, and multi-node clusters for optimal inference performance.**

The `llmfit` project by AlexsJones analyzes hardware compatibility for large language models, evaluating every feasible execution path through its core analysis engine. Understanding these **LLM run modes** enables developers to select the optimal execution strategy based on available hardware constraints and performance requirements.

## The RunMode Enum in llmfit-core

The execution strategy classification lives in [[`llmfit-core/src/fit.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/fit.rs)](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/fit.rs) at lines 88-94, where the `RunMode` enum is deliberately decoupled from the **FitLevel** (memory fit classification). This separation allows `llmfit` to evaluate both *how well* a model fits and *how* it will run independently.

The `RunMode` type is embedded within the [`ModelFit`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/fit.rs#L40-L44) struct (defined at lines 40-44), which tracks:
- `memory_required_gb`: Total memory needed for the selected execution path
- `memory_available_gb`: Host memory capacity
- `moe_offloaded_gb`: Specific to MoE models, tracking offloaded expert weights
- `estimated_tps`: Predicted tokens-per-second performance

## The Five LLM Run Modes Explained

### Gpu Mode

**Gpu** mode loads all model weights directly into GPU VRAM, providing the fastest inference path available. This mode requires sufficient VRAM capacity for the entire model and is the default target when hardware resources permit full acceleration.

### MoeOffload Mode

**MoeOffload** (Mixture-of-Experts offloading) is specifically designed for MoE architectures. In this mode, `llmfit` keeps active experts resident in VRAM while offloading inactive experts to system RAM. This hybrid approach balances the massive parameter counts of MoE models against limited GPU memory.

### CpuOffload Mode

**CpuOffload** creates a hybrid execution environment where the model is partially resident on the GPU with the remainder spilling into system RAM. This mode activates when model size exceeds available VRAM but the system still benefits from partial GPU acceleration.

### CpuOnly Mode

**CpuOnly** executes the model entirely within system RAM without utilizing the GPU. This is the slowest execution path, reserved for situations where no suitable GPU memory is available or when running on CPU-only inference servers.

### TensorParallel Mode

**TensorParallel** distributes the model across multiple nodes using NCCL (NVIDIA Collective Communications Library), enabling large-scale parallel inference across clusters. This mode is essential for models too large for single-node memory or when maximizing throughput via distributed computing.

## How RunMode Integrates with Model Evaluation

The core analysis engine uses `RunMode` to compute precise memory requirements and generate runtime configurations:

- **Memory Planning**: The `memory_required_gb` field in `ModelFit` calculates exact VRAM and RAM needs for the selected run mode
- **CLI Generation**: In [[`llmfit-tui/src/display.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-tui/src/display.rs)](https://github.com/AlexsJones/llmfit/blob/main/llmfit-tui/src/display.rs) (lines 71-99), `llmfit` translates `RunMode` into appropriate command-line flags for underlying runtimes like `llama-cli`
- **API Serialization**: [[`llmfit-tui/src/serve_shared.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-tui/src/serve_shared.rs)](https://github.com/AlexsJones/llmfit/blob/main/llmfit-tui/src/serve_shared.rs) handles serializing `RunMode` variants for the Axum HTTP API
- **Desktop UI Mapping**: [[`llmfit-desktop/src/main.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-desktop/src/main.rs)](https://github.com/AlexsJones/llmfit/blob/main/llmfit-desktop/src/main.rs) converts technical run modes into user-friendly interface labels

Additionally, [[`llmfit-core/src/plan.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/plan.rs)](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/plan.rs) relates planning paths (`PlanRunPath`) to their corresponding `RunMode` variants during the optimization phase.

## Practical Implementation Examples

### Creating a ModelFit with Specific RunMode

```rust
use llmfit_core::{
    fit::{RunMode, ModelFit, FitLevel, ScoreComponents},
    models::LlmModel,
};

let model = LlmModel::mock(); // placeholder for actual model definition
let fit = ModelFit {
    model,
    fit_level: FitLevel::Perfect,
    run_mode: RunMode::Gpu,
    memory_required_gb: 12.0,
    memory_available_gb: 16.0,
    utilization_pct: 75.0,
    notes: vec![],
    moe_offloaded_gb: None,
    score: 92.3,
    score_components: ScoreComponents {
        quality: 95.0,
        speed: 90.0,
        fit: 80.0,
        context: 85.0,
    },
    estimated_tps: 1200.0,
    best_quant: "q4_0".into(),
    use_case: Default::default(),
    runtime: Default::default(),
};

```

### Generating Runtime Commands

```rust
let cmd = llmfit_tui::display::generate_llamacpp_command(&fit);
println!("{}", cmd.unwrap());
// Output: "llama-cli -hf some/repo:q4_0 -ngl all -c 2048"

```

### Querying Run Mode via HTTP API

```bash
curl http://localhost:3000/api/v1/models/llama-2-7b | jq '.run_mode'

# Returns: "gpu"

```

## Summary

- **llmfit** classifies hardware execution strategies into five distinct `RunMode` variants: Gpu, MoeOffload, CpuOffload, CpuOnly, and TensorParallel.
- The `RunMode` enum in [`llmfit-core/src/fit.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/fit.rs) operates independently from memory fit levels, enabling separate evaluation of execution feasibility and memory efficiency.
- Each mode dictates specific memory allocation patterns across GPU VRAM and system RAM, with direct impact on inference speed measured in `estimated_tps`.
- The system automatically generates appropriate runtime commands and API responses based on the selected run mode, bridging hardware analysis and actual execution.

## Frequently Asked Questions

### Which LLM run mode provides the best inference performance?

**Gpu** mode delivers the highest performance by keeping all model weights in video memory, eliminating data transfer bottlenecks between CPU and GPU. According to the `llmfit` scoring system, this mode typically yields the highest `estimated_tps` values and quality scores when sufficient VRAM is available.

### What distinguishes MoeOffload from standard CpuOffload?

**MoeOffload** is specifically optimized for Mixture-of-Experts architectures, selectively keeping active experts in VRAM while offloading inactive ones to RAM. In contrast, **CpuOffload** operates on the entire model architecture indiscriminately, splitting weights between GPU and CPU memory based on capacity rather than architectural hot-paths.

### How does llmfit determine which run mode to recommend?

The recommendation engine in `llmfit-core` evaluates available hardware resources through the `ModelFit` struct, comparing `memory_required_gb` against `memory_available_gb` for each possible `RunMode` variant. When VRAM is insufficient for **Gpu** mode, the system cascades through **MoeOffload**, **CpuOffload**, and finally **CpuOnly** based on the specific memory constraints and model architecture.

### Can TensorParallel mode function on a single multi-GPU machine?

While **TensorParallel** is designed for multi-node cluster execution via NCCL, the underlying implementation supports any NCCL-compatible topology, including single-node multi-GPU configurations. However, `llmfit` typically reserves this mode for distributed scenarios where model parameters exceed single-node memory capacity.