LLM Run Modes in llmfit: A Complete Guide to GPU, CPU, and Distributed Execution

llmfit defines five distinct LLM run modes—Gpu, MoeOffload, CpuOffload, CpuOnly, and TensorParallel—that determine exactly how model weights are distributed across GPU VRAM, system RAM, and multi-node clusters for optimal inference performance.

The llmfit project by AlexsJones analyzes hardware compatibility for large language models, evaluating every feasible execution path through its core analysis engine. Understanding these LLM run modes enables developers to select the optimal execution strategy based on available hardware constraints and performance requirements.

The RunMode Enum in llmfit-core

The execution strategy classification lives in [llmfit-core/src/fit.rs](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/fit.rs) at lines 88-94, where the RunMode enum is deliberately decoupled from the FitLevel (memory fit classification). This separation allows llmfit to evaluate both how well a model fits and how it will run independently.

The RunMode type is embedded within the ModelFit struct (defined at lines 40-44), which tracks:

• memory_required_gb: Total memory needed for the selected execution path
• memory_available_gb: Host memory capacity
• moe_offloaded_gb: Specific to MoE models, tracking offloaded expert weights
• estimated_tps: Predicted tokens-per-second performance

The Five LLM Run Modes Explained

Gpu Mode

Gpu mode loads all model weights directly into GPU VRAM, providing the fastest inference path available. This mode requires sufficient VRAM capacity for the entire model and is the default target when hardware resources permit full acceleration.

MoeOffload Mode

MoeOffload (Mixture-of-Experts offloading) is specifically designed for MoE architectures. In this mode, llmfit keeps active experts resident in VRAM while offloading inactive experts to system RAM. This hybrid approach balances the massive parameter counts of MoE models against limited GPU memory.

CpuOffload Mode

CpuOffload creates a hybrid execution environment where the model is partially resident on the GPU with the remainder spilling into system RAM. This mode activates when model size exceeds available VRAM but the system still benefits from partial GPU acceleration.

CpuOnly Mode

CpuOnly executes the model entirely within system RAM without utilizing the GPU. This is the slowest execution path, reserved for situations where no suitable GPU memory is available or when running on CPU-only inference servers.

TensorParallel Mode

TensorParallel distributes the model across multiple nodes using NCCL (NVIDIA Collective Communications Library), enabling large-scale parallel inference across clusters. This mode is essential for models too large for single-node memory or when maximizing throughput via distributed computing.

How RunMode Integrates with Model Evaluation

The core analysis engine uses RunMode to compute precise memory requirements and generate runtime configurations:

• Memory Planning: The memory_required_gb field in ModelFit calculates exact VRAM and RAM needs for the selected run mode
• CLI Generation: In [llmfit-tui/src/display.rs](https://github.com/AlexsJones/llmfit/blob/main/llmfit-tui/src/display.rs) (lines 71-99), llmfit translates RunMode into appropriate command-line flags for underlying runtimes like llama-cli
• API Serialization: [llmfit-tui/src/serve_shared.rs](https://github.com/AlexsJones/llmfit/blob/main/llmfit-tui/src/serve_shared.rs) handles serializing RunMode variants for the Axum HTTP API
• Desktop UI Mapping: [llmfit-desktop/src/main.rs](https://github.com/AlexsJones/llmfit/blob/main/llmfit-desktop/src/main.rs) converts technical run modes into user-friendly interface labels

Additionally, [llmfit-core/src/plan.rs](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/plan.rs) relates planning paths (PlanRunPath) to their corresponding RunMode variants during the optimization phase.

Practical Implementation Examples

Creating a ModelFit with Specific RunMode

use llmfit_core::{
    fit::{RunMode, ModelFit, FitLevel, ScoreComponents},
    models::LlmModel,
};

let model = LlmModel::mock(); // placeholder for actual model definition
let fit = ModelFit {
    model,
    fit_level: FitLevel::Perfect,
    run_mode: RunMode::Gpu,
    memory_required_gb: 12.0,
    memory_available_gb: 16.0,
    utilization_pct: 75.0,
    notes: vec![],
    moe_offloaded_gb: None,
    score: 92.3,
    score_components: ScoreComponents {
        quality: 95.0,
        speed: 90.0,
        fit: 80.0,
        context: 85.0,
    },
    estimated_tps: 1200.0,
    best_quant: "q4_0".into(),
    use_case: Default::default(),
    runtime: Default::default(),
};

Generating Runtime Commands

let cmd = llmfit_tui::display::generate_llamacpp_command(&fit);
println!("{}", cmd.unwrap());
// Output: "llama-cli -hf some/repo:q4_0 -ngl all -c 2048"

Querying Run Mode via HTTP API

curl http://localhost:3000/api/v1/models/llama-2-7b | jq '.run_mode'

# Returns: "gpu"

Summary

• llmfit classifies hardware execution strategies into five distinct RunMode variants: Gpu, MoeOffload, CpuOffload, CpuOnly, and TensorParallel.
• The RunMode enum in llmfit-core/src/fit.rs operates independently from memory fit levels, enabling separate evaluation of execution feasibility and memory efficiency.
• Each mode dictates specific memory allocation patterns across GPU VRAM and system RAM, with direct impact on inference speed measured in estimated_tps.
• The system automatically generates appropriate runtime commands and API responses based on the selected run mode, bridging hardware analysis and actual execution.

Frequently Asked Questions

Which LLM run mode provides the best inference performance?

Gpu mode delivers the highest performance by keeping all model weights in video memory, eliminating data transfer bottlenecks between CPU and GPU. According to the llmfit scoring system, this mode typically yields the highest estimated_tps values and quality scores when sufficient VRAM is available.

What distinguishes MoeOffload from standard CpuOffload?

MoeOffload is specifically optimized for Mixture-of-Experts architectures, selectively keeping active experts in VRAM while offloading inactive ones to RAM. In contrast, CpuOffload operates on the entire model architecture indiscriminately, splitting weights between GPU and CPU memory based on capacity rather than architectural hot-paths.

How does llmfit determine which run mode to recommend?

The recommendation engine in llmfit-core evaluates available hardware resources through the ModelFit struct, comparing memory_required_gb against memory_available_gb for each possible RunMode variant. When VRAM is insufficient for Gpu mode, the system cascades through MoeOffload, CpuOffload, and finally CpuOnly based on the specific memory constraints and model architecture.

Can TensorParallel mode function on a single multi-GPU machine?

While TensorParallel is designed for multi-node cluster execution via NCCL, the underlying implementation supports any NCCL-compatible topology, including single-node multi-GPU configurations. However, llmfit typically reserves this mode for distributed scenarios where model parameters exceed single-node memory capacity.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}