How llmfit's Cluster Mode and TensorParallel Enable Multi-Node Inference

llmfit automatically detects when a model exceeds single-GPU capacity and falls back to TensorParallel across multiple nodes via NCCL-backed vLLM, generating human-readable cluster descriptions for transparent distributed inference.

The llmfit inference engine (AlexsJones/llmfit) provides transparent scaling from single-GPU to multi-node deployments through two tightly integrated concepts: cluster mode for cross-machine orchestration and TensorParallel for tensor sharding across GPUs. This article examines how these mechanisms work in the Rust-based core, with direct references to the source implementation.

What Cluster Mode Means in llmfit

Cluster mode activates when local hardware cannot satisfy a model's VRAM requirements. Rather than failing, llmfit's analysis engine selects distributed execution and annotates the deployment plan with cluster-specific metadata.

In llmfit-core/src/fit.rs, the RunMode::TensorParallel variant triggers this path. When selected, the code generates descriptions like "Cluster: tensor-parallel across N nodes via vLLM (TP=X)"—see the cluster-mode comment block.

The detection logic relies on hardware inspection in hardware.rs, where the cluster_mode flag indicates whether the environment supports multi-node execution:

// From llmfit-core/src/hardware.rs (lines 66-68)
// cluster_mode: true when NCCL/InfiniBand environment variables are present,
// indicating potential for cross-node tensor parallelism

This flag feeds into the planning phase to determine when TensorParallel becomes mandatory versus optional.

TensorParallel: The Core Distributed Primitive

TensorParallel splits weight tensors across GPUs, enabling inference of models larger than any single device's memory. llmfit implements this through a dedicated enum variant and provider delegation.

RunMode Definition

The TensorParallel variant appears in the RunMode enum:

pub enum RunMode {
    // ... other variants ...
    TensorParallel {
        tp_size: u8,           // GPUs per tensor-parallel group
        num_nodes: u16,        // Total nodes in cluster
        memory_per_gpu_gb: f32,
    },
}

This structure captures both intra-node and inter-node parallelism parameters.

Planning and Memory Calculation

plan.rs computes the required topology. For a 70B parameter model with Q4_K_M quantization:

use llmfit_core::{ModelFit, RunMode, Quantization};

let specs = SystemSpecs::detect()?;
let model = db.find("Meta/Llama-2-70b-chat-hf")?;
let fit = ModelFit::analyze(
    &specs,
    &model,
    Quantization::Q4_K_M,
    UseCase::Chat,  // May force tensor-parallel for latency requirements
)?;

// fit.run_mode will be TensorParallel if:
// - model.min_vram_gb > specs.total_vram_gb, OR
// - use_case.latency_target requires parallel decode

The planner returns (RunMode, memory_required, pool) where the TensorParallel branch calculates per-node memory budgets and validates NCCL connectivity.

Execution via vLLM Provider

The actual distributed runtime is delegated to vLLM through providers.rs. When RunMode::TensorParallel is selected:

1. The provider constructs vLLM launch arguments including --tensor-parallel-size
2. NCCL environment variables (NCCL_SOCKET_IFNAME, NCCL_IB_DISABLE, etc.) are propagated
3. Ray or native PyTorch distributed initializes the cross-node process group

This abstraction allows llmfit to support multiple backends while presenting a unified interface.

User-Facing Output and Transparency

Both CLI and TUI surfaces render distributed configurations clearly. The UI code in [tui_ui.rs](https://github.com/AlexsJones/llmfit/blob/main/llmfit-tui/src/tui_ui.rs#L886-L898) and [display.rs](https://github.com/AlexsJones/llmfit/blob/main/llmfit-tui/src/display.rs#L643-L704) matches on RunMode::TensorParallel to:

• Abbreviate as "TP" in compact views
• Expand to full cluster descriptions in detailed views
• Surface warnings when cluster_mode is false but TensorParallel is required

Example CLI output:

$ cargo run -- fit --model "Meta/Llama-2-70b-chat-hf" --quantization Q4_K_M

Run mode: TP
Cluster: tensor-parallel across 4 nodes via vLLM (TP=2)
Memory per GPU: 18.5 GB
Estimated throughput: 24.3 tok/s

Forcing TensorParallel Explicitly

Advanced users can override automatic detection:

// Explicit tensor-parallel request
let fit = ModelFit::analyze_with_forced_runtime(
    &specs,
    &model,
    RunMode::TensorParallel,  // Bypass single-GPU optimization
    Quantization::Q4_K_M,
    UseCase::CodeGeneration,  // Latency-sensitive; force parallelism
)?;

This path skips the VRAM threshold check and proceeds directly to multi-node planning.

Key Implementation Files

File	Responsibility	Line References
llmfit-core/src/fit.rs	RunMode enum, cluster description generation	L188-196, L400-412
llmfit-core/src/hardware.rs	cluster_mode detection, hardware specification	L66-68
llmfit-core/src/providers.rs	Backend selection, vLLM/NCCL orchestration	full file
llmfit-core/src/plan.rs	Topology calculation, memory budgeting	full file
llmfit-tui/src/tui_ui.rs	TUI rendering of run modes	L886-898
llmfit-tui/src/display.rs	CLI formatting, compact and detailed views	L643-704

Summary

• Automatic fallback: llmfit detects VRAM shortfalls via hardware.rs and selects TensorParallel without manual intervention
• Structured run mode: The RunMode::TensorParallel variant encapsulates topology, enabling type-safe distributed planning
• Backend delegation: Actual multi-node execution flows through provider abstractions to vLLM with NCCL
• Transparent UI: Both abbreviated ("TP") and detailed cluster descriptions surface distributed configurations clearly
• Explicit override: analyze_with_forced_runtime() permits bypassing single-GPU optimization when latency or throughput requirements demand parallelism

Frequently Asked Questions

What triggers llmfit to enter cluster mode?

Cluster mode activates when SystemSpecs::detect() determines that model.min_vram_gb exceeds specs.total_vram_gb and the cluster_mode environment flag is present (NCCL/InfiniBand variables). The planner then selects RunMode::TensorParallel and calculates multi-node topology in plan.rs. This automatic fallback ensures large models like Llama-2-70B can run even on constrained hardware configurations.

How does TensorParallel differ from PipelineParallel in llmfit?

llmfit currently implements TensorParallel (intra-layer sharding) rather than PipelineParallel (inter-layer partitioning). TensorParallel splits individual weight matrices across GPUs, requiring all-reduce communication after each layer but avoiding pipeline bubbles. The RunMode enum reserves capacity for future PipelineParallel variants, but as of the main branch, only TensorParallel is exposed for multi-node execution.

Can I run TensorParallel on a single multi-GPU node?

Yes. The same TensorParallel mechanism handles both single-node multi-GPU and true multi-node configurations. The num_nodes field in the variant distinguishes these cases: num_nodes: 1 with tp_size: 4 indicates four GPUs on one machine, while num_nodes: 2 with tp_size: 4 indicates eight total GPUs across two nodes. NCCL optimizes communication paths accordingly.

What backend does llmfit use for distributed inference?

llmfit delegates to vLLM for TensorParallel execution, launched through the provider abstraction in providers.rs. The provider constructs vLLM command-line arguments including --tensor-parallel-size and propagates NCCL environment variables for cross-node communication. This design allows future extension to TensorRT-LLM or other distributed backends without changing the core planning logic.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}