How llmfit Handles Multi-GPU Configurations Across Different Vendors

llmfit detects all GPUs from every vendor on a system, aggregates their VRAM into a unified memory pool, and selects the largest discrete GPU as primary while keeping mixed-vendor setups active for tensor-parallel inference.

The llmfit hardware detection system in AlexsJones/llmfit solves the challenge of evaluating LLM model fit on heterogeneous machines—whether you are running 2× NVIDIA RTX 4090s alongside an AMD Radeon, or mixing CUDA and ROCm devices in a single inference cluster. This guide explains how the multi-GPU configuration pipeline works, from vendor-agnostic detection to unified memory handling.

How llmfit Detects GPUs from Every Vendor

The entry point is SystemSpecs::detect() in llmfit-core/src/hardware.rs. Inside this method, detect_all_gpus() (lines 59–106) executes a non-short-circuiting cascade that queries GPU vendors sequentially rather than stopping at the first match.

The vendor detection cascade

The function calls these vendor-specific routines in order:

• detect_nvidia_gpus() – Parses nvidia-smi or falls back to /sys filesystem queries for CUDA devices. Includes special handling for NVIDIA Grace and DGX Spark unified-memory systems.
• detect_amd_gpu_rocm_info() – Uses rocm-smi for ROCm-enabled AMD GPUs.
• detect_amd_gpu_sysfs_info() – Reads /sys/class/drm for AMD hardware without ROCm tooling.
• detect_gpu_windows_info() – PowerShell WMI or wmic queries on Windows hosts.
• detect_intel_gpus() – lspci and sysfs parsing for Intel Arc and integrated graphics.
• detect_apple_gpu() – Metal framework queries on macOS Apple Silicon.
• detect_ascend_npus() – npu-smi for Huawei Ascend AI accelerators.
• detect_vulkan_gpu_info() – Generic Vulkan fallback for any remaining GPUs.

Each function returns Vec<GpuInfo>, allowing multiple same-model GPUs to be reported with a count field rather than duplicate entries.

The GpuInfo Struct and Per-GPU Metadata

Detected GPUs populate this Rust structure defined in hardware.rs:

pub struct GpuInfo {
    pub name: String,
    pub vram_gb: Option<f64>,
    pub backend: GpuBackend,
    pub count: u32,          // >1 for same-model multi-GPU (e.g. 2× RTX 4090)
    pub unified_memory: bool,
}

The backend field uses the GpuBackend enum (CUDA, ROCm, Metal, Vulkan, SYCL, AscendCL) consumed by llmfit-core/src/providers.rs for dispatching inference workloads.

Merging Multi-Vendor Results Without Short-Circuiting

A critical design choice appears in lines 60–62 of hardware.rs: the code explicitly does not short-circuit when NVIDIA GPUs are detected. This preserves AMD, Intel, or other vendors in the same gpus vector.

The merging logic:

1. Concatenates all vendor-specific Vec<GpuInfo> results.
2. Deduplicates same-model GPUs by aggregating their count.
3. Preserves mixed-vendor entries for heterogeneous pool calculations.

This means a machine with both NVIDIA and AMD GPUs retains both in the active hardware profile—unlike many ML frameworks that default to a single vendor.

Primary GPU Selection and Total VRAM Aggregation

After detection, llmfit performs two ranking operations (lines 100–118):

Primary GPU selection – Sorts by VRAM descending and picks the first entry:

gpus.sort_by(|a, b| {
    let va = a.vram_gb.unwrap_or(0.0);
    let vb = b.vram_gb.unwrap_or(0.0);
    vb.partial_cmp(&va).unwrap()
});
let primary_gpu = gpus.first().cloned();

Total VRAM pool calculation – Sums VRAM across all detected GPUs, accounting for multi-card setups:

let total_gpu_vram_gb = {
    let sum: f64 = gpus
        .iter()
        .filter_map(|g| g.vram_gb.map(|v| v * g.count as f64))
        .sum();
    if sum > 0.0 { Some(sum) } else { None }
};

The total_gpu_vram_gb value enables tensor-parallel inference decisions in fit.rs (line 402) and resource claims in share.rs (line 150).

Discrete vs. Integrated GPU Preference

On non-macOS platforms, llmfit applies a preference filter (lines 298–305). The code examines PCI device class codes to distinguish discrete GPUs from integrated graphics. This ensures:

• Primary selection favors dGPUs for inference workloads.
• iGPU VRAM still counts toward the total pool for memory-tier calculations.

This matters for laptops with hybrid Intel+NVIDIA or AMD APU+dGPU configurations.

Unified Memory Handling for Apple Silicon, AMD APUs, and NVIDIA Grace

Three GPU categories share system RAM rather than dedicated VRAM. llmfit handles these specially (lines 330–346):

Platform	Detection Logic	VRAM Reporting
Apple Silicon	Metal query in detect_apple_gpu()	Total system RAM
AMD APUs	unified_memory = true in AMD path	Total system RAM
NVIDIA Grace / DGX Spark	Flag set inside NVIDIA detection	Total system RAM

The unified_memory boolean influences fit scoring because the same memory pool serves both CPU and GPU workloads—unlike discrete GPUs with dedicated HBM or GDDR.

Practical Example: Detecting a Mixed-Vendor System

This code exercises the full detection pipeline:

use llmfit_core::hardware::SystemSpecs;

fn main() {
    // Detect the current machine's hardware profile
    let specs = SystemSpecs::detect();

    // Print a summary of all GPUs
    for gpu in &specs.gpus {
        println!(
            "{} – {} GB ({}), count: {}, unified: {}",
            gpu.name,
            gpu.vram_gb.map_or("unknown".into(), |v| format!("{:.1}", v)),
            gpu.backend.label(),
            gpu.count,
            gpu.unified_memory,
        );
    }

    // Show the aggregated VRAM used for multi-GPU inference
    if let Some(total) = specs.total_gpu_vram_gb {
        println!("Total GPU VRAM pool: {:.1} GB", total);
    } else {
        println!("No GPU VRAM detected");
    }
}

On a machine with 2× RTX 4090 and an AMD Radeon 7600, output is:

NVIDIA GeForce RTX 4090 – 24.0 GB (CUDA), count: 2, unified: false
AMD Radeon 7600 – 8.0 GB (Vulkan), count: 1, unified: false
Total GPU VRAM pool: 56.0 GB

The same binary produces correct output on Apple Silicon (unified memory reported), Windows mixed Intel/NVIDIA laptops, and ROCm-only Linux servers.

Source Files and Cross-Module Integration

File	Role in Multi-GPU Handling
llmfit-core/src/hardware.rs	Core detection, GpuInfo struct, aggregation logic
llmfit-core/src/fit.rs	Consumes total_gpu_vram_gb for model fit scoring (line 402)
llmfit-core/src/share.rs	Uses aggregated VRAM for benchmark claims and resource tiers (line 150)
llmfit-core/src/providers.rs	Dispatches to GpuBackend-specific inference implementations

Summary

• llmfit queries every GPU vendor sequentially without short-circuiting, enabling true multi-vendor detection.
• Same-model GPUs merge via the count field; mixed vendors coexist in the active hardware profile.
• Total VRAM aggregates across all cards to support tensor-parallel model splitting decisions.
• Discrete GPUs are preferred for primary selection, but integrated GPU memory still contributes to the pool.
• Unified-memory platforms (Apple Silicon, AMD APUs, NVIDIA Grace) report system RAM as VRAM with appropriate flags.

Frequently Asked Questions

Does llmfit support running inference across NVIDIA and AMD GPUs simultaneously?

Yes. llmfit-core/src/hardware.rs preserves both vendors in the detected GPU list, and total_gpu_vram_gb aggregates their memory. However, actual cross-vendor tensor parallelism depends on backend support in providers.rs—the detection layer makes the hardware visible, but inference dispatching may require vendor-specific pipelines.

How does llmfit handle a machine with 4 identical GPUs versus 4 different GPUs?

Identical GPUs are merged: four RTX 4090s appear as one entry with count: 4. Different GPUs remain separate entries. Both configurations sum to the same total_gpu_vram_gb, but the GpuInfo vector length differs.

What happens if nvidia-smi is missing but NVIDIA GPUs are present?

The NVIDIA detection path falls back to sysfs parsing in /sys/bus/pci/devices/ when nvidia-smi is unavailable. This ensures containerized or minimal environments still report CUDA-capable hardware.

Does Apple Silicon report VRAM correctly for large models?

Yes. Apple Silicon detection sets unified_memory: true and reports total system RAM as vram_gb. The fit scoring logic in fit.rs accounts for unified memory constraints since CPU and GPU share the same pool.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}