# How llmfit Handles Multi-GPU Configurations Across Different Vendors

> llmfit seamlessly manages multi-GPU setups from various vendors, unifying VRAM and enabling efficient tensor-parallel inference on mixed hardware.

- Repository: [Alex Jones/llmfit](https://github.com/AlexsJones/llmfit)
- Tags: how-to-guide
- Published: 2026-08-20

---

**llmfit detects all GPUs from every vendor on a system, aggregates their VRAM into a unified memory pool, and selects the largest discrete GPU as primary while keeping mixed-vendor setups active for tensor-parallel inference.**

The `llmfit` hardware detection system in `AlexsJones/llmfit` solves the challenge of evaluating LLM model fit on heterogeneous machines—whether you are running **2× NVIDIA RTX 4090s alongside an AMD Radeon**, or mixing **CUDA and ROCm devices in a single inference cluster**. This guide explains how the **multi-GPU configuration** pipeline works, from vendor-agnostic detection to unified memory handling.

## How llmfit Detects GPUs from Every Vendor

The entry point is `SystemSpecs::detect()` in [`llmfit-core/src/hardware.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/hardware.rs). Inside this method, `detect_all_gpus()` (lines 59–106) executes a **non-short-circuiting cascade** that queries GPU vendors sequentially rather than stopping at the first match.

### The vendor detection cascade

The function calls these vendor-specific routines in order:

- **`detect_nvidia_gpus()`** – Parses `nvidia-smi` or falls back to `/sys` filesystem queries for CUDA devices. Includes special handling for **NVIDIA Grace** and **DGX Spark** unified-memory systems.
- **`detect_amd_gpu_rocm_info()`** – Uses `rocm-smi` for ROCm-enabled AMD GPUs.
- **`detect_amd_gpu_sysfs_info()`** – Reads `/sys/class/drm` for AMD hardware without ROCm tooling.
- **`detect_gpu_windows_info()`** – PowerShell WMI or `wmic` queries on Windows hosts.
- **`detect_intel_gpus()`** – `lspci` and sysfs parsing for Intel Arc and integrated graphics.
- **`detect_apple_gpu()`** – Metal framework queries on macOS Apple Silicon.
- **`detect_ascend_npus()`** – `npu-smi` for Huawei Ascend AI accelerators.
- **`detect_vulkan_gpu_info()`** – Generic Vulkan fallback for any remaining GPUs.

Each function returns `Vec<GpuInfo>`, allowing multiple same-model GPUs to be reported with a `count` field rather than duplicate entries.

## The GpuInfo Struct and Per-GPU Metadata

Detected GPUs populate this Rust structure defined in [`hardware.rs`](https://github.com/AlexsJones/llmfit/blob/main/hardware.rs):

```rust
pub struct GpuInfo {
    pub name: String,
    pub vram_gb: Option<f64>,
    pub backend: GpuBackend,
    pub count: u32,          // >1 for same-model multi-GPU (e.g. 2× RTX 4090)
    pub unified_memory: bool,
}

```

The **`backend`** field uses the `GpuBackend` enum (CUDA, ROCm, Metal, Vulkan, SYCL, AscendCL) consumed by [`llmfit-core/src/providers.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/providers.rs) for dispatching inference workloads.

## Merging Multi-Vendor Results Without Short-Circuiting

A critical design choice appears in lines 60–62 of [`hardware.rs`](https://github.com/AlexsJones/llmfit/blob/main/hardware.rs): the code **explicitly does not short-circuit** when NVIDIA GPUs are detected. This preserves AMD, Intel, or other vendors in the same `gpus` vector.

The merging logic:

1. Concatenates all vendor-specific `Vec<GpuInfo>` results.
2. **Deduplicates same-model GPUs** by aggregating their `count`.
3. **Preserves mixed-vendor entries** for heterogeneous pool calculations.

This means a machine with both NVIDIA and AMD GPUs retains both in the active hardware profile—unlike many ML frameworks that default to a single vendor.

## Primary GPU Selection and Total VRAM Aggregation

After detection, `llmfit` performs two ranking operations (lines 100–118):

**Primary GPU selection** – Sorts by VRAM descending and picks the first entry:

```rust
gpus.sort_by(|a, b| {
    let va = a.vram_gb.unwrap_or(0.0);
    let vb = b.vram_gb.unwrap_or(0.0);
    vb.partial_cmp(&va).unwrap()
});
let primary_gpu = gpus.first().cloned();

```

**Total VRAM pool calculation** – Sums VRAM across *all* detected GPUs, accounting for multi-card setups:

```rust
let total_gpu_vram_gb = {
    let sum: f64 = gpus
        .iter()
        .filter_map(|g| g.vram_gb.map(|v| v * g.count as f64))
        .sum();
    if sum > 0.0 { Some(sum) } else { None }
};

```

The **`total_gpu_vram_gb`** value enables **tensor-parallel inference decisions** in [`fit.rs`](https://github.com/AlexsJones/llmfit/blob/main/fit.rs) (line 402) and resource claims in [`share.rs`](https://github.com/AlexsJones/llmfit/blob/main/share.rs) (line 150).

## Discrete vs. Integrated GPU Preference

On non-macOS platforms, `llmfit` applies a preference filter (lines 298–305). The code examines PCI device class codes to distinguish discrete GPUs from integrated graphics. This ensures:

- **Primary selection favors dGPUs** for inference workloads.
- **iGPU VRAM still counts** toward the total pool for memory-tier calculations.

This matters for laptops with hybrid Intel+NVIDIA or AMD APU+dGPU configurations.

## Unified Memory Handling for Apple Silicon, AMD APUs, and NVIDIA Grace

Three GPU categories share system RAM rather than dedicated VRAM. `llmfit` handles these specially (lines 330–346):

| Platform | Detection Logic | VRAM Reporting |
|----------|-----------------|--------------|
| Apple Silicon | Metal query in `detect_apple_gpu()` | Total system RAM |
| AMD APUs | `unified_memory = true` in AMD path | Total system RAM |
| NVIDIA Grace / DGX Spark | Flag set inside NVIDIA detection | Total system RAM |

The `unified_memory` boolean influences fit scoring because the same memory pool serves both CPU and GPU workloads—unlike discrete GPUs with dedicated HBM or GDDR.

## Practical Example: Detecting a Mixed-Vendor System

This code exercises the full detection pipeline:

```rust
use llmfit_core::hardware::SystemSpecs;

fn main() {
    // Detect the current machine's hardware profile
    let specs = SystemSpecs::detect();

    // Print a summary of all GPUs
    for gpu in &specs.gpus {
        println!(
            "{} – {} GB ({}), count: {}, unified: {}",
            gpu.name,
            gpu.vram_gb.map_or("unknown".into(), |v| format!("{:.1}", v)),
            gpu.backend.label(),
            gpu.count,
            gpu.unified_memory,
        );
    }

    // Show the aggregated VRAM used for multi-GPU inference
    if let Some(total) = specs.total_gpu_vram_gb {
        println!("Total GPU VRAM pool: {:.1} GB", total);
    } else {
        println!("No GPU VRAM detected");
    }
}

```

On a machine with **2× RTX 4090** and an **AMD Radeon 7600**, output is:

```text
NVIDIA GeForce RTX 4090 – 24.0 GB (CUDA), count: 2, unified: false
AMD Radeon 7600 – 8.0 GB (Vulkan), count: 1, unified: false
Total GPU VRAM pool: 56.0 GB

```

The same binary produces correct output on Apple Silicon (unified memory reported), Windows mixed Intel/NVIDIA laptops, and ROCm-only Linux servers.

## Source Files and Cross-Module Integration

| File | Role in Multi-GPU Handling |
|------|---------------------------|
| [`llmfit-core/src/hardware.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/hardware.rs) | Core detection, `GpuInfo` struct, aggregation logic |
| [`llmfit-core/src/fit.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/fit.rs) | Consumes `total_gpu_vram_gb` for model fit scoring (line 402) |
| [`llmfit-core/src/share.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/share.rs) | Uses aggregated VRAM for benchmark claims and resource tiers (line 150) |
| [`llmfit-core/src/providers.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/providers.rs) | Dispatches to `GpuBackend`-specific inference implementations |

## Summary

- **llmfit** queries every GPU vendor sequentially without short-circuiting, enabling true multi-vendor detection.
- Same-model GPUs merge via the `count` field; mixed vendors coexist in the active hardware profile.
- Total VRAM aggregates across all cards to support tensor-parallel model splitting decisions.
- Discrete GPUs are preferred for primary selection, but integrated GPU memory still contributes to the pool.
- Unified-memory platforms (Apple Silicon, AMD APUs, NVIDIA Grace) report system RAM as VRAM with appropriate flags.

## Frequently Asked Questions

### Does llmfit support running inference across NVIDIA and AMD GPUs simultaneously?

Yes. [`llmfit-core/src/hardware.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/hardware.rs) preserves both vendors in the detected GPU list, and `total_gpu_vram_gb` aggregates their memory. However, actual cross-vendor tensor parallelism depends on backend support in [`providers.rs`](https://github.com/AlexsJones/llmfit/blob/main/providers.rs)—the detection layer makes the hardware visible, but inference dispatching may require vendor-specific pipelines.

### How does llmfit handle a machine with 4 identical GPUs versus 4 different GPUs?

Identical GPUs are merged: four RTX 4090s appear as one entry with `count: 4`. Different GPUs remain separate entries. Both configurations sum to the same `total_gpu_vram_gb`, but the `GpuInfo` vector length differs.

### What happens if nvidia-smi is missing but NVIDIA GPUs are present?

The NVIDIA detection path falls back to sysfs parsing in `/sys/bus/pci/devices/` when `nvidia-smi` is unavailable. This ensures containerized or minimal environments still report CUDA-capable hardware.

### Does Apple Silicon report VRAM correctly for large models?

Yes. Apple Silicon detection sets `unified_memory: true` and reports total system RAM as `vram_gb`. The fit scoring logic in [`fit.rs`](https://github.com/AlexsJones/llmfit/blob/main/fit.rs) accounts for unified memory constraints since CPU and GPU share the same pool.