How ODS Handles Different GPU Backends: NVIDIA, AMD, and Apple Silicon Detection

ODS detects your GPU during the installation phase by scanning PCI vendor IDs and system paths, then sets environment variables that determine which container orchestration files, driver configurations, and resource limits to apply.

The Osmantic/ODS (Open-Source Distributed Stack) repository automates GPU backend detection through a centralized shell library that identifies hardware from NVIDIA, AMD, and Apple Silicon before the stack deploys any containers. Understanding how ODS handles different GPU backends ensures you can troubleshoot detection failures and optimize resource allocation for your specific hardware.

The Detection Pipeline

ODS populates a standardized set of environment variables during the detect_gpu phase in [ods/installers/lib/detection.sh](https://github.com/Osmantic/ODS/blob/main/ods/installers/lib/detection.sh). These variables drive backend-specific orchestration decisions:

• GPU_BACKEND: One of nvidia, amd, intel, apple, jetson, or cpu (fallback)
• GPU_NAME: Human-readable model name (e.g., “RTX 4090”)
• GPU_VRAM: Total video memory in MiB (or unified RAM for Apple/Jetson)
• GPU_COUNT: Number of GPUs detected
• GPU_MEMORY_TYPE: discrete, unified, mixed, or none
• GPU_DEVICE_ID: PCI device ID or unique identifier for Apple/Jetson
• HAS_NPU: Set to true for AMD systems with a Ryzen AI NPU

If no GPU is detected, the script falls back to CPU-only mode (GPU_BACKEND=cpu) and issues a warning (line 119-126).

NVIDIA GPU Detection

The NVIDIA detection branch handles everything from legacy GTX cards to Blackwell architecture and Grace-Hopper unified memory systems.

Hardware Identification

The script scans /sys/class/drm/*/device/vendor for the NVIDIA PCI vendor ID 0x10de (line 335-344). On WSL2 environments where PCI paths may not exist, it falls back to checking for the presence of the nvidia-smi binary (line 445-448).

VRAM and Multi-GPU Naming

Once identified, ODS executes nvidia-smi --query-gpu=name,memory.total (line 511-518) to populate GPU_NAME and GPU_VRAM. For systems reporting 0 or [N/A] VRAM—common in Grace-Hopper unified memory architectures—the script treats system RAM as video memory (line 561-573) and sets GPU_MEMORY_TYPE=unified.

For multi-GPU configurations, the script constructs concise display names like “RTX 4090 × 2” or “RTX 4090 + RTX 4080” (line 581-595).

Blackwell and Secure Boot

Later installation phases invoke fix_nvidia_secure_boot (line 166) to verify that Blackwell GPUs use open kernel modules. If Secure Boot is enabled, ODS triggers a key-enrollment flow to ensure the NVIDIA driver loads properly.

AMD GPU Detection

The AMD branch supports both discrete RDNA GPUs and integrated APUs with unified memory.

Device Discovery and Classification

The detection logic scans /sys/class/drm/*/device/vendor for the AMD PCI vendor ID 0x1002 (line 430-438), collecting all matching card directories into amd_card_dirs. It then reads mem_info_vram_total and mem_info_gtt_total for each device (line 446-452) to distinguish between discrete GPUs (dedicated VRAM), APUs (unified memory), or mixed configurations (line 562-670).

NPU Detection for Hybrid Inference

ODS checks for /sys/class/misc/amdnpu or AMD NPU entries in lspci (line 706-709). When found, it sets HAS_NPU=true, enabling Lemonade-hybrid mode for distributing workloads between GPU and NPU.

Apple Silicon Detection

For macOS hosts, ODS delegates initial detection to [ods/scripts/detect-hardware.sh](https://github.com/Osmantic/ODS/blob/main/ods/scripts/detect-hardware.sh). The script runs only when the OS detection reports macos (line 50-52), then extracts the CPU brand via sysctl -n machdep.cpu.brand_string (line 53) and unified memory size via hw.memsize (line 55-56).

The function returns identifiers like “Apple M1 Pro” with “16GB unified”, causing the main detection script to set GPU_BACKEND=apple and GPU_MEMORY_TYPE=unified, treating system RAM as VRAM for container resource calculations.

From Detection to Deployment

After identification, the GPU_BACKEND value drives several downstream configuration steps:

Backend Contracts

The load_backend_contract function (line 72-88) maps the detected backend to a JSON configuration in config/backends/, specifying the LLM engine, driver versions, and container runtime requirements for NVIDIA, AMD, or Apple Silicon.

Compose Overlay Selection

The [ods/scripts/resolve-compose-stack.sh](https://github.com/Osmantic/ODS/blob/main/ods/scripts/resolve-compose-stack.sh) script maps each backend to a specific Docker Compose overlay file (line 151-158):

• docker-compose.nvidia.yml for CUDA workloads
• docker-compose.amd.yml for ROCm stacks
• docker-compose.apple.yml for Apple Silicon optimized containers

Resource Tier Assignment

The [ods/installers/lib/tier-map.sh](https://github.com/Osmantic/ODS/blob/main/ods/installers/lib/tier-map.sh) module uses GPU_VRAM and GPU_MEMORY_TYPE to assign performance tiers, capping CPU usage appropriately when running in CPU-only fallback mode.

You can query the current detection status at any time:

# Query detected GPU status

$ ods gpu status
GPU Backend: nvidia
GPU Name: RTX 4090
VRAM: 24576 MB
Memory Type: discrete

For testing or forcing a specific backend:

# Force Apple Silicon backend for testing

$ GPU_BACKEND=apple ods gpu status
GPU Backend: apple
GPU Name: Apple M2 Pro
VRAM: 16000 MB
Memory Type: unified

Summary

• ODS detects GPU hardware by scanning PCI vendor IDs (0x10de for NVIDIA, 0x1002 for AMD) and macOS system calls for Apple Silicon.
• The detect_gpu function in ods/installers/lib/detection.sh exports standardized environment variables that determine backend selection.
• NVIDIA detection includes special handling for Blackwell Secure Boot and unified memory (Grace-Hopper) configurations.
• AMD detection parses mem_info_vram_total to distinguish discrete GPUs from APUs and detects Ryzen AI NPUs via /sys/class/misc/amdnpu.
• Apple Silicon relies on sysctl calls to identify M1/M2/M3 chips and unified memory sizes.
• The detected GPU_BACKEND value selects Docker Compose overlays, backend contracts, and resource limits automatically.

Frequently Asked Questions

How does ODS detect GPUs in WSL2 environments where PCI paths are unavailable?

When /sys/class/drm/*/device/vendor does not exist, ODS falls back to checking for the nvidia-smi binary presence (line 445-448 in detection.sh). If found, it assumes an NVIDIA GPU and proceeds to query VRAM and device names through the SMI interface rather than PCI enumeration.

Can ODS run on systems with multiple different GPU vendors installed simultaneously?

Currently, ODS selects a single GPU_BACKEND based on detection priority (NVIDIA, then AMD, then Apple). For mixed-GPU systems, the first detected vendor takes precedence. Multi-GPU support within the same vendor (e.g., two RTX 4090s) is fully supported via the GPU_COUNT variable and multi-GPU naming logic.

What happens if my NVIDIA GPU reports 0 MB of VRAM during detection?

ODS interprets zero or [N/A] VRAM values as indicators of unified memory architectures like NVIDIA Grace or certain virtualized environments (line 561-573). It falls back to using system RAM as the VRAM value and sets GPU_MEMORY_TYPE=unified, ensuring container limits are still enforced correctly.

Does ODS support AMD integrated graphics (APUs) without dedicated VRAM?

Yes. The AMD detection branch reads both mem_info_vram_total and mem_info_gtt_total to calculate available graphics memory (line 446-452). For APUs, it identifies the lack of dedicated VRAM and sets GPU_MEMORY_TYPE=unified, allowing the stack to run on Ryzen integrated graphics with appropriate memory constraints.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}