Which Libraries Does WhichLLM Use for NVIDIA GPU Detection?

WhichLLM relies on the pynvml library as its primary method for NVIDIA GPU detection, with a robust fallback to the nvidia-smi command-line utility invoked via Python's subprocess module when NVML initialization fails.

WhichLLM is an open-source tool designed to match language models with appropriate hardware configurations. To accurately inventory NVIDIA GPUs, the project implements a dual-path detection system in src/whichllm/hardware/nvidia.py that combines direct library integration with system command execution. The libraries used for NVIDIA GPU detection in WhichLLM are specifically chosen to ensure reliable hardware enumeration while maintaining graceful degradation when Python dependencies are unavailable.

Primary Detection via pynvml

The primary detection path leverages pynvml, a Python wrapper for the NVIDIA Management Library (NVML). In src/whichllm/hardware/nvidia.py, the module attempts to import pynvml at line 10 and initializes the NVML session via nvmlInit() to query GPU properties directly from the driver.

When available, WhichLLM calls several NVML functions to populate hardware metadata:

• nvmlDeviceGetCount() – enumerates available GPUs
• nvmlDeviceGetName() – retrieves the GPU model identifier
• nvmlDeviceGetMemoryInfo() – obtains VRAM capacity in bytes
• nvmlSystemGetDriverVersion() – extracts the CUDA driver version

These data points are used to instantiate GPUInfo objects defined in src/whichllm/hardware/types.py. The implementation handles specialized cases such as unified-memory GPUs (Apple Silicon or integrated graphics) and extracts compute capability information where available.

NVML Implementation Details

The core logic resides in lines 122-148 of nvidia.py, where the code iterates through detected devices and constructs hardware profiles. The implementation wraps NVML calls in try/except blocks (lines 112-119) to catch initialization failures, immediately triggering the fallback mechanism if pynvml cannot interface with the NVIDIA driver.

Fallback Detection via nvidia-smi

When pynvml is not installed or NVML initialization fails, WhichLLM automatically switches to a subprocess-based approach using the system's nvidia-smi binary. This fallback is implemented in the _detect_nvidia_gpus_via_smi function (lines 71-80 of nvidia.py).

The fallback mechanism executes nvidia-smi with specific CSV formatting flags via subprocess.run(), then parses the output using regular expressions to extract GPU names and memory statistics. This approach requires no Python dependencies beyond the standard library but depends on the NVIDIA driver utilities being installed and available in the system PATH.

Error Handling Strategy

The detection logic uses guarded import statements and conditional initialization:

# src/whichllm/hardware/nvidia.py (lines 10-14)

try:
    import pynvml
    NVML_AVAILABLE = True
except ImportError:
    NVML_AVAILABLE = False

This pattern ensures that WhichLLM remains functional even in environments without NVIDIA hardware or the NVML Python bindings, allowing the tool to proceed with CPU-only model recommendations.

Implementation Architecture

The NVIDIA detection module follows a layered architecture:

Component	File Path	Responsibility
Detection Engine	src/whichllm/hardware/nvidia.py	Implements both NVML and nvidia-smi detection paths
Data Models	src/whichllm/hardware/types.py	Defines GPUInfo dataclass for hardware metadata
Test Suite	tests/test_nvidia_detection.py	Validates both primary and fallback detection paths

The detect_nvidia_gpus() function serves as the public API, automatically selecting the appropriate backend based on environment availability.

Practical Usage Example

To detect NVIDIA GPUs in a Python environment using WhichLLM:

from whichllm.hardware.nvidia import detect_nvidia_gpus

# Detect NVIDIA GPUs on the current machine

gpus = detect_nvidia_gpus()

for gpu in gpus:
    print(f"Name: {gpu.name}")
    print(f"VRAM: {gpu.vram_bytes / (1024**3):.1f} GiB")
    print(f"CUDA version: {gpu.cuda_version}")
    print(f"Compute capability: {gpu.compute_capability}")
    print("-" * 30)

If pynvml is unavailable, the same function automatically switches to the nvidia-smi fallback without requiring any code changes or additional configuration.

Summary

• Primary library: pynvml provides direct NVML access for GPU enumeration, VRAM detection, and CUDA version queries in src/whichllm/hardware/nvidia.py
• Fallback mechanism: Python's built-in subprocess module executes nvidia-smi when NVML is unavailable (lines 71-80)
• Automatic selection: The detect_nvidia_gpus() function handles backend selection transparently via try/except blocks (lines 10-14 and 112-119)
• Data modeling: Hardware metadata is standardized through the GPUInfo class defined in src/whichllm/hardware/types.py

Frequently Asked Questions

What is the primary library for NVIDIA GPU detection in WhichLLM?

The primary library is pynvml, a Python wrapper for the NVIDIA Management Library (NVML). It provides direct access to GPU properties including device names, memory capacity, and driver versions through the NVIDIA driver interface.

How does WhichLLM handle environments where pynvml is not installed?

WhichLLM implements a graceful fallback using Python's subprocess module to execute the nvidia-smi command-line utility. This occurs automatically in the _detect_nvidia_gpus_via_smi function when the pynvml import fails or NVML initialization raises an exception.

Can WhichLLM detect NVIDIA GPUs without NVIDIA drivers installed?

No. Both detection methods require NVIDIA drivers to be present on the system. The pynvml library requires the NVML shared libraries included with the driver package, while the nvidia-smi fallback relies on the command-line utility that ships with NVIDIA drivers. Without drivers, neither path can enumerate hardware.

Where is the GPU detection logic implemented in the WhichLLM codebase?

The core detection logic resides in src/whichllm/hardware/nvidia.py, specifically in the detect_nvidia_gpus() function (primary NVML path, lines 122-148) and the _detect_nvidia_gpus_via_smi() helper (fallback path, lines 71-80). The data structures used to represent GPU information are defined in src/whichllm/hardware/types.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}