# How to Configure WorkerPool for GPU Passthrough in Agent Substrate

> Learn to configure WorkerPool for GPU passthrough in Agent Substrate. Set nvidia.com/gpu in resourceRequests, ensure the NVIDIA device plugin is running, and optionally set runtimeClassName: nvidia for direct CUDA access.

- Repository: [Agent Substrate/substrate](https://github.com/agent-substrate/substrate)
- Tags: how-to-guide
- Published: 2026-08-22

---

**Configure GPU passthrough by defining `nvidia.com/gpu` in the WorkerPool `resourceRequests` field, ensuring underlying nodes run the NVIDIA device plugin, and optionally setting `runtimeClassName: nvidia` for direct CUDA access.**

Agent Substrate manages ephemeral workloads through **WorkerPools**, custom Kubernetes resources defined in the `agent-substrate/substrate` repository that map short-lived actors to ready Pods. To expose physical GPU devices to workers, you configure the WorkerPool specification to request GPU resources while ensuring the node supplies the corresponding device capacity.

## Prerequisites for GPU Node Configuration

Before configuring the WorkerPool, prepare your cluster nodes to expose GPU devices.

- **Install NVIDIA drivers** on the host operating system to enable hardware access.
- **Deploy the NVIDIA GPU device plugin** as a DaemonSet. The official Helm chart (`nvidia-device-plugin`) registers the `nvidia.com/gpu` resource with the kubelet.
- **Verify node capacity** demonstrates GPU availability:

```bash
kubectl describe node <gpu-node-name> | grep nvidia.com/gpu

```

The output must show allocatable GPU resources before the WorkerPool can schedule GPU-bound workers.

## Defining GPU Resources in the WorkerPool Spec

The WorkerPool custom resource definition (CRD) accepts a `resourceRequests` map that translates to container resource requirements for every worker spawned from the pool. As defined in the API types, this field allows you to request specific hardware devices like GPUs.

### YAML Configuration for GPU Passthrough

Create a WorkerPool that binds to GPU-equipped nodes and requests one GPU per worker:

```yaml
apiVersion: substrate.agent-substrate.github.com/v1alpha1
kind: WorkerPool
metadata:
  name: gpu-pool
  namespace: default
spec:
  nodeSelector:
    accelerator: nvidia-gpu
  resourceRequests:
    nvidia.com/gpu: "1"
  maxWorkers: 10

```

**Key configuration details:**
- **nodeSelector**: Targets nodes labeled with GPU hardware (e.g., `accelerator: nvidia-gpu`).
- **resourceRequests**: Requests the `nvidia.com/gpu` resource; the value represents the count of GPUs per worker.
- **maxWorkers**: Limits concurrent workers to prevent GPU oversubscription (unless using MPS).

### Programmatic Configuration Using the Go Client

For dynamic provisioning, use the generated typed client defined in [`pkg/client/clientset/versioned/typed/api/v1alpha1/workerpool.go`](https://github.com/agent-substrate/substrate/blob/main/pkg/client/clientset/versioned/typed/api/v1alpha1/workerpool.go):

```go
import (
    "context"
    apiv1alpha1 "github.com/agent-substrate/substrate/pkg/apis/api/v1alpha1"
    clientset "github.com/agent-substrate/substrate/pkg/client/clientset/versioned"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

func createGPUWorkerPool(cs clientset.Interface) error {
    pool := &apiv1alpha1.WorkerPool{
        ObjectMeta: metav1.ObjectMeta{
            Name:      "gpu-pool",
            Namespace: "default",
        },
        Spec: apiv1alpha1.WorkerPoolSpec{
            NodeSelector: map[string]string{
                "accelerator": "nvidia-gpu",
            },
            ResourceRequests: map[string]string{
                "nvidia.com/gpu": "1",
            },
            MaxWorkers: 10,
        },
    }
    _, err := cs.ApiV1alpha1().WorkerPools("default").Create(context.Background(), pool, metav1.CreateOptions{})
    return err
}

```

The client implementation handles serialization of the `WorkerPool` spec according to the protobuf definitions found in [`pkg/proto/ateapipb/ateapi.pb.go`](https://github.com/agent-substrate/substrate/blob/main/pkg/proto/ateapipb/ateapi.pb.go), where the `WorkerPool` field tracks the pool assignment for individual workers.

## Advanced GPU Passthrough Options

For workloads requiring direct CUDA access or shared GPU contexts, configure additional runtime parameters.

- **NVIDIA Container Runtime**: Set `runtimeClassName: nvidia` in the WorkerPool spec (if your cluster supports the `nvidia` runtime class) to ensure the container uses `nvidia-container-runtime` instead of the default runc.
- **Multi-Process Service (MPS)**: For time-sharing GPUs across multiple workers, reference the `GPUSharingConfig` in the NodeConfig (as hinted in the protobuf definitions) to enable MPS strategy instead of exclusive GPU assignment.
- **Network Annotations**: If using SR-IOV or specific GPU network overlays, add the annotation `k8s.v1.cni.cncf.io/networks: gpu-net` to the WorkerPool metadata to attach specialized network interfaces alongside the GPU device.

## Validation and Testing

Deploy the configuration and verify GPU allocation:

```bash
kubectl apply -f gpu-pool.yaml

```

Inspect running workers to confirm GPU device injection:

```bash

# List workers belonging to the pool

kubectl get workers -n default -l substrate.io/worker-pool=gpu-pool

# Verify nvidia-smi inside a worker pod

kubectl exec -it <worker-pod-name> -- nvidia-smi

```

Successful output from `nvidia-smi` inside the container confirms that the WorkerPool configuration correctly passed through the GPU device from the node to the worker runtime.

## Key Source Files

Understanding these implementation files helps when debugging GPU scheduling issues or extending the API:

- **[`pkg/proto/ateapipb/ateapi.pb.go`](https://github.com/agent-substrate/substrate/blob/main/pkg/proto/ateapipb/ateapi.pb.go)**: Contains the generated protobuf definitions including the `WorkerPool` field used to associate workers with specific pools.
- **[`pkg/client/clientset/versioned/typed/api/v1alpha1/workerpool.go`](https://github.com/agent-substrate/substrate/blob/main/pkg/client/clientset/versioned/typed/api/v1alpha1/workerpool.go)**: Implements the typed client interface for creating, updating, and deleting WorkerPool resources programmatically.
- **[`pkg/client/listers/api/v1alpha1/workerpool.go`](https://github.com/agent-substrate/substrate/blob/main/pkg/client/listers/api/v1alpha1/workerpool.go)**: Provides lister functionality for cached read operations on WorkerPool objects.
- **[`pkg/client/informers/externalversions/api/v1alpha1/workerpool.go`](https://github.com/agent-substrate/substrate/blob/main/pkg/client/informers/externalversions/api/v1alpha1/workerpool.go)**: Defines the informer that watches WorkerPool CRD changes and maintains synchronization with the Kubernetes API server.

## Summary

- **WorkerPools** in Agent Substrate require the `resourceRequests` field to specify `nvidia.com/gpu` for GPU passthrough.
- Nodes must run NVIDIA drivers and the **NVIDIA device plugin** to advertise GPU capacity to the cluster.
- Use **`runtimeClassName: nvidia`** for CUDA-compatible container runtimes.
- The **generated Go client** in `pkg/client/clientset/versioned` supports programmatic WorkerPool creation with GPU resource constraints.
- Validate configurations by executing `nvidia-smi` inside running worker pods to confirm device visibility.

## Frequently Asked Questions

### How do I enable GPU sharing for multiple concurrent workers?

Configure the **MPS (Multi-Process Service)** strategy in the NodeConfig or WorkerPool annotations. MPS allows multiple CUDA processes to share a single GPU context, though you must ensure the `maxWorkers` value does not exceed the GPU's memory and compute capacity. Without MPS, each worker typically requires a dedicated GPU to avoid resource conflicts.

### What runtime class should I specify for NVIDIA GPU containers?

Set **`runtimeClassName: nvidia`** in the WorkerPool specification if your cluster has registered the `nvidia` runtime class. This ensures the containerd or CRI-O runtime invokes `nvidia-container-runtime`, which configures the GPU device hooks and library mounts required for CUDA applications. If using standard containerd without the NVIDIA runtime class, the GPU device plugin alone handles device injection via the Device Plugin API.

### How do I verify that a worker actually received the GPU device?

Execute `kubectl exec -it <worker-pod-name> -- nvidia-smi` to check if the NVIDIA System Management Interface detects the hardware. If the command returns GPU details (name, temperature, memory usage), the passthrough succeeded. If the command fails with "command not found" or "no devices found," verify that the node reports `nvidia.com/gpu` capacity and that the WorkerPool `nodeSelector` targets the correct GPU-equipped nodes.

### Can I use AMD or Intel GPUs with Agent Substrate WorkerPools?

Yes, provided the respective **device plugins** (AMD GPU device plugin or Intel GPU plugin) are installed on the nodes and advertise resources (e.g., `amd.com/gpu` or `gpu.intel.com/i915`). Replace `nvidia.com/gpu` with the appropriate vendor resource name in the WorkerPool `resourceRequests` field. Ensure the container image includes the necessary vendor-specific drivers and runtime libraries.