How to Set Up Multi-Slot GPU Access with SlotWorker in Forge

Create independent SlotWorker instances for each GPU by setting CUDA_VISIBLE_DEVICES before client instantiation, then run them concurrently to achieve parallel workflow execution across multiple devices.

The antoinezambelli/forge repository provides a SlotWorker class that serializes workflow execution on inference slots, but scaling across multiple GPUs requires specific environment variable isolation. Since the underlying LLM client selects GPU devices at import-time, you must bind each worker to a specific device before the client loads. This approach leverages the WorkflowRunner and ContextManager classes to create isolated inference pipelines per GPU.

Understanding the Architecture

The SlotWorker Abstraction

SlotWorker (defined in src/forge/core/slot_worker.py) is a thin wrapper around a WorkflowRunner that guarantees serialized execution of workflows on a single inference slot. It manages an internal priority queue and handles preemption via async tasks. Importantly, SlotWorker does not perform GPU allocation itself—it delegates all inference operations to the runner's client.

GPU Binding Mechanics

In src/forge/core/runner.py, the WorkflowRunner instantiates the LLM client (e.g., OllamaClient, Anthropic client, or LLaMA-File wrapper). These clients typically use torch or llama_cpp under the hood, which detect CUDA devices when the library first imports. The HardwareProfile class in src/forge/context/hardware.py parses nvidia-smi to identify available GPUs, but the actual device binding occurs through CUDA_VISIBLE_DEVICES. Because GPU selection happens at import-time, you must set this environment variable before creating the client instance.

Single-Process Multi-GPU Setup

For applications that manage multiple GPUs within one Python process, instantiate separate ContextManager, client, and runner objects for each device:

import os
import asyncio

from forge.core.slot_worker import SlotWorker
from forge.core.runner import WorkflowRunner
from forge.context.manager import ContextManager
from forge.context.strategies import NoCompact
from forge.clients.ollama import OllamaClient

# Configure GPU 0

os.environ["CUDA_VISIBLE_DEVICES"] = "0"
ctx0 = ContextManager(strategy=NoCompact())
client0 = OllamaClient()  # Binds to GPU 0

runner0 = WorkflowRunner(client=client0, context_manager=ctx0)
worker0 = SlotWorker(runner0)

# Configure GPU 1

os.environ["CUDA_VISIBLE_DEVICES"] = "1"
ctx1 = ContextManager(strategy=NoCompact())
client1 = OllamaClient()  # Binds to GPU 1

runner1 = WorkflowRunner(client=client1, context_manager=ctx1)
worker1 = SlotWorker(runner1)

async def main():
    # Start both workers (each launches an internal asyncio loop)

    await asyncio.gather(worker0.start(), worker1.start())
    
    # Submit workflows to specific GPUs

    result0 = await worker0.submit(workflow_instance, "task for GPU 0")
    result1 = await worker1.submit(workflow_instance, "task for GPU 1")
    
    # Cleanup

    await asyncio.gather(worker0.stop(), worker1.stop())

asyncio.run(main())

The CUDA_VISIBLE_DEVICES variable restricts the Python process's GPU visibility before each client instantiation, ensuring client0 sees only GPU 0 and client1 sees only GPU 1.

Multi-Process Isolation for Production

For hard isolation between GPUs—preventing memory leaks or driver conflicts—launch separate OS processes with distinct environment variables:

#!/usr/bin/env bash

# launch_worker.sh

GPU_ID=$1
shift
export CUDA_VISIBLE_DEVICES=$GPU_ID
python -m forge.scripts.run_worker "$@"

# run_worker.py

import asyncio
from forge.core.slot_worker import SlotWorker
from forge.core.runner import WorkflowRunner
from forge.context.manager import ContextManager
from forge.context.strategies import NoCompact
from forge.clients.ollama import OllamaClient

async def start_worker():
    # Environment already set by wrapper script

    ctx = ContextManager(strategy=NoCompact())
    client = OllamaClient()
    runner = WorkflowRunner(client=client, context_manager=ctx)
    worker = SlotWorker(runner)
    await worker.start()
    
    try:
        await asyncio.Future()  # Run forever

    finally:
        await worker.stop()

if __name__ == "__main__":
    asyncio.run(start_worker())

Start workers on different GPUs:

./launch_worker.sh 0 &
./launch_worker.sh 1 &

This pattern ensures complete process isolation, with each worker binding exclusively to its assigned GPU through the inherited environment variable.

Key Source Files

• src/forge/core/slot_worker.py – Defines the SlotWorker class with priority queue management and workflow serialization logic.
• src/forge/core/runner.py – Contains WorkflowRunner, which executes workflows using the LLM client and receives cancellation events from the worker.
• src/forge/context/hardware.py – Implements HardwareProfile for GPU detection via nvidia-smi parsing and device enumeration.
• src/forge/core/workflow.py – Defines the Workflow model (tools, steps, terminal tool specifications) that workers execute.

Summary

• Set CUDA_VISIBLE_DEVICES before client instantiation to bind each worker to a specific GPU at the driver level.
• Create one SlotWorker per GPU, each wrapping a dedicated WorkflowRunner with its own ContextManager.
• Start workers concurrently using asyncio.gather() or separate OS processes to achieve true parallel execution across devices.
• Submit workflows to specific workers based on your load balancing strategy; each worker serializes tasks on its assigned GPU while tasks across workers run in parallel.

Frequently Asked Questions

How does SlotWorker handle GPU allocation?

SlotWorker does not allocate GPUs directly. It wraps a WorkflowRunner that uses an LLM client (e.g., OllamaClient), and the client binds to a GPU when instantiated based on the current CUDA_VISIBLE_DEVICES environment variable. The worker only manages the execution queue for its assigned runner.

Can I change GPU assignments after starting a SlotWorker?

No. GPU binding occurs when the underlying client library (such as torch or llama_cpp) loads, which typically happens during WorkflowRunner or client instantiation. To use a different GPU, you must create a new worker instance with the appropriate environment variable set before client creation.

What is the difference between single-process and multi-process setups?

A single-process setup creates multiple SlotWorker instances in one Python process with different CUDA_VISIBLE_DEVICES values set sequentially before each client instantiation. A multi-process setup launches separate Python processes, each with a fixed CUDA_VISIBLE_DEVICES value, providing hard isolation that prevents driver state pollution and memory fragmentation between GPUs.

How does the HardwareProfile interact with GPU selection?

The HardwareProfile class in src/forge/context/hardware.py detects available GPUs by parsing nvidia-smi output, but it only provides information. The actual GPU binding occurs through environment variables that restrict device visibility before the client library imports. You can use HardwareProfile to validate available devices before assigning workers, but the runtime binding depends on CUDA_VISIBLE_DEVICES.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}