Needle JAX Backends: CUDA and Metal Support with Installation Guide

Needle supports three JAX execution platforms: CUDA for NVIDIA GPUs, Metal for Apple Silicon, and CPU fallback, installed via the gpu or metal extras in pip install "cactus-needle[gpu]" or pip install "cactus-needle[metal]".

Needle is an open-source inference and fine-tuning framework built on top of JAX. Because the library delegates all tensor operations to the JAX runtime, it automatically inherits support for hardware-specific accelerators, but you must install the correct optional dependencies to enable GPU execution on your platform.

Supported JAX Backends for Needle

Needle detects available hardware through JAX’s device API and will automatically utilize an accelerator if the appropriate backend is installed. The repository explicitly configures three execution paths:

CUDA (NVIDIA GPUs)

The CUDA backend enables execution on NVIDIA GPUs across Linux, macOS, and Windows. When installed, Needle’s training and inference code automatically targets the GPU device (jax.default_backend() == "gpu"). According to the source, the needle/model/run.py file contains the core inference loop that moves parameters to the active device using jax.device_put and compiles operations with @jax.jit, ensuring CUDA kernels are utilized when available.

Metal (Apple Silicon)

The Metal backend provides acceleration on Apple Silicon GPUs (M1/M2/M3). This requires the jax-metal plugin. As implemented in needle/model/finetune.py at lines 9–11, Needle forces the Metal plugin to import before any standard JAX import, ensuring the Metal backend registers successfully. This backend reports as "gpu" when calling jax.default_backend() on macOS.

CPU (Fallback)

CPU execution requires no additional installation. The base JAX wheels include a CPU-only runtime that serves as the default when no GPU or Metal backend is detected. Needle will transparently fall back to CPU execution if accelerator libraries are missing.

Installation Instructions

You install accelerator support through Python extras defined in the package configuration. The repository documentation in README.md (lines 106–116) defines the specific commands for each platform.

Installing CUDA Support

To enable NVIDIA GPU acceleration, install Needle with the gpu extra:

pip install "cactus-needle[gpu]"

This command pulls a CUDA-enabled JAX build that links against the NVIDIA CUDA toolkit. Your system must already have compatible NVIDIA drivers installed. Once complete, any call to jax.jit or jax.device_put in Needle’s inference pipeline will automatically execute on the GPU.

Installing Metal Support

For Apple Silicon Macs, install the metal extra:

pip install "cactus-needle[metal]"

This extra pins JAX to version 0.4.38 or earlier and installs the compatible jax-metal plugin. The plugin stopped functioning in later JAX versions, so the dependency constraint is enforced to maintain compatibility. The needle/model/finetune.py module handles the import sequencing to ensure the Metal backend initializes before JAX loads the CPU runtime.

Verifying Your Backend

After installation, verify which accelerator Needle is using:

import needle
import jax

# Initialize agent (weights auto-download)

agent = needle.Needle()

# Check active backend

print("Active JAX backend:", jax.default_backend())  # "cpu", "gpu", or "metal"

How Needle Configures JAX Backends

Needle does not manually select devices in user-facing code; instead, it relies on JAX’s default device placement. Two critical source files manage this behavior:

• needle/model/run.py: Contains the inference loop that places tensors on the default device using jax.device_put and wraps compute-intensive functions with @jax.jit. When a CUDA or Metal backend is active, these calls trigger compiled kernels on the accelerator.
• needle/model/finetune.py: Lines 9–11 explicitly import the Metal plugin before importing JAX. This sequencing ensures that on Apple Silicon machines, the Metal backend registers as the default GPU device rather than falling back to CPU.

Forcing a Specific Backend

If you need to override automatic device selection (for example, to isolate a specific GPU on a multi-GPU machine), set environment variables before importing JAX:

import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"  # Use first GPU only

os.environ["XLA_FLAGS"] = "--xla_gpu_force_compilation_parallelism=1"

# Reimport JAX to apply changes

import jax
print("Forced backend:", jax.default_backend())

On macOS, the Metal backend is selected automatically when the jax-metal plugin is present; no additional environment variables are required unless you wish to force CPU-only mode by uninstalling the plugin.

Summary

• Needle runs on JAX and supports CUDA, Metal, and CPU backends.
• Install CUDA support with pip install "cactus-needle[gpu]" (requires NVIDIA drivers).
• Install Metal support with pip install "cactus-needle[metal]" (pins JAX ≤0.4.38 for compatibility).
• The needle/model/run.py file handles device placement via jax.device_put and @jax.jit.
• The needle/model/finetune.py file ensures Metal plugin initialization on Apple Silicon.
• Verify your backend anytime using jax.default_backend().

Frequently Asked Questions

Why does Metal support require an older JAX version?

The jax-metal plugin for Apple Silicon stopped functioning after JAX version 0.4.38. To maintain compatibility, the metal extra pins the JAX dependency to this version or earlier, as noted in the source comments of needle/model/finetune.py and the documentation in doc/finetuning.md.

Can I use Needle with AMD GPUs?

No. Needle relies on JAX, which currently only supports NVIDIA GPUs via CUDA and Apple GPUs via Metal. AMD ROCm support is not available in the JAX wheels distributed with Needle’s extras.

How do I confirm Needle is utilizing my GPU and not falling back to CPU?

After initializing a needle.Needle() agent, check jax.default_backend(). If it returns "gpu", your model weights and compute graphs are being placed on the accelerator. If it returns "cpu", verify your installation command included the correct extra ([gpu] or [metal]) and that your drivers are properly installed.

What happens if I install both CUDA and Metal extras on the same machine?

You should only install the extra matching your hardware. Installing both on a single machine (e.g., a Linux server with NVIDIA GPUs) will not break Needle, but the Metal dependencies are unnecessary and may conflict with your CUDA environment. Install only cactus-needle[gpu] for NVIDIA systems and only cactus-needle[metal] for Apple Silicon Macs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}