# Needle JAX Backends: CUDA and Metal Support with Installation Guide

> Explore Needle's JAX backends, including CUDA and Metal support for NVIDIA GPUs and Apple Silicon. Get a step-by-step installation guide to accelerate your JAX computations.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: how-to-guide
- Published: 2026-08-18

---

**Needle supports three JAX execution platforms: CUDA for NVIDIA GPUs, Metal for Apple Silicon, and CPU fallback, installed via the `gpu` or `metal` extras in `pip install "cactus-needle[gpu]"` or `pip install "cactus-needle[metal]"`.**

Needle is an open-source inference and fine-tuning framework built on top of JAX. Because the library delegates all tensor operations to the JAX runtime, it automatically inherits support for hardware-specific accelerators, but you must install the correct optional dependencies to enable GPU execution on your platform.

## Supported JAX Backends for Needle

Needle detects available hardware through JAX’s device API and will automatically utilize an accelerator if the appropriate backend is installed. The repository explicitly configures three execution paths:

### CUDA (NVIDIA GPUs)

The **CUDA** backend enables execution on NVIDIA GPUs across Linux, macOS, and Windows. When installed, Needle’s training and inference code automatically targets the GPU device (`jax.default_backend() == "gpu"`). According to the source, the [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py) file contains the core inference loop that moves parameters to the active device using `jax.device_put` and compiles operations with `@jax.jit`, ensuring CUDA kernels are utilized when available.

### Metal (Apple Silicon)

The **Metal** backend provides acceleration on Apple Silicon GPUs (M1/M2/M3). This requires the *jax-metal* plugin. As implemented in [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py) at lines 9–11, Needle forces the Metal plugin to import before any standard JAX import, ensuring the Metal backend registers successfully. This backend reports as `"gpu"` when calling `jax.default_backend()` on macOS.

### CPU (Fallback)

**CPU** execution requires no additional installation. The base JAX wheels include a CPU-only runtime that serves as the default when no GPU or Metal backend is detected. Needle will transparently fall back to CPU execution if accelerator libraries are missing.

## Installation Instructions

You install accelerator support through Python extras defined in the package configuration. The repository documentation in [`README.md`](https://github.com/cactus-compute/needle/blob/main/README.md) (lines 106–116) defines the specific commands for each platform.

### Installing CUDA Support

To enable NVIDIA GPU acceleration, install Needle with the `gpu` extra:

```bash
pip install "cactus-needle[gpu]"

```

This command pulls a CUDA-enabled JAX build that links against the NVIDIA CUDA toolkit. Your system must already have compatible NVIDIA drivers installed. Once complete, any call to `jax.jit` or `jax.device_put` in Needle’s inference pipeline will automatically execute on the GPU.

### Installing Metal Support

For Apple Silicon Macs, install the `metal` extra:

```bash
pip install "cactus-needle[metal]"

```

This extra pins JAX to version 0.4.38 or earlier and installs the compatible *jax-metal* plugin. The plugin stopped functioning in later JAX versions, so the dependency constraint is enforced to maintain compatibility. The [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py) module handles the import sequencing to ensure the Metal backend initializes before JAX loads the CPU runtime.

### Verifying Your Backend

After installation, verify which accelerator Needle is using:

```python
import needle
import jax

# Initialize agent (weights auto-download)

agent = needle.Needle()

# Check active backend

print("Active JAX backend:", jax.default_backend())  # "cpu", "gpu", or "metal"

```

## How Needle Configures JAX Backends

Needle does not manually select devices in user-facing code; instead, it relies on JAX’s default device placement. Two critical source files manage this behavior:

- **[`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py)**: Contains the inference loop that places tensors on the default device using `jax.device_put` and wraps compute-intensive functions with `@jax.jit`. When a CUDA or Metal backend is active, these calls trigger compiled kernels on the accelerator.
- **[`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py)**: Lines 9–11 explicitly import the Metal plugin before importing JAX. This sequencing ensures that on Apple Silicon machines, the Metal backend registers as the default GPU device rather than falling back to CPU.

## Forcing a Specific Backend

If you need to override automatic device selection (for example, to isolate a specific GPU on a multi-GPU machine), set environment variables before importing JAX:

```python
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"  # Use first GPU only

os.environ["XLA_FLAGS"] = "--xla_gpu_force_compilation_parallelism=1"

# Reimport JAX to apply changes

import jax
print("Forced backend:", jax.default_backend())

```

On macOS, the Metal backend is selected automatically when the `jax-metal` plugin is present; no additional environment variables are required unless you wish to force CPU-only mode by uninstalling the plugin.

## Summary

- **Needle** runs on JAX and supports **CUDA**, **Metal**, and **CPU** backends.
- Install CUDA support with **`pip install "cactus-needle[gpu]"`** (requires NVIDIA drivers).
- Install Metal support with **`pip install "cactus-needle[metal]"`** (pins JAX ≤0.4.38 for compatibility).
- The **[`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py)** file handles device placement via `jax.device_put` and `@jax.jit`.
- The **[`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py)** file ensures Metal plugin initialization on Apple Silicon.
- Verify your backend anytime using **`jax.default_backend()`**.

## Frequently Asked Questions

### Why does Metal support require an older JAX version?

The **jax-metal** plugin for Apple Silicon stopped functioning after JAX version 0.4.38. To maintain compatibility, the `metal` extra pins the JAX dependency to this version or earlier, as noted in the source comments of [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py) and the documentation in [`doc/finetuning.md`](https://github.com/cactus-compute/needle/blob/main/doc/finetuning.md).

### Can I use Needle with AMD GPUs?

No. Needle relies on JAX, which currently only supports NVIDIA GPUs via CUDA and Apple GPUs via Metal. AMD ROCm support is not available in the JAX wheels distributed with Needle’s extras.

### How do I confirm Needle is utilizing my GPU and not falling back to CPU?

After initializing a `needle.Needle()` agent, check `jax.default_backend()`. If it returns `"gpu"`, your model weights and compute graphs are being placed on the accelerator. If it returns `"cpu"`, verify your installation command included the correct extra (`[gpu]` or `[metal]`) and that your drivers are properly installed.

### What happens if I install both CUDA and Metal extras on the same machine?

You should only install the extra matching your hardware. Installing both on a single machine (e.g., a Linux server with NVIDIA GPUs) will not break Needle, but the Metal dependencies are unnecessary and may conflict with your CUDA environment. Install only `cactus-needle[gpu]` for NVIDIA systems and only `cactus-needle[metal]` for Apple Silicon Macs.