# Does Needle 2 Perform Network Requests During Inference? A Code-Level Analysis

> Discover if Needle 2 makes network requests during inference. Learn how it runs text generation locally on JAX tensors, ensuring all network calls are for setup only.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-09-11

---

**No. Needle 2 executes text generation entirely on local hardware using JAX tensors, with all network activity restricted to the one-time setup phase for downloading checkpoints and native binaries.**

Needle 2 is an open-source inference engine designed for high-performance local text generation. Developers evaluating the `cactus-compute/needle` repository often ask whether the library phones home or requires persistent connectivity during the actual inference phase. Examination of the source code confirms that the core generation pipeline operates strictly offline, with networking logic isolated to initial asset fetching.

## The Inference Pipeline: Zero Network Calls

The primary inference entry points in [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py) — **`generate`** and **`batch_generate`** — operate on local JAX buffers without ever invoking HTTP clients. When you call these functions, the execution flow follows this strictly local path:

- **Prompt encoding** happens via the local tokenizer.
- **Buffer allocation** creates JAX tensors in host or GPU memory.
- **JIT-compiled decoding** repeatedly calls `_get_decode_fn`, which executes **`model.apply`** for the forward pass.
- **Token selection** occurs deterministically or stochastically on local hardware.

There are no invocations of `urllib`, `requests`, `httpx`, or any other networking library inside these hot-path functions. The `decode_fn` compiled by JAX runs the model’s forward pass entirely within the local execution environment, making inference suitable for air-gapped deployments once assets are cached.

## Where Network Requests Actually Occur

Network activity in Needle 2 is strictly limited to three setup-phase operations outside the inference path. These functions in the codebase handle asset acquisition before local execution begins:

### Checkpoint Loading via `load_checkpoint`

In [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py), the **`load_checkpoint`** function checks for model weights on disk. If the requested checkpoint is not present locally, it calls **`hf_hub_download`** to pull the file from the Hugging Face Hub. This occurs once, typically during initialization:

```python
params, config = load_checkpoint("checkpoints/needle2_v1.ckpt")

# If missing locally, hf_hub_download triggers a network request

```

### Runtime Library Fetching

The **`fetch_library`** function in [`needle/agent/fetch.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/fetch.py) downloads the native Needle runtime (e.g., `libneedle.so`) from the Hugging Face Hub. This retrieves a platform-specific wheel and extracts the shared library required for optimized local execution.

### Platform-Specific Binary Downloads

Also in [`needle/agent/fetch.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/fetch.py), the **`download_platform`** function lists and copies platform-specific binaries from the Hub to the local cache. Like checkpoint loading, this is a one-time setup operation that populates local storage with required dependencies.

## Local Inference: Working Offline

Once checkpoints and binaries reside in local storage, you can disconnect from the network entirely. The following examples demonstrate pure offline execution using pre-downloaded assets:

```python
from needle.model.run import load_checkpoint, generate
from needle.model.architecture import SimpleAttentionNetwork
from needle.model.tokenizer import get_tokenizer

params, config = load_checkpoint("checkpoints/needle2_v1.ckpt")
model = SimpleAttentionNetwork(config)
tokenizer = get_tokenizer(config.vocab_size)

prompt = "Explain quantum entanglement in simple terms."
output = generate(model, params, tokenizer, prompt, max_new_tokens=150)
print(output)

```

Batch processing follows the same local-only pattern:

```python
from needle.model.run import load_checkpoint, batch_generate

params, config = load_checkpoint("checkpoints/needle2_v1.ckpt")
model = SimpleAttentionNetwork(config)
tokenizer = get_tokenizer(config.vocab_size)

prompts = [
    "Write a haiku about sunrise.",
    "Summarize the plot of *The Matrix*."
]
results = batch_generate(model, params, tokenizer, prompts, max_new_tokens=50)

for r in results:
    print(r["text"])

```

Both `generate` and `batch_generate` execute the **`model.apply`** function through JIT-compiled JAX operations, never reaching out to external endpoints during the token-by-token generation loop.

## Verifying the Absence of Network Dependencies

A review of the dependency imports and call graphs in the inference functions confirms the offline capability:

- **[`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py)** contains no HTTP client imports in the generation logic.
- The **`_get_decode_fn`** helper builds a pure JAX computation graph.
- Token selection logic uses local random number generation or greedy argmax operations.

Only **[`needle/agent/fetch.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/fetch.py)** and the initialization block of **`load_checkpoint`** in [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py) reference networking code, and these execute before the inference loop begins.

## Summary

- Needle 2 performs **zero network requests** during the actual text generation phase.
- The **`generate`** and **`batch_generate`** functions operate exclusively on local JAX tensors and JIT-compiled model weights.
- Network activity occurs only during **initial setup**: downloading checkpoints via `load_checkpoint`, fetching the runtime library via `fetch_library`, or acquiring platform binaries via `download_platform`.
- Once assets are cached locally, the inference engine runs entirely offline with no calls to `urllib`, `requests`, or `httpx`.

## Frequently Asked Questions

### Does Needle 2 require an internet connection to generate text?

No. After the initial download of model checkpoints and native libraries, Needle 2 runs completely offline. The `generate` and `batch_generate` functions in [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py) use only local JAX operations and never invoke network APIs during token generation.

### What happens if I try to load a checkpoint without internet access?

If the checkpoint file already exists in your local cache or specified path, `load_checkpoint` loads it from disk without any network activity. Only if the file is missing does the function attempt to call `hf_hub_download`, which would fail without connectivity.

### Which source files contain the networking code?

Network requests are isolated to **[`needle/agent/fetch.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/fetch.py)** (containing `fetch_library` and `download_platform`) and the checkpoint initialization logic in **[`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py)**. The core inference implementations in these files contain no HTTP client code.

### Is batch generation also network-free?

Yes. The **`batch_generate`** function uses the same local execution pipeline as single-sequence generation. It processes prompts through local tokenizers and executes the model via JIT-compiled JAX operations without external network dependencies.