# Best Practices for Using Needle: A Complete Guide to Tool Calling, Extraction, and Fine-Tuning

> Master Needle's tool calling, data extraction, and fine-tuning with best practices. Optimize your edge device AI with this 45M parameter model and unified Python interface.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: best-practices
- Published: 2026-08-27

---

**Needle is a 45M-parameter self-contained model that provides a unified Python interface for tool calling, structured data extraction, and LoRA fine-tuning on edge devices.**

If you're building AI-powered applications that need to run offline or on resource-constrained hardware, Needle offers a compelling alternative to cloud-dependent APIs. This guide covers the essential best practices for using Needle effectively, drawn directly from the cactus-compute/needle source code.

---

## Understand Needle's Architecture

Before diving into code, you should understand how Needle achieves its tiny footprint. The model is built around a **Simple Attention Network** implemented in [[`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py). Key architectural decisions include:

- **Hadamard-based MLP** for efficient feedforward computations
- **GQA (Grouped Query Attention)** for memory-efficient attention
- **Byte-level grammar compilation** from Pydantic schemas, ensuring syntactically valid JSON output

The entire model ships as a single 14MB binary (`.cact` file) managed by the engine in [[`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py)](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py). This engine is downloaded once and cached locally for all subsequent inferences.

---

## Follow the Tool Contract

The most important best practice for using Needle is **precise tool definitions**. Needle uses Python type hints and docstrings to construct tool schemas automatically.

### Write Clear Tool Functions

```python
import needle

@needle.tool
def get_weather(city: str):
    """Get the current weather for a city."""
    # Real implementation would call an API; here we mock it.

    return {"city": city, "temp_c": 27, "sky": "clear"}

agent = needle.Needle(tools=[get_weather])
response = agent.run("What's the weather like in Lagos right now?")
print(response["results"])

# [{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]

```

**Key requirements:**
- Use explicit **type hints** for all arguments—these become the JSON schema
- Write **concise, descriptive docstrings**—the model uses these to decide when to invoke the tool
- Keep the **tool catalog focused**—while Needle has a retrieval head for large catalogs, it performs best with ≤20 tools

---

## Leverage Grammar-Based Structured Extraction

One of Needle's standout features is **guaranteed-valid JSON output**. When you pass a Pydantic model to `needle.extract`, the decoder in [[`needle/model/decode.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/decode.py)](https://github.com/cactus-compute/needle/blob/main/needle/model/decode.py) compiles a byte-level grammar that restricts token generation token-by-token.

```python
from pydantic import BaseModel
import needle

class Invoice(BaseModel):
    vendor: str
    total: float
    due_date: str

text = "Invoice from Acme Corp, $1,200.00, due 2026-09-01"
invoice = needle.extract(text, Invoice)
print(invoice.vendor, invoice.total)   # → Acme Corp 1200.0

```

This approach eliminates post-processing and validation errors common with generic LLM outputs.

---

## Use Confidence Gating for Production Reliability

Every response from Needle includes a calibrated **confidence score**. The inference loop in [[`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py)](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py) exposes this for threshold-based decision making:

```python
response = agent.run("Dim the kitchen lights to 10%")

if response["confidence"] > 0.85:
    execute_action(response["results"])
else:
    escalate_to_human_review(response)

```

Set thresholds based on your application's risk tolerance. Lower thresholds for exploratory tasks, higher thresholds for consequential actions.

---

## Fine-Tune Only When Necessary

The base 45M model handles most tool-calling scenarios. However, if you need higher accuracy on a specialized tool set, Needle supports **LoRA fine-tuning** with a reproducible CLI workflow:

### 1. Generate Synthetic Training Data (Optional)

```bash
export OPENROUTER_API_KEY=sk-...
needle generate-data --tools my_tools.json --num-samples 500 --output data.jsonl

```

### 2. Fine-Tune with LoRA

```bash
needle finetune data.jsonl --epochs 10 --lora-rank 16 --lora-alpha 32

```

### 3. Export a Merged Model

```bash
needle build checkpoints/needle2.pkl --lora checkpoints/needle_lora.pkl --out my_needle.cact

```

### 4. Load Your Tuned Model

```python
import needle

agent = needle.Needle(weights="my_needle.cact", tools=[get_weather])
print(agent.run("Dim the kitchen lights to 10%"))

```

**Best practice:** Start with the base model, measure performance on your target tasks, and fine-tune only if accuracy gaps exist.

---

## Cache the Engine for Offline Deployment

The Needle engine downloads from Hugging Face on first use. For production deployments:

- Include the `.cact` file in your deployment artifacts
- Set appropriate cache directories if running in containerized environments
- Verify the 28MB RAM footprint stays constant regardless of conversation length, thanks to the 256-token sliding window with KV-sinks

---

## Prefer the CLI for Reproducibility

Needle's command-line interface in [[`needle/cli.py`](https://github.com/cactus-compute/needle/blob/main/needle/cli.py)](https://github.com/cactus-compute/needle/blob/main/needle/cli.py) encapsulates the full pipeline:

| Command | Purpose |
|---------|---------|
| `needle build` | Merge base weights with LoRA adapters |
| `needle finetune` | Train LoRA adapters on your data |
| `needle generate-data` | Create synthetic training examples |
| `needle playground` | Interactive testing environment |

Using these commands ensures your workflows are version-controllable and shareable.

---

## Summary

- **Describe tools precisely** with type hints and concise docstrings
- **Use Pydantic schemas** for guaranteed-valid structured extraction
- **Start with the base model** and fine-tune only when accuracy requires it
- **Cache `.cact` files** locally for offline, self-contained deployments
- **Apply confidence thresholds** to filter low-certainty responses
- **Keep tool catalogs focused** for optimal retrieval performance
- **Use CLI commands** to ensure reproducible workflows

---

## Frequently Asked Questions

### How large is the Needle model and what are its system requirements?

Needle is a 45M-parameter model that ships as a 14MB binary. At runtime, it uses approximately 28MB of RAM regardless of conversation length, thanks to bounded memory techniques including a 256-token sliding window with KV-sinks. This makes it suitable for edge devices and offline deployments.

### Does Needle support function calling with multiple tools?

Yes. Needle can handle multiple tools through the `@needle.tool` decorator and the `Needle(tools=[...])` interface. For large tool catalogs, the retrieval head in [[`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py)](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py) automatically selects the top 5 candidate tools per turn to maintain fast inference, though best practice is to keep catalogs at 20 or fewer tools.

### How does Needle ensure JSON output is always valid?

Needle compiles a byte-level grammar from your Pydantic schema in [[`needle/model/decode.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/decode.py)](https://github.com/cactus-compute/needle/blob/main/needle/model/decode.py). This grammar constrains the decoder to generate only syntactically valid JSON tokens, eliminating the need for post-hoc parsing or retry logic common with unconstrained language models.

### Can I use Needle without an internet connection?

Yes, after the initial engine download. Best practice is to cache the `.cact` file locally and include it in deployment artifacts. Once cached, Needle operates entirely offline with no API calls or network dependencies.