# How Needle's `generate-data` Synthesizes Training Examples: Complete Technical Guide

> Learn how Needle's generate-data synthesizes training examples using OpenRouter LLMs to create tool-calling scenarios from JSON schemas. Explore deduplication and parallel processing.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-08-20

---

**Needle synthesizes training examples by using OpenRouter-hosted LLMs to generate tool-calling scenarios based on user-supplied JSON schemas, with built-in deduplication and parallel batch processing.**

The `needle generate-data` command is the entry point for creating synthetic datasets that teach language models when and how to invoke external tools. This article explains the complete synthesis pipeline implemented in the cactus-compute/needle repository, from prompt construction to final JSONL output.

## Overview of the Synthesis Pipeline

Data generation in Needle follows an eight-step workflow orchestrated through [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py). The process combines structured prompting, concurrent API calls, and content-aware deduplication to produce high-quality training examples at scale.

The core entry point is `generate_main` (lines 78-80), which delegates to `generate_dataset` for batch coordination and `generate_examples` for individual LLM interactions.

## Step 1: Tool Schema Preparation

Before generation begins, you must supply a JSON file describing the tools your model should learn to call. These schemas define function names, descriptions, and parameter specifications.

```bash
needle generate-data \
  --tools my_tools.json \
  --num-samples 500 \
  --output synthetic_data.jsonl

```

The CLI parser in [`needle/cli.py`](https://github.com/cactus-compute/needle/blob/main/needle/cli.py) (lines 152-199) validates these arguments and forwards them to `generate_main`, which loads the tool definitions and passes them to `generate_dataset`.

## Step 2: Batch Generation with Thread Pooling

The `generate_dataset` function uses a **ThreadPoolExecutor** to parallelize API requests. Each worker invokes `generate_examples` to produce a batch of *n* examples (default 25).

```python
from needle.model.finetune import generate_dataset
import json

with open("my_tools.json") as f:
    tools = json.load(f)

examples = generate_dataset(
    tools,
    num_samples=200,
    model="anthropic/claude-2.0",
    batch_size=25,
    workers=8,
)

```

The thread pool scheduler (lines 108-119) manages up to `workers` concurrent requests, maximizing throughput while respecting OpenRouter rate limits.

## Step 3: Prompt Construction for LLM Inference

Every batch request assembles two prompt components:

- **`_GEN_SYSTEM`** — A system prompt establishing the generation task and output format
- **`_GEN_TEMPLATE`** — A user prompt embedding the tool JSON, desired example count (`n`), and "refusals" count instructing the LLM to include scenarios where no tool should be called

These prompts are constructed in `generate_examples` (lines 85-88) and sent via `_openrouter` to the selected model. The default model is `openrouter.ai/anthropic/claude-2.0`, though any OpenRouter-hosted model can be specified.

## Step 4: LLM Inference via OpenRouter

The `_openrouter` helper (imported from [`needle/agent/fetch.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/fetch.py)) handles the HTTP POST to OpenRouter's chat completions endpoint. It transmits the combined system and user messages, returning the raw text response for parsing.

```python

# From needle/agent/fetch.py — simplified structure

def _openrouter(messages, model, temperature=0.7):
    response = requests.post(
        "https://openrouter.ai/api/v1/chat/completions",
        headers={"Authorization": f"Bearer {api_key}"},
        json={"model": model, "messages": messages, "temperature": temperature}
    )
    return response.json()["choices"][0]["message"]["content"]

```

## Step 5: Response Parsing and Validation

Responses are expected as **JSON-encoded arrays**. The `_parse_array` function extracts this array, and each row undergoes post-processing to guarantee the `"tools"` field is present (lines 89-91).

Partial or malformed responses are discarded, ensuring downstream fine-tuning receives valid training examples.

## Step 6: Deduplication by Content Hash

As batches complete, `generate_dataset` applies content-aware deduplication using `_dedup_key` (lines 94-97). This function:

1. Lower-cases the query string
2. Hashes the query together with its answers/function_calls
3. Retains only novel examples

This prevents near-duplicate scenarios from consuming the generation budget and improves dataset diversity.

## Step 7: Progress Tracking and Termination

The generation loop continues until `num_samples` unique examples are collected. A safety limit of `target = int(num_samples * 1.3)` prevents infinite loops when deduplication rates are high.

Progress reporting occurs through either:
- An optional `progress` callback function for programmatic integration
- Console print statements (lines 36-41) for CLI visibility

## Step 8: Output Generation

The final list of synthetic examples is returned to `generate_main` and written as **JSONL** (one JSON object per line). Each row follows this schema:

```json
{
  "query": "How many pods are running in the Kubernetes cluster?",
  "tools": [{ "name": "k8s_list_pods", "description": "...", "parameters": {...} }],
  "answers": [
    { "name": "k8s_list_pods", "arguments": {} }
  ],
  "reasoning": "The user asks for pod count; the appropriate tool is k8s_list_pods."
}

```

This format matches the expectations of `render_example` (lines 94-106), which prepares data for LoRA fine-tuning.

## Key Implementation Files

| File | Primary Role |
|------|--------------|
| [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py) | Core synthesis logic: `generate_examples`, `generate_dataset`, `generate_main` |
| [`needle/cli.py`](https://github.com/cactus-compute/needle/blob/main/needle/cli.py) | CLI registration and argument parsing for `generate-data` subcommand |
| [`needle/agent/fetch.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/fetch.py) | `_openrouter` HTTP wrapper for OpenRouter API access |

## Summary

- **Tool schemas drive generation**: Input JSON defines available functions and their signatures
- **Parallel batches maximize throughput**: ThreadPoolExecutor with configurable worker count
- **Structured prompting ensures quality**: System + user prompts with embedded tool definitions
- **Deduplication guarantees diversity**: Content hashing prevents redundant examples
- **JSONL output enables immediate fine-tuning**: Compatible with Needle's LoRA training pipeline

## Frequently Asked Questions

### What model does Needle use for synthetic data generation?

By default, Needle uses `anthropic/claude-2.0` via OpenRouter. You can override this with any model available on OpenRouter using the `--model` flag, such as `--model openai/gpt-4` or `--model meta-llama/llama-2-70b`.

### How does Needle prevent duplicate training examples?

Needle deduplicates using `_dedup_key`, which hashes the lower-cased query combined with its answers or function calls. Only examples with unique hashes are retained, ensuring dataset variety even when generating thousands of samples.

### Can I generate data without using the CLI?

Yes. Import `generate_dataset` from `needle.model.finetune` and call it programmatically with your tool schemas. This returns a list of example dictionaries that you can process, filter, or serialize manually.

### What is the "refusals" parameter in prompt construction?

The refusals count instructs the LLM to include a specific number of examples where **no tool should be called**. This trains the model to recognize queries outside the tool scope and respond appropriately without forced function invocation.