# How Needle Synthesizes Training Data with OpenRouter and Uses `--generate` for Data Expansion

> Discover how Needle synthesizes training data with OpenRouter and expands datasets using the --generate flag. Learn to automate synthetic data creation for efficient model training.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: how-to-guide
- Published: 2026-08-14

---

**Needle uses OpenRouter's API to automatically generate synthetic training examples via the `--generate` flag on the `finetune` CLI command, which extracts tool schemas from existing data, prompts a language model for varied examples, deduplicates results, and augments the training dataset before finetuning begins.**

Needle, an open-source tool-calling and extraction model framework from cactus-compute, provides built-in **data synthesis capabilities** through OpenRouter. This feature lets practitioners expand limited training datasets without manual annotation, directly addressing the cold-start problem common in function-calling model development.

---

## Overview of the OpenRouter Data Synthesis Pipeline

At its core, Needle's synthesis engine lives in [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py). The pipeline works by analyzing your existing training examples, extracting the tool schemas they use, and prompting a capable language model (via OpenRouter) to generate realistic new examples that follow those same patterns.

The process involves four interconnected stages:

1. **Schema extraction** – Parse existing training data to identify all tool definitions
2. **Parallel generation** – Use threaded workers to synthesize examples in batches
3. **Response parsing** – Extract structured JSON from model outputs
4. **Deduplication** – Remove duplicate examples before augmentation

---

## The `--generate` Flag in the Finetune CLI

The `--generate` flag is defined in [`needle/cli.py`](https://github.com/cactus-compute/needle/blob/main/needle/cli.py) (lines 32-38) and wired into the `finetune_local` function. When you specify a positive integer, Needle triggers automatic dataset expansion before training begins.

```bash
needle finetune training_data.jsonl --generate 500 --model deepseek/deepseek-v4-flash

```

This command sequence:

- Loads `training_data.jsonl`
- Calls `_collect_tools()` to extract unique tool schemas
- Invokes `augment_jsonl()` with the requested count (500)
- Writes results to a `.augmented.jsonl` file
- Proceeds with LoRA finetuning on the enlarged dataset

The flag accepts these related parameters:

| Flag | Purpose | Default |
|------|---------|---------|
| `--generate` | Number of synthetic examples to create | `0` (disabled) |
| `--model` | OpenRouter model identifier for generation | Required when `--generate` > 0 |
| `--workers` | Parallel generation threads | `8` |

---

## Core Implementation: From Prompt to Augmented Dataset

### Step 1: Building the OpenRouter Request

The `_openrouter` helper function (lines 56-66 in [`finetune.py`](https://github.com/cactus-compute/needle/blob/main/finetune.py)) constructs the API call:

```python
def _openrouter(messages: list, model: str, api_key: str, temperature: float = 0.7):
    headers = {
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json"
    }
    payload = {
        "model": model,
        "messages": messages,
        "temperature": temperature
    }
    response = requests.post(
        "https://openrouter.ai/api/v1/chat/completions",
        headers=headers,
        json=payload,
        timeout=120
    )
    response.raise_for_status()
    return response.json()["choices"][0]["message"]["content"]

```

This standardizes all OpenRouter communication, handling authentication via the `OPENROUTER_API_KEY` environment variable and enforcing a 120-second timeout.

### Step 2: Prompt Engineering for Consistent Output

Needle uses two carefully crafted constants to guide generation (lines 31-53):

- **`_GEN_SYSTEM`** – Instructs the model to "generate training data for a tool-calling and extraction model" with specific formatting requirements
- **`_GEN_TEMPLATE`** – A user message template that injects:
  - The available tool schemas (as JSON)
  - The requested number of examples
  - Required output structure (JSON array)

These prompts ensure generated examples match Needle's expected training format without post-processing ambiguity.

### Step 3: Single Example Generation

The `generate_examples` function (lines 80-92) orchestrates one synthesis call:

```python
def generate_examples(tools: list, n: int, model: str, api_key: str) -> list:
    messages = [
        {"role": "system", "content": _GEN_SYSTEM},
        {"role": "user", "content": _GEN_TEMPLATE.format(
            tools=json.dumps(tools, indent=2),
            n=n
        )}
    ]
    raw = _openrouter(messages, model, api_key)
    # Extract JSON array from potential markdown code blocks

    text = raw.strip()
    if text.startswith("```"):
        text = text.split("```")[1]
        if text.startswith("json"):
            text = text[4:]
    parsed = json.loads(text.strip())
    
    # Attach tool definitions to each example

    for ex in parsed:
        ex["tools"] = tools
    return parsed

```

This handles common model behaviors like wrapping JSON in markdown fences and ensures every generated example retains tool context.

### Step 4: Scalable Dataset Generation

The `generate_dataset` function (lines 99-141) manages production-scale synthesis:

```python
def generate_dataset(tools: list, total: int, model: str, api_key: str, workers: int = 8):
    batch_size = max(1, total // workers)
    remainder = total - (batch_size * workers)
    
    futures = []
    with ThreadPoolExecutor(max_workers=workers) as executor:
        for i in range(workers):
            count = batch_size + (1 if i < remainder else 0)
            futures.append(executor.submit(
                generate_examples, tools, count, model, api_key
            ))
        
        all_examples = []
        for future in tqdm(as_completed(futures), total=len(futures), desc="Generating"):
            all_examples.extend(future.result())
    
    # Deduplicate based on query + answers fingerprint

    seen = set()
    unique = []
    for ex in all_examples:
        key = (ex.get("query", ""), json.dumps(ex.get("answers", []), sort_keys=True))
        if key not in seen:
            seen.add(key)
            unique.append(ex)
    
    return unique

```

Key features include **parallel execution** with configurable workers, **progress tracking** via `tqdm`, and **semantic deduplication** that prevents near-duplicate queries from bloating the dataset.

---

## Practical Usage Examples

### Expanding an Existing Training File

```bash
export OPENROUTER_API_KEY=sk-xxxxxxxxxxxx

needle finetune \
    my_data.jsonl \
    --checkpoint deepseek/deepseek-v4-flash \
    --epochs 4 \
    --generate 500 \
    --model deepseek/deepseek-v4-flash \
    --workers 12

```

This workflow:
1. Reads `my_data.jsonl` to understand your tool schemas
2. Generates 500 new examples using 12 parallel workers
3. Creates `my_data.augmented.jsonl` with combined original + synthetic data
4. Executes LoRA finetuning on the augmented dataset

### Standalone Data Generation Without Training

For cases where you want synthetic data without immediate training, use the dedicated subcommand:

```bash
needle generate-data \
    --tools schemas.json \
    --num-samples 200 \
    --batch-size 25 \
    --workers 16 \
    --model deepseek/deepseek-v4-flash \
    --output synthetic_data.jsonl

```

This bypasses the `finetune` workflow entirely, invoking `generate_dataset` directly with a tool schema file rather than inferring schemas from training examples.

---

## Key Source Files and Their Responsibilities

| File | Role | Critical Functions |
|------|------|------------------|
| [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py) | Core synthesis engine | `_openrouter()`, `generate_examples()`, `generate_dataset()`, `augment_jsonl()` |
| [`needle/cli.py`](https://github.com/cactus-compute/needle/blob/main/needle/cli.py) | CLI interface | `--generate` flag definition, `finetune_local()` orchestration |
| [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py) | Training execution | Loads augmented data for actual LoRA training |
| [`tests/test_generate.py`](https://github.com/cactus-compute/needle/blob/main/tests/test_generate.py) | Pipeline validation | Unit tests for generation and deduplication logic |

---

## Summary

- **Needle's data synthesis** uses OpenRouter to generate realistic training examples matching your tool schemas, implemented primarily in [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py)
- **The `--generate` flag** triggers automatic expansion: specify the count of desired examples on the `finetune` CLI
- **Parallel generation** with configurable workers and built-in deduplication ensures scalable, clean dataset expansion
- **Two entry points** exist: integrated augmentation via `finetune --generate` or standalone generation via `needle generate-data`
- **Environment setup** requires only `OPENROUTER_API_KEY` with any OpenRouter-supported model identifier

---

## Frequently Asked Questions

### What OpenRouter models work best for data synthesis in Needle?

According to the Needle source code, any model available through OpenRouter's `/chat/completions` endpoint can be specified via `--model`. The examples in [`cli.py`](https://github.com/cactus-compute/needle/blob/main/cli.py) reference `deepseek/deepseek-v4-flash`, but you can substitute any capable instruction-tuned model. Models with strong JSON-following capabilities produce more parseable outputs, reducing generation failures.

### How does Needle prevent duplicate synthetic examples?

The `generate_dataset` function implements fingerprint-based deduplication using a tuple of `(query, serialized_answers)` as the unique key. This catches semantically identical examples even if formatting differs. The deduplication runs after all parallel workers complete, ensuring the final dataset contains only unique training instances.

### Can I use data synthesis without an existing training file?

Yes. The `generate-data` subcommand accepts a `--tools` parameter pointing to a JSON file containing tool schemas. This bypasses the schema extraction from training data, letting you bootstrap a synthetic dataset from tool definitions alone before collecting any real examples.

### What happens if OpenRouter returns malformed JSON?

The `generate_examples` function includes defensive parsing that strips markdown code fences (```json ... ```) before JSON parsing. If parsing still fails, the exception propagates up and that batch is lost. The ThreadPoolExecutor in `generate_dataset` handles individual batch failures gracefully, so one bad response doesn't crash the entire generation run.