How Needle Synthesizes Training Data with OpenRouter and Uses `--generate` for Data Expansion

Needle uses OpenRouter's API to automatically generate synthetic training examples via the --generate flag on the finetune CLI command, which extracts tool schemas from existing data, prompts a language model for varied examples, deduplicates results, and augments the training dataset before finetuning begins.

Needle, an open-source tool-calling and extraction model framework from cactus-compute, provides built-in data synthesis capabilities through OpenRouter. This feature lets practitioners expand limited training datasets without manual annotation, directly addressing the cold-start problem common in function-calling model development.

---

Overview of the OpenRouter Data Synthesis Pipeline

At its core, Needle's synthesis engine lives in needle/model/finetune.py. The pipeline works by analyzing your existing training examples, extracting the tool schemas they use, and prompting a capable language model (via OpenRouter) to generate realistic new examples that follow those same patterns.

The process involves four interconnected stages:

1. Schema extraction – Parse existing training data to identify all tool definitions
2. Parallel generation – Use threaded workers to synthesize examples in batches
3. Response parsing – Extract structured JSON from model outputs
4. Deduplication – Remove duplicate examples before augmentation

---

The --generate Flag in the Finetune CLI

The --generate flag is defined in needle/cli.py (lines 32-38) and wired into the finetune_local function. When you specify a positive integer, Needle triggers automatic dataset expansion before training begins.

needle finetune training_data.jsonl --generate 500 --model deepseek/deepseek-v4-flash

This command sequence:

• Loads training_data.jsonl
• Calls _collect_tools() to extract unique tool schemas
• Invokes augment_jsonl() with the requested count (500)
• Writes results to a .augmented.jsonl file
• Proceeds with LoRA finetuning on the enlarged dataset

The flag accepts these related parameters:

Flag	Purpose	Default
--generate	Number of synthetic examples to create	0 (disabled)
--model	OpenRouter model identifier for generation	Required when --generate > 0
--workers	Parallel generation threads	8

---

Core Implementation: From Prompt to Augmented Dataset

Step 1: Building the OpenRouter Request

The _openrouter helper function (lines 56-66 in finetune.py) constructs the API call:

def _openrouter(messages: list, model: str, api_key: str, temperature: float = 0.7):
    headers = {
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json"
    }
    payload = {
        "model": model,
        "messages": messages,
        "temperature": temperature
    }
    response = requests.post(
        "https://openrouter.ai/api/v1/chat/completions",
        headers=headers,
        json=payload,
        timeout=120
    )
    response.raise_for_status()
    return response.json()["choices"][0]["message"]["content"]

This standardizes all OpenRouter communication, handling authentication via the OPENROUTER_API_KEY environment variable and enforcing a 120-second timeout.

Step 2: Prompt Engineering for Consistent Output

Needle uses two carefully crafted constants to guide generation (lines 31-53):

• _GEN_SYSTEM – Instructs the model to "generate training data for a tool-calling and extraction model" with specific formatting requirements
• _GEN_TEMPLATE – A user message template that injects:
  • The available tool schemas (as JSON)
  • The requested number of examples
  • Required output structure (JSON array)

These prompts ensure generated examples match Needle's expected training format without post-processing ambiguity.

Step 3: Single Example Generation

The generate_examples function (lines 80-92) orchestrates one synthesis call:

def generate_examples(tools: list, n: int, model: str, api_key: str) -> list:
    messages = [
        {"role": "system", "content": _GEN_SYSTEM},
        {"role": "user", "content": _GEN_TEMPLATE.format(
            tools=json.dumps(tools, indent=2),
            n=n
        )}
    ]
    raw = _openrouter(messages, model, api_key)
    # Extract JSON array from potential markdown code blocks

    text = raw.strip()
    if text.startswith("```"):
        text = text.split("```")[1]
        if text.startswith("json"):
            text = text[4:]
    parsed = json.loads(text.strip())
    
    # Attach tool definitions to each example

    for ex in parsed:
        ex["tools"] = tools
    return parsed

This handles common model behaviors like wrapping JSON in markdown fences and ensures every generated example retains tool context.

Step 4: Scalable Dataset Generation

The generate_dataset function (lines 99-141) manages production-scale synthesis:

def generate_dataset(tools: list, total: int, model: str, api_key: str, workers: int = 8):
    batch_size = max(1, total // workers)
    remainder = total - (batch_size * workers)
    
    futures = []
    with ThreadPoolExecutor(max_workers=workers) as executor:
        for i in range(workers):
            count = batch_size + (1 if i < remainder else 0)
            futures.append(executor.submit(
                generate_examples, tools, count, model, api_key
            ))
        
        all_examples = []
        for future in tqdm(as_completed(futures), total=len(futures), desc="Generating"):
            all_examples.extend(future.result())
    
    # Deduplicate based on query + answers fingerprint

    seen = set()
    unique = []
    for ex in all_examples:
        key = (ex.get("query", ""), json.dumps(ex.get("answers", []), sort_keys=True))
        if key not in seen:
            seen.add(key)
            unique.append(ex)
    
    return unique

Key features include parallel execution with configurable workers, progress tracking via tqdm, and semantic deduplication that prevents near-duplicate queries from bloating the dataset.

---

Practical Usage Examples

Expanding an Existing Training File

export OPENROUTER_API_KEY=sk-xxxxxxxxxxxx

needle finetune \
    my_data.jsonl \
    --checkpoint deepseek/deepseek-v4-flash \
    --epochs 4 \
    --generate 500 \
    --model deepseek/deepseek-v4-flash \
    --workers 12

This workflow:

1. Reads my_data.jsonl to understand your tool schemas
2. Generates 500 new examples using 12 parallel workers
3. Creates my_data.augmented.jsonl with combined original + synthetic data
4. Executes LoRA finetuning on the augmented dataset

Standalone Data Generation Without Training

For cases where you want synthetic data without immediate training, use the dedicated subcommand:

needle generate-data \
    --tools schemas.json \
    --num-samples 200 \
    --batch-size 25 \
    --workers 16 \
    --model deepseek/deepseek-v4-flash \
    --output synthetic_data.jsonl

This bypasses the finetune workflow entirely, invoking generate_dataset directly with a tool schema file rather than inferring schemas from training examples.

---

Key Source Files and Their Responsibilities

File	Role	Critical Functions
needle/model/finetune.py	Core synthesis engine	_openrouter(), generate_examples(), generate_dataset(), augment_jsonl()
needle/cli.py	CLI interface	--generate flag definition, finetune_local() orchestration
needle/model/run.py	Training execution	Loads augmented data for actual LoRA training
tests/test_generate.py	Pipeline validation	Unit tests for generation and deduplication logic

---

Summary

• Needle's data synthesis uses OpenRouter to generate realistic training examples matching your tool schemas, implemented primarily in needle/model/finetune.py
• The --generate flag triggers automatic expansion: specify the count of desired examples on the finetune CLI
• Parallel generation with configurable workers and built-in deduplication ensures scalable, clean dataset expansion
• Two entry points exist: integrated augmentation via finetune --generate or standalone generation via needle generate-data
• Environment setup requires only OPENROUTER_API_KEY with any OpenRouter-supported model identifier

---

Frequently Asked Questions

What OpenRouter models work best for data synthesis in Needle?

According to the Needle source code, any model available through OpenRouter's /chat/completions endpoint can be specified via --model. The examples in cli.py reference deepseek/deepseek-v4-flash, but you can substitute any capable instruction-tuned model. Models with strong JSON-following capabilities produce more parseable outputs, reducing generation failures.

How does Needle prevent duplicate synthetic examples?

The generate_dataset function implements fingerprint-based deduplication using a tuple of (query, serialized_answers) as the unique key. This catches semantically identical examples even if formatting differs. The deduplication runs after all parallel workers complete, ensuring the final dataset contains only unique training instances.

Can I use data synthesis without an existing training file?

Yes. The generate-data subcommand accepts a --tools parameter pointing to a JSON file containing tool schemas. This bypasses the schema extraction from training data, letting you bootstrap a synthetic dataset from tool definitions alone before collecting any real examples.

What happens if OpenRouter returns malformed JSON?

The generate_examples function includes defensive parsing that strips markdown code fences (json ... ) before JSON parsing. If parsing still fails, the exception propagates up and that batch is lost. The ThreadPoolExecutor in generate_dataset handles individual batch failures gracefully, so one bad response doesn't crash the entire generation run.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}