How Needle's `generate-data` Synthesizes Training Examples: Complete Technical Guide

Needle synthesizes training examples by using OpenRouter-hosted LLMs to generate tool-calling scenarios based on user-supplied JSON schemas, with built-in deduplication and parallel batch processing.

The needle generate-data command is the entry point for creating synthetic datasets that teach language models when and how to invoke external tools. This article explains the complete synthesis pipeline implemented in the cactus-compute/needle repository, from prompt construction to final JSONL output.

Overview of the Synthesis Pipeline

Data generation in Needle follows an eight-step workflow orchestrated through needle/model/finetune.py. The process combines structured prompting, concurrent API calls, and content-aware deduplication to produce high-quality training examples at scale.

The core entry point is generate_main (lines 78-80), which delegates to generate_dataset for batch coordination and generate_examples for individual LLM interactions.

Step 1: Tool Schema Preparation

Before generation begins, you must supply a JSON file describing the tools your model should learn to call. These schemas define function names, descriptions, and parameter specifications.

needle generate-data \
  --tools my_tools.json \
  --num-samples 500 \
  --output synthetic_data.jsonl

The CLI parser in needle/cli.py (lines 152-199) validates these arguments and forwards them to generate_main, which loads the tool definitions and passes them to generate_dataset.

Step 2: Batch Generation with Thread Pooling

The generate_dataset function uses a ThreadPoolExecutor to parallelize API requests. Each worker invokes generate_examples to produce a batch of n examples (default 25).

from needle.model.finetune import generate_dataset
import json

with open("my_tools.json") as f:
    tools = json.load(f)

examples = generate_dataset(
    tools,
    num_samples=200,
    model="anthropic/claude-2.0",
    batch_size=25,
    workers=8,
)

The thread pool scheduler (lines 108-119) manages up to workers concurrent requests, maximizing throughput while respecting OpenRouter rate limits.

Step 3: Prompt Construction for LLM Inference

Every batch request assembles two prompt components:

• _GEN_SYSTEM — A system prompt establishing the generation task and output format
• _GEN_TEMPLATE — A user prompt embedding the tool JSON, desired example count (n), and "refusals" count instructing the LLM to include scenarios where no tool should be called

These prompts are constructed in generate_examples (lines 85-88) and sent via _openrouter to the selected model. The default model is openrouter.ai/anthropic/claude-2.0, though any OpenRouter-hosted model can be specified.

Step 4: LLM Inference via OpenRouter

The _openrouter helper (imported from needle/agent/fetch.py) handles the HTTP POST to OpenRouter's chat completions endpoint. It transmits the combined system and user messages, returning the raw text response for parsing.

# From needle/agent/fetch.py — simplified structure

def _openrouter(messages, model, temperature=0.7):
    response = requests.post(
        "https://openrouter.ai/api/v1/chat/completions",
        headers={"Authorization": f"Bearer {api_key}"},
        json={"model": model, "messages": messages, "temperature": temperature}
    )
    return response.json()["choices"][0]["message"]["content"]

Step 5: Response Parsing and Validation

Responses are expected as JSON-encoded arrays. The _parse_array function extracts this array, and each row undergoes post-processing to guarantee the "tools" field is present (lines 89-91).

Partial or malformed responses are discarded, ensuring downstream fine-tuning receives valid training examples.

Step 6: Deduplication by Content Hash

As batches complete, generate_dataset applies content-aware deduplication using _dedup_key (lines 94-97). This function:

1. Lower-cases the query string
2. Hashes the query together with its answers/function_calls
3. Retains only novel examples

This prevents near-duplicate scenarios from consuming the generation budget and improves dataset diversity.

Step 7: Progress Tracking and Termination

The generation loop continues until num_samples unique examples are collected. A safety limit of target = int(num_samples * 1.3) prevents infinite loops when deduplication rates are high.

Progress reporting occurs through either:

• An optional progress callback function for programmatic integration
• Console print statements (lines 36-41) for CLI visibility

Step 8: Output Generation

The final list of synthetic examples is returned to generate_main and written as JSONL (one JSON object per line). Each row follows this schema:

{
  "query": "How many pods are running in the Kubernetes cluster?",
  "tools": [{ "name": "k8s_list_pods", "description": "...", "parameters": {...} }],
  "answers": [
    { "name": "k8s_list_pods", "arguments": {} }
  ],
  "reasoning": "The user asks for pod count; the appropriate tool is k8s_list_pods."
}

This format matches the expectations of render_example (lines 94-106), which prepares data for LoRA fine-tuning.

Key Implementation Files

File	Primary Role
needle/model/finetune.py	Core synthesis logic: generate_examples, generate_dataset, generate_main
needle/cli.py	CLI registration and argument parsing for generate-data subcommand
needle/agent/fetch.py	_openrouter HTTP wrapper for OpenRouter API access

Summary

• Tool schemas drive generation: Input JSON defines available functions and their signatures
• Parallel batches maximize throughput: ThreadPoolExecutor with configurable worker count
• Structured prompting ensures quality: System + user prompts with embedded tool definitions
• Deduplication guarantees diversity: Content hashing prevents redundant examples
• JSONL output enables immediate fine-tuning: Compatible with Needle's LoRA training pipeline

Frequently Asked Questions

What model does Needle use for synthetic data generation?

By default, Needle uses anthropic/claude-2.0 via OpenRouter. You can override this with any model available on OpenRouter using the --model flag, such as --model openai/gpt-4 or --model meta-llama/llama-2-70b.

How does Needle prevent duplicate training examples?

Needle deduplicates using _dedup_key, which hashes the lower-cased query combined with its answers or function calls. Only examples with unique hashes are retained, ensuring dataset variety even when generating thousands of samples.

Can I generate data without using the CLI?

Yes. Import generate_dataset from needle.model.finetune and call it programmatically with your tool schemas. This returns a list of example dictionaries that you can process, filter, or serialize manually.

What is the "refusals" parameter in prompt construction?

The refusals count instructs the LLM to include a specific number of examples where no tool should be called. This trains the model to recognize queries outside the tool scope and respond appropriately without forced function invocation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}