How to Synthesize Training Data for Needle 2: A Complete Guide

Needle 2 automatically generates synthetic training examples by sending tool schemas to an OpenRouter model via the generate-data CLI command, producing JSONL datasets that pair realistic user queries with schema-validated tool calls.

When fine-tuning Needle 2 on tool-calling capabilities, obtaining thousands of hand-crafted examples can block development. The cactus-compute/needle repository solves this through an automated synthesize training data pipeline that generates high-quality, schema-compliant training pairs using large language models via OpenRouter.

How the Needle 2 Data Synthesis Pipeline Works

The synthesis process implemented in needle/model/finetune.py follows a five-step workflow orchestrated through needle/cli.py.

Step 1: Schema Parsing and Seed Generation

The pipeline begins by reading your tool definitions. The CLI parses the --tools argument (lines 53-55 in needle/cli.py) to load a JSON schema file that follows the OpenAPI-style format. This schema defines available tools, their argument types, valid enums, and required parameters, serving as the foundation for generating realistic examples.

Step 2: OpenRouter Model Integration

For each requested sample, the generate_main function in needle/model/finetune.py constructs prompts containing your tool schemas and sends them to an OpenRouter model. By default, it uses deepseek/deepseek-v4-flash, though you can override this with the --model flag (lines 59-60 in needle/cli.py). The model generates both a realistic user query and the corresponding tool call that respects the schema constraints defined in your input file.

Step 3: Parallel Request Processing

To accelerate generation, requests dispatch concurrently across multiple workers. The --workers parameter (lines 57-58 in needle/cli.py) defaults to 16 parallel connections, significantly reducing the time required to build large datasets without overwhelming the API endpoint.

Step 4: JSONL Formatting and Output Generation

Each OpenRouter response is structured into a standardized JSONL line containing four fields: query, tools, answers, and optionally reasoning. This format aligns exactly with the fine-tuning specification documented in doc/finetuning.md (lines 5-11). The final dataset writes to your specified --output path (lines 60-61 in needle/cli.py), defaulting to data.jsonl if not provided.

Prerequisites for Synthesizing Training Data

Before running the generator, you must configure your environment and define your tool schemas.

OpenRouter API Configuration

The generator requires an active OpenRouter account. Export your API key as OPENROUTER_API_KEY (as documented in the README, lines 100-106). Without this environment variable, the generate_main function cannot authenticate requests to the OpenRouter endpoint.

Tool Schema Structure

Your input JSON must define tools using standard JSON Schema format. Each tool requires a name, parameter object with types and constraints, and required fields array. The schema drives the generation process, ensuring synthetic examples respect enum values, type constraints, and required parameter rules.

Generating Synthetic Training Data with the CLI

Execute the full synthesis pipeline using the generate-data command. Below are practical examples for creating datasets from scratch.

Creating a new dataset from a tool schema:

# Define your tools following OpenAPI JSON Schema format

cat > tools.json <<EOF
{
  "tools": [
    {
      "name": "set_lights",
      "parameters": {
        "type": "object",
        "properties": {
          "room": { "type": "string", "enum": ["kitchen","study","bedroom"] },
          "brightness": { "type": "integer", "minimum": 0, "maximum": 100 }
        },
        "required": ["room"]
      }
    }
  ]
}
EOF

# Set your OpenRouter API key

export OPENROUTER_API_KEY=sk-or-...

# Generate 500 synthetic examples

needle generate-data \
    --tools tools.json \
    --num-samples 500 \
    --output synthetic_data.jsonl

This produces synthetic_data.jsonl containing 500 training rows where each line pairs a generated user query with a valid set_lights tool call respecting the room enum and brightness constraints.

Augmenting Existing Training Datasets

If you already possess hand-written examples, expand them without starting from scratch. Use the --augment flag (parsed at lines 53-55 in needle/cli.py) to load an existing JSONL file and append new synthetic entries:

needle generate-data \
    --augment hand_written.jsonl \
    --num-samples 300 \
    --output combined_data.jsonl

This approach preserves your curated examples while adding diverse variations generated by the OpenRouter model, creating a hybrid dataset for more robust fine-tuning.

Fine-Tuning on Generated Data

Once synthesized, use the dataset directly in the fine-tuning workflow:

needle finetune combined_data.jsonl --epochs 10 --generate 200

The generate-data output format matches exactly what the finetune command expects, ensuring seamless integration between synthesis and training phases.

Quality Control Best Practices

While the pipeline enforces schema constraints through the JSON definitions, synthetic data requires validation before production use.

Inspect generated samples by examining the first few lines of your output file:

head -n 5 synthetic_data.jsonl

Look for out-of-domain queries or argument values that, while technically valid according to the schema, may not represent realistic user interactions. Filter these rows manually or adjust your tool schema descriptions to guide the OpenRouter model toward higher-quality generations.

You can also adjust the base model via --model if you observe quality issues with the default deepseek/deepseek-v4-flash, selecting alternative OpenRouter endpoints that better match your domain requirements.

Summary

• Needle 2 synthesizes training data via the generate-data CLI command, which invokes generate_main in needle/model/finetune.py to query OpenRouter models.
• The pipeline requires a JSON Schema tool definition (--tools) and an OpenRouter API key (OPENROUTER_API_KEY) to function.
• Requests execute in parallel using 16 workers by default, with each response formatted as JSONL containing query, tools, answers, and optional reasoning fields.
• Use --augment to expand existing datasets, and always validate synthetic examples for domain relevance before fine-tuning.

Frequently Asked Questions

What file format does Needle 2 expect for tool schemas?

Needle 2 expects standard OpenAPI-style JSON Schema files. The schema must define tools with names, parameter objects containing type definitions and constraints (enums, min/max values), and required field arrays. The CLI parser at lines 53-55 in needle/cli.py validates this structure before initiating generation.

Can I use a different model than the default for data generation?

Yes. While the default generator uses deepseek/deepseek-v4-flash, you can specify any OpenRouter-supported model using the --model flag (lines 59-60 in needle/cli.py). This allows you to balance generation cost, speed, and quality by selecting cheaper or more capable alternatives.

How does Needle 2 ensure generated tool calls match the schema?

The generate_main function includes your complete tool schema in every prompt sent to OpenRouter. This in-context schema enforcement instructs the model to respect argument types, required fields, and enum constraints. Additionally, the output formatting logic in needle/model/finetune.py structures the response into the standard JSONL format expected by the fine-tuning pipeline.

Is there a limit to how many samples I can generate?

There is no hardcoded limit in the Needle 2 codebase itself. The --num-samples parameter accepts any integer, though generation speed depends on your OpenRouter rate limits and the --workers setting (default 16). For large datasets exceeding thousands of samples, consider generating in batches to avoid API timeouts or quota exhaustion.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}