How to Synthesize Training Data for Needle Fine‑Tuning Using the `generate-data` CLI Command

needle generate-data creates synthetic training examples via OpenRouter API, outputting JSON‑L records ready for fine‑tuning workflows.

Needle is an open‑source framework for building and fine‑tuning language agents. This guide covers the complete command syntax, configuration options, and practical patterns for generating high‑quality synthetic training data for Needle fine‑tuning.

Understanding the generate-data Command

The generate-data sub‑command is defined in [needle/cli.py](https://github.com/cactus-compute/needle/blob/main/needle/cli.py#L56-L66) at lines 56–66. It orchestrates parallel API calls to OpenRouter, constructs prompt payloads, and streams generated examples directly to JSON‑L format.

This command serves two primary purposes:

• Bootstrap new datasets when limited training data exists
• Augment existing datasets to improve model coverage and robustness

Command Options Reference

Option	Default	Purpose
--num-samples	100	Total synthetic examples to generate
--batch-size	25	Examples per OpenRouter API call
--workers	16	Concurrent API request threads
--model	deepseek/deepseek-v4-flash	OpenRouter model identifier
--output	stdout	Destination JSONL file path
--augment	None	Existing JSONL file to expand
--tools	None	JSON string of tool schemas for seeding

Basic Usage Examples

Generate a New Synthetic Dataset

Create 200 training examples and save to synthetic.jsonl:

needle generate-data --num-samples 200 --output synthetic.jsonl

Augment an Existing Dataset

Add 50 new samples to existing.jsonl and write combined results:

needle generate-data \
  --augment existing.jsonl \
  --num-samples 50 \
  --output augmented.jsonl

Control Concurrency and Model Selection

Use 8 workers with Meta‑Llama‑3 for generation:

needle generate-data \
  --workers 8 \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --num-samples 150 \
  --output llama3-data.jsonl

Seed with Custom Tool Schemas

For tool‑calling fine‑tuning, inject schema definitions:

needle generate-data \
  --tools '{"my_tool": {"type":"function","description":"Fetch weather data","parameters":{"properties":{"city":{"type":"string"}}}}}' \
  --num-samples 100 \
  --output tool-seeded.jsonl

How Data Synthesis Works Internally

The synthesis pipeline operates across three core components:

1. CLI parsing — [needle/cli.py](https://github.com/cactus-compute/needle/blob/main/needle/cli.py) validates arguments and builds the generation config

2. Concurrency management — [needle/_worker.py](https://github.com/cactus-compute/needle/blob/main/needle/_worker.py) implements the thread pool for parallel OpenRouter requests

3. Tool schema handling — [needle/agent/tools.py](https://github.com/cactus-compute/needle/blob/main/needle/agent/tools.py) processes --tools JSON into generation prompts

Generated outputs flow directly into [needle/model/finetune.py](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py), which consumes JSONL records during the needle finetune step.

Performance Tuning for Large Runs

Adjust these parameters based on API rate limits and latency:

• Reduce --workers if hitting OpenRouter rate limits
• Increase --batch-size to reduce total API calls (check model context limits)
• Use faster models (deepseek/deepseek-v4-flash default) for cost‑efficient iteration

Summary

• needle generate-data is the dedicated CLI command for synthetic training data generation
• Outputs JSON‑L format compatible with needle finetune
• Supports dataset augmentation via --augment flag
• Parallelizes requests through configurable worker pools
• Accepts tool schemas to steer generation for specialized fine‑tuning

Frequently Asked Questions

What file format does generate-data produce?

The command writes JSON Lines (JSONL) format, with one training example per line. This matches the input expectation of [needle/model/finetune.py](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py).

Can I use a different LLM provider instead of OpenRouter?

The current implementation in needle/_worker.py targets OpenRouter specifically. To use alternative providers, you would need to modify the worker's request construction and authentication handling.

How do I resume an interrupted generation run?

There is no built-in resume mechanism. Recommended approach: use --output to write partial results, then run subsequent generations with --augment pointing to the partial file.

What is the relationship between --batch-size and --workers?

--batch-size controls examples per API request; --workers controls simultaneous requests. Total concurrent examples in flight equals batch-size × workers. Default configuration allows 400 examples in parallel (25 × 16).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}