Needle JSONL Dataset Format: Structure, Fields, and Parsing

Needle uses a line-delimited JSON (JSONL) format where each line is a single training example containing a mandatory "query" field and optional "reasoning", "answers", "tools", and "system" fields.

Needle, the fine-tuning framework for tool-calling language models developed by cactus-compute, defines a strict JSONL dataset format that bridges raw training data and the tokenization pipeline. Understanding this structure is essential for preparing custom datasets or extending Needle's data generation capabilities.

Core Structure of Needle's JSONL Format

Each line in a Needle JSONL file is a standalone JSON object. The parser in needle/model/finetune.py specifically looks for the load_jsonl function, which reads every line, skips empty lines, and filters out any object missing a "query" field.

Mandatory Fields

Every valid record must include:

• query (string): The user request or passage that the model must process. This is the only strictly required field—the load_jsonl parser explicitly drops lines where this key is absent.

Optional Fields

Records may additionally contain:

• reasoning (string): A natural-language explanation of how the model derived its answer from the query. Added during dataset generation to support chain-of-thought training.

• answers or function_calls (list of objects): One or more tool-call specifications. Each object requires:

  • name: The tool schema name
  • arguments: A JSON object whose keys exactly match the schema's parameters

• tools (list of objects): The tool schemas used to generate the example. The generate_examples function in needle/model/finetune.py adds this field automatically during dataset augmentation. Each tool object contains name, description, and parameters keys.

• system (string): An optional system prompt attached to the example. Rarely used in Needle's default pipeline.

From JSONL to Training Tensors

The load_jsonl function in needle/model/finetune.py handles the complete ingestion pipeline:

1. Reads the file line by line
2. Skips empty or malformed lines
3. Retains only objects with a "query" field
4. Calls _encode to tokenize each example using the model's tokenizer from needle/model/tokenizer.py
5. Returns two NumPy arrays: token IDs and attention masks

These arrays feed directly into Needle's fine-tuning loop.

Minimal JSONL Examples

Save any of these examples as train.jsonl and point Needle's CLI at the file:

needle finetune --jsonl_path train.jsonl

Basic tool call

{"query":"What is the current weather in Paris?","answers":[{"name":"weather","arguments":{"city":"Paris"}}]}

With reasoning chain

{"query":"Summarize the following article: ...","reasoning":"Extract the main points from the given text","answers":[{"name":"summarize","arguments":{"text":"...","max_sentences":3}}]}

With attached tool schemas

{"query":"Translate 'Hello' into Japanese","tools":[{"name":"translate","description":"Translate text between languages","parameters":{"source_lang":"en","target_lang":"ja","text":"string"}}],"answers":[{"name":"translate","arguments":{"source_lang":"en","target_lang":"ja","text":"Hello"}}]}

Key Source Files

File	Function	Purpose
needle/model/finetune.py	load_jsonl	Parses JSONL and returns token/ID arrays
needle/model/finetune.py	generate_examples	Produces JSONL-compatible examples from tool schemas
needle/model/tokenizer.py	Tokenizer class	Provides _encode and special token IDs
needle/cli.py	CLI entry point	Accepts --jsonl_path argument

Generation Pipeline Integration

The generate_examples function creates training data that matches the JSONL format exactly. When augmenting a seed dataset, this function:

• Generates "query" and "answers" fields for each example
• Optionally includes "reasoning" explanations
• Attaches the source "tools" schemas for provenance tracking

This output structure ensures compatibility with load_jsonl without additional transformation.

Summary

• Needle's JSONL format requires one JSON object per line with a mandatory "query" field
• The load_jsonl parser in needle/model/finetune.py filters invalid records and tokenizes valid ones via _encode
• Tool calls use the answers or function_calls key with name and arguments sub-fields
• The generate_examples helper produces properly structured training data for augmentation workflows
• Optional fields like reasoning, tools, and system support advanced training scenarios

Frequently Asked Questions

What happens if a JSONL line lacks a "query" field?

The load_jsonl parser silently skips it. Only lines containing a "query" key proceed to tokenization.

Can I use "function_calls" instead of "answers"?

Yes. Both keys are accepted for the tool-call list. The parser treats them equivalently.

How does Needle validate tool arguments against schemas?

Argument validation occurs during dataset generation via generate_examples, not during load_jsonl. The training data should already contain valid argument objects whose keys match the corresponding tool schema's parameters.

Where are system prompts specified in the JSONL format?

Use the optional "system" field at the top level of each record. However, this field is rarely used in Needle's default pipeline, which typically handles system prompts through other configuration mechanisms.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}