# Needle JSONL Dataset Format: Structure, Fields, and Parsing

> Explore the Needle JSONL dataset format. Understand its structure, mandatory query field, and optional reasoning, answers, tools, and system fields for effective data parsing.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: api-reference
- Published: 2026-08-16

---

**Needle uses a line-delimited JSON (JSONL) format where each line is a single training example containing a mandatory `"query"` field and optional `"reasoning"`, `"answers"`, `"tools"`, and `"system"` fields.**

Needle, the fine-tuning framework for tool-calling language models developed by cactus-compute, defines a strict JSONL dataset format that bridges raw training data and the tokenization pipeline. Understanding this structure is essential for preparing custom datasets or extending Needle's data generation capabilities.

## Core Structure of Needle's JSONL Format

Each line in a Needle JSONL file is a standalone JSON object. The parser in [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py) specifically looks for the `load_jsonl` function, which reads every line, skips empty lines, and filters out any object missing a `"query"` field.

### Mandatory Fields

Every valid record must include:

- **`query`** (string): The user request or passage that the model must process. This is the only strictly required field—the `load_jsonl` parser explicitly drops lines where this key is absent.

### Optional Fields

Records may additionally contain:

- **`reasoning`** (string): A natural-language explanation of how the model derived its answer from the query. Added during dataset generation to support chain-of-thought training.

- **`answers`** or **`function_calls`** (list of objects): One or more tool-call specifications. Each object requires:
  - **`name`**: The tool schema name
  - **`arguments`**: A JSON object whose keys exactly match the schema's parameters

- **`tools`** (list of objects): The tool schemas used to generate the example. The `generate_examples` function in [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py) adds this field automatically during dataset augmentation. Each tool object contains `name`, `description`, and `parameters` keys.

- **`system`** (string): An optional system prompt attached to the example. Rarely used in Needle's default pipeline.

## From JSONL to Training Tensors

The `load_jsonl` function in [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py) handles the complete ingestion pipeline:

1. Reads the file line by line
2. Skips empty or malformed lines
3. Retains only objects with a `"query"` field
4. Calls `_encode` to tokenize each example using the model's tokenizer from [`needle/model/tokenizer.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/tokenizer.py)
5. Returns two NumPy arrays: token IDs and attention masks

These arrays feed directly into Needle's fine-tuning loop.

## Minimal JSONL Examples

Save any of these examples as `train.jsonl` and point Needle's CLI at the file:

```bash
needle finetune --jsonl_path train.jsonl

```

### Basic tool call

```json
{"query":"What is the current weather in Paris?","answers":[{"name":"weather","arguments":{"city":"Paris"}}]}

```

### With reasoning chain

```json
{"query":"Summarize the following article: ...","reasoning":"Extract the main points from the given text","answers":[{"name":"summarize","arguments":{"text":"...","max_sentences":3}}]}

```

### With attached tool schemas

```json
{"query":"Translate 'Hello' into Japanese","tools":[{"name":"translate","description":"Translate text between languages","parameters":{"source_lang":"en","target_lang":"ja","text":"string"}}],"answers":[{"name":"translate","arguments":{"source_lang":"en","target_lang":"ja","text":"Hello"}}]}

```

## Key Source Files

| File | Function | Purpose |
|------|----------|---------|
| [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py) | `load_jsonl` | Parses JSONL and returns token/ID arrays |
| [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py) | `generate_examples` | Produces JSONL-compatible examples from tool schemas |
| [`needle/model/tokenizer.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/tokenizer.py) | Tokenizer class | Provides `_encode` and special token IDs |
| [`needle/cli.py`](https://github.com/cactus-compute/needle/blob/main/needle/cli.py) | CLI entry point | Accepts `--jsonl_path` argument |

## Generation Pipeline Integration

The `generate_examples` function creates training data that matches the JSONL format exactly. When augmenting a seed dataset, this function:

- Generates `"query"` and `"answers"` fields for each example
- Optionally includes `"reasoning"` explanations
- Attaches the source `"tools"` schemas for provenance tracking

This output structure ensures compatibility with `load_jsonl` without additional transformation.

## Summary

- **Needle's JSONL format** requires one JSON object per line with a mandatory `"query"` field
- The **`load_jsonl`** parser in [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py) filters invalid records and tokenizes valid ones via `_encode`
- Tool calls use the **`answers`** or **`function_calls`** key with `name` and `arguments` sub-fields
- The **`generate_examples`** helper produces properly structured training data for augmentation workflows
- Optional fields like **`reasoning`**, **`tools`**, and **`system`** support advanced training scenarios

## Frequently Asked Questions

### What happens if a JSONL line lacks a "query" field?

The `load_jsonl` parser silently skips it. Only lines containing a `"query"` key proceed to tokenization.

### Can I use "function_calls" instead of "answers"?

Yes. Both keys are accepted for the tool-call list. The parser treats them equivalently.

### How does Needle validate tool arguments against schemas?

Argument validation occurs during dataset generation via `generate_examples`, not during `load_jsonl`. The training data should already contain valid argument objects whose keys match the corresponding tool schema's parameters.

### Where are system prompts specified in the JSONL format?

Use the optional `"system"` field at the top level of each record. However, this field is rarely used in Needle's default pipeline, which typically handles system prompts through other configuration mechanisms.