# How the jsontool.py Module Works in jsoncsv: Flattening and Restoring Nested JSON

> Discover how jsontool.py in jsoncsv flattens nested JSON to dot notation and restores it. Understand the core transformation logic for efficient data handling.

- Repository: [alingse/jsoncsv](https://github.com/alingse/jsoncsv)
- Tags: internals
- Published: 2026-02-24

---

**The [`jsontool.py`](https://github.com/alingse/jsoncsv/blob/main/jsontool.py) module provides the core bidirectional transformation logic that converts nested JSON objects into flat dot-notation dictionaries and reconstructs them back to their original hierarchical structure.**

The `jsoncsv` library enables seamless conversion between hierarchical JSON data and flat CSV formats through a compact transformation engine. At the heart of this functionality sits the **[`jsoncsv/jsontool.py`](https://github.com/alingse/jsoncsv/blob/main/jsoncsv/jsontool.py)** module, which implements recursive tree traversal algorithms to bridge the gap between nested objects and path-based key-value representations. Understanding how this module processes data is essential for developers building data pipelines that require JSON normalization or denormalization.

## Core Functions in jsontool.py

The module exposes six primary functions that handle distinct aspects of the conversion process, from leaf node extraction to stream processing.

### gen_leaf: Recursive Tree Traversal

The **`gen_leaf`** function serves as the foundational iterator that performs depth-first traversal of JSON objects. Located at [lines 32-47](https://github.com/alingse/jsoncsv/blob/master/jsoncsv/jsontool.py#L32-L47), it accepts a nested structure and yields `(path, value)` tuples for every leaf node.

The function initializes with an empty `path` list and recursively descends through dictionaries and lists. For each container element, it appends the current key or index to the path before continuing recursion. When encountering a scalar value or empty container, it yields the accumulated path alongside the value.

### is_array_index: Detecting List Structures

To automatically distinguish between objects and arrays during reconstruction, the module uses **`is_array_index`** ([lines 49-58](https://github.com/alingse/jsoncsv/blob/master/jsoncsv/jsontool.py#L49-L58)). This utility inspects a collection of keys to determine whether they represent consecutive integer indexes.

The function checks whether keys are integers or string representations of integers when the `enable_str` parameter is active. This detection enables the reconstruction logic to choose between creating a Python list or dictionary for child elements.

### from_leaf: Reconstructing Hierarchy

The inverse operation of `gen_leaf`, **`from_leaf`** ([lines 61-90](https://github.com/alingse/jsoncsv/blob/master/jsoncsv/jsontool.py#L61-L90)) rebuilds nested JSON structures from flat path tuples. It operates through a three-step process:

1. Extracts the first element of each path to obtain "head" keys
2. Groups leaf tuples by their head values
3. Recursively processes each group, using `is_array_index` to determine whether to instantiate a list or dict

This recursive grouping strategy efficiently reconstructs the original data hierarchy without requiring schema definitions.

### expand: Flattening to Dot Notation

The **`expand`** function ([lines 93-106](https://github.com/alingse/jsoncsv/blob/master/jsoncsv/jsontool.py#L93-L106)) provides the primary interface for JSON-to-CSV preparation. It consumes a nested object and produces a one-level dictionary where keys represent dot-separated paths.

By default, the function uses `"."` as the separator, though this is configurable. When `safe=True`, it invokes `encode_safe_key` from [`utils.py`](https://github.com/alingse/jsoncsv/blob/main/utils.py) to escape separator characters that appear within actual key names, preventing path corruption.

### restore: Unflattening Dictionary Data

Operating as the reverse of `expand`, **`restore`** ([lines 108-122](https://github.com/alingse/jsoncsv/blob/master/jsoncsv/jsontool.py#L108-L122)) converts flat dictionaries back into nested structures. It splits each key by the specified separator—or uses `decode_safe_key` when safety mode is enabled—to regenerate path arrays, then delegates the reconstruction to `from_leaf`.

### convert_json: Stream Processing Interface

For command-line and high-volume processing scenarios, **`convert_json`** ([lines 124-162](https://github.com/alingse/jsoncsv/blob/master/jsoncsv/jsontool.py#L124-L162)) handles streaming transformations. This function accepts file-like objects for input and output, applying either `expand` or `restore` to each JSON object in the stream.

It supports both newline-delimited JSON (NDJSON) and standard JSON array inputs through the `json_array` parameter, writing each transformed result as a JSON line to the output stream.

## Architecture and Data Flow

The module implements a bidirectional pipeline that handles two primary transformation directions.

### Flattening Pipeline

When converting nested JSON to flat structures, the pipeline executes in three stages:

1. **Extraction**: `gen_leaf` traverses the object tree, yielding every leaf value with its complete path
2. **Path Serialization**: `expand` joins path components using the configured separator
3. **Safety Encoding**: Optional escaping via [`utils.py`](https://github.com/alingse/jsoncsv/blob/main/utils.py) helpers ensures keys containing the separator character remain intact

The resulting dictionary maps dot-notation strings to scalar values, ready for CSV column headers.

### Restoration Pipeline

The reverse process reconstructs hierarchy from flat data:

1. **Path Parsing**: `restore` splits keys or decodes safe keys to regenerate path arrays
2. **Grouping**: `from_leaf` organizes tuples by head keys and recurses through subgroups
3. **Type Detection**: `is_array_index` determines whether each group becomes a list or dict

This automatic type detection eliminates the need for explicit schema mapping during reconstruction.

## Handling Keys with Special Characters

JSON data frequently contains keys that include the dot character, which conflicts with the default path separator. The module addresses this through **safety mode**, implemented via `encode_safe_key` and `decode_safe_key` in **[`jsoncsv/utils.py`](https://github.com/alingse/jsoncsv/blob/main/jsoncsv/utils.py)**.

When `safe=True` is passed to `expand` or `restore`, the module escapes separator characters within keys before flattening and reverses the process during restoration. This ensures round-trip fidelity even for complex key names like `"user.name"` or `"item.price"`.

## Practical Examples

### Flattening Nested Objects

Convert hierarchical data to a CSV-compatible flat dictionary:

```python
from jsoncsv.jsontool import expand

nested = {
    "user": {"id": 1, "name": "Alice"},
    "orders": [{"id": 10, "total": 99.9}, {"id": 11, "total": 12.5}]
}

flat = expand(nested)
print(flat)

```

Output:

```python
{
  'user.id': 1,
  'user.name': 'Alice',
  'orders.0.id': 10,
  'orders.0.total': 99.9,
  'orders.1.id': 11,
  'orders.1.total': 12.5
}

```

### Restoring Flat Structures

Reconstruct the original hierarchy from dot-notation keys:

```python
from jsoncsv.jsontool import restore

flat = {
    "user.id": 1,
    "user.name": "Alice",
    "orders.0.id": 10,
    "orders.0.total": 99.9,
    "orders.1.id": 11,
    "orders.1.total": 12.5
}

nested = restore(flat)
print(nested)

```

Result:

```python
{'user': {'id': 1, 'name': 'Alice'},
 'orders': [{'id': 10, 'total': 99.9},
            {'id': 11, 'total': 12.5}]}

```

### Processing JSON Streams

Transform line-delimited JSON files using the streaming interface:

```python
from jsoncsv.jsontool import convert_json, expand

with open("input.ndjson", "r", encoding="utf-8") as fin, \
     open("output.ndjson", "w", encoding="utf-8") as fout:
    convert_json(fin, fout, func=expand, separator='.')

```

### Safe Mode for Complex Keys

Handle keys containing the separator character without data corruption:

```python
from jsoncsv.jsontool import expand, restore

obj = {"a.b": {"c": 1}}
flat_safe = expand(obj, safe=True)      # Encodes 'a.b' to avoid splitting

restored = restore(flat_safe, safe=True)
assert restored == obj

```

## Summary

- The **`gen_leaf`** function recursively traverses JSON trees to extract path-value pairs, while **`from_leaf`** reconstructs hierarchies through recursive grouping.
- **`expand`** and **`restore`** provide the public API for bidirectional conversion between nested objects and flat dot-notation dictionaries.
- **`is_array_index`** automatically detects array structures during reconstruction, eliminating the need for explicit schema definitions.
- The **`convert_json`** function enables high-performance stream processing for large datasets and command-line integration.
- Safety mode via **`encode_safe_key`** and **`decode_safe_key`** ensures reliable round-trip conversion when keys contain separator characters.

## Frequently Asked Questions

### What is the primary purpose of jsontool.py in the jsoncsv library?

The [`jsontool.py`](https://github.com/alingse/jsoncsv/blob/main/jsontool.py) module implements the core transformation engine that powers the entire jsoncsv workflow. It provides the mathematical operations required to flatten nested JSON into CSV-compatible dot-notation dictionaries and restore them back to hierarchical structures, handling both streaming and batch processing scenarios.

### How does jsontool.py handle JSON arrays versus objects?

During the restoration phase, the `is_array_index` function analyzes the keys of each group to determine whether they form consecutive integer indexes. If the keys are integers (or string representations thereof), `from_leaf` instantiates a Python list; otherwise, it creates a dictionary. This automatic detection happens at every level of the reconstruction process.

### Can jsontool.py process extremely large JSON files without loading everything into memory?

Yes, the **`convert_json`** function supports streaming processing by accepting file-like objects and processing JSON objects line-by-line. When reading newline-delimited JSON (NDJSON), it transforms each object individually and writes results immediately to the output stream, maintaining constant memory regardless of file size.

### What happens if my JSON keys contain dots when using the default separator?

Without safety mode, dots within keys would be interpreted as path separators during restoration, corrupting the data structure. By passing `safe=True` to `expand` and `restore`, the module uses `encode_safe_key` to escape separator characters before flattening and `decode_safe_key` to restore them during unflattening, ensuring perfect round-trip fidelity.