How the jsontool.py Module Works in jsoncsv: Flattening and Restoring Nested JSON

The jsontool.py module provides the core bidirectional transformation logic that converts nested JSON objects into flat dot-notation dictionaries and reconstructs them back to their original hierarchical structure.

The jsoncsv library enables seamless conversion between hierarchical JSON data and flat CSV formats through a compact transformation engine. At the heart of this functionality sits the jsoncsv/jsontool.py module, which implements recursive tree traversal algorithms to bridge the gap between nested objects and path-based key-value representations. Understanding how this module processes data is essential for developers building data pipelines that require JSON normalization or denormalization.

Core Functions in jsontool.py

The module exposes six primary functions that handle distinct aspects of the conversion process, from leaf node extraction to stream processing.

gen_leaf: Recursive Tree Traversal

The gen_leaf function serves as the foundational iterator that performs depth-first traversal of JSON objects. Located at lines 32-47, it accepts a nested structure and yields (path, value) tuples for every leaf node.

The function initializes with an empty path list and recursively descends through dictionaries and lists. For each container element, it appends the current key or index to the path before continuing recursion. When encountering a scalar value or empty container, it yields the accumulated path alongside the value.

is_array_index: Detecting List Structures

To automatically distinguish between objects and arrays during reconstruction, the module uses is_array_index (lines 49-58). This utility inspects a collection of keys to determine whether they represent consecutive integer indexes.

The function checks whether keys are integers or string representations of integers when the enable_str parameter is active. This detection enables the reconstruction logic to choose between creating a Python list or dictionary for child elements.

from_leaf: Reconstructing Hierarchy

The inverse operation of gen_leaf, from_leaf (lines 61-90) rebuilds nested JSON structures from flat path tuples. It operates through a three-step process:

1. Extracts the first element of each path to obtain "head" keys
2. Groups leaf tuples by their head values
3. Recursively processes each group, using is_array_index to determine whether to instantiate a list or dict

This recursive grouping strategy efficiently reconstructs the original data hierarchy without requiring schema definitions.

expand: Flattening to Dot Notation

The expand function (lines 93-106) provides the primary interface for JSON-to-CSV preparation. It consumes a nested object and produces a one-level dictionary where keys represent dot-separated paths.

By default, the function uses "." as the separator, though this is configurable. When safe=True, it invokes encode_safe_key from utils.py to escape separator characters that appear within actual key names, preventing path corruption.

restore: Unflattening Dictionary Data

Operating as the reverse of expand, restore (lines 108-122) converts flat dictionaries back into nested structures. It splits each key by the specified separator—or uses decode_safe_key when safety mode is enabled—to regenerate path arrays, then delegates the reconstruction to from_leaf.

convert_json: Stream Processing Interface

For command-line and high-volume processing scenarios, convert_json (lines 124-162) handles streaming transformations. This function accepts file-like objects for input and output, applying either expand or restore to each JSON object in the stream.

It supports both newline-delimited JSON (NDJSON) and standard JSON array inputs through the json_array parameter, writing each transformed result as a JSON line to the output stream.

Architecture and Data Flow

The module implements a bidirectional pipeline that handles two primary transformation directions.

Flattening Pipeline

When converting nested JSON to flat structures, the pipeline executes in three stages:

1. Extraction: gen_leaf traverses the object tree, yielding every leaf value with its complete path
2. Path Serialization: expand joins path components using the configured separator
3. Safety Encoding: Optional escaping via utils.py helpers ensures keys containing the separator character remain intact

The resulting dictionary maps dot-notation strings to scalar values, ready for CSV column headers.

Restoration Pipeline

The reverse process reconstructs hierarchy from flat data:

1. Path Parsing: restore splits keys or decodes safe keys to regenerate path arrays
2. Grouping: from_leaf organizes tuples by head keys and recurses through subgroups
3. Type Detection: is_array_index determines whether each group becomes a list or dict

This automatic type detection eliminates the need for explicit schema mapping during reconstruction.

Handling Keys with Special Characters

JSON data frequently contains keys that include the dot character, which conflicts with the default path separator. The module addresses this through safety mode, implemented via encode_safe_key and decode_safe_key in jsoncsv/utils.py.

When safe=True is passed to expand or restore, the module escapes separator characters within keys before flattening and reverses the process during restoration. This ensures round-trip fidelity even for complex key names like "user.name" or "item.price".

Practical Examples

Flattening Nested Objects

Convert hierarchical data to a CSV-compatible flat dictionary:

from jsoncsv.jsontool import expand

nested = {
    "user": {"id": 1, "name": "Alice"},
    "orders": [{"id": 10, "total": 99.9}, {"id": 11, "total": 12.5}]
}

flat = expand(nested)
print(flat)

Output:

{
  'user.id': 1,
  'user.name': 'Alice',
  'orders.0.id': 10,
  'orders.0.total': 99.9,
  'orders.1.id': 11,
  'orders.1.total': 12.5
}

Restoring Flat Structures

Reconstruct the original hierarchy from dot-notation keys:

from jsoncsv.jsontool import restore

flat = {
    "user.id": 1,
    "user.name": "Alice",
    "orders.0.id": 10,
    "orders.0.total": 99.9,
    "orders.1.id": 11,
    "orders.1.total": 12.5
}

nested = restore(flat)
print(nested)

Result:

{'user': {'id': 1, 'name': 'Alice'},
 'orders': [{'id': 10, 'total': 99.9},
            {'id': 11, 'total': 12.5}]}

Processing JSON Streams

Transform line-delimited JSON files using the streaming interface:

from jsoncsv.jsontool import convert_json, expand

with open("input.ndjson", "r", encoding="utf-8") as fin, \
     open("output.ndjson", "w", encoding="utf-8") as fout:
    convert_json(fin, fout, func=expand, separator='.')

Safe Mode for Complex Keys

Handle keys containing the separator character without data corruption:

from jsoncsv.jsontool import expand, restore

obj = {"a.b": {"c": 1}}
flat_safe = expand(obj, safe=True)      # Encodes 'a.b' to avoid splitting

restored = restore(flat_safe, safe=True)
assert restored == obj

Summary

• The gen_leaf function recursively traverses JSON trees to extract path-value pairs, while from_leaf reconstructs hierarchies through recursive grouping.
• expand and restore provide the public API for bidirectional conversion between nested objects and flat dot-notation dictionaries.
• is_array_index automatically detects array structures during reconstruction, eliminating the need for explicit schema definitions.
• The convert_json function enables high-performance stream processing for large datasets and command-line integration.
• Safety mode via encode_safe_key and decode_safe_key ensures reliable round-trip conversion when keys contain separator characters.

Frequently Asked Questions

What is the primary purpose of jsontool.py in the jsoncsv library?

The jsontool.py module implements the core transformation engine that powers the entire jsoncsv workflow. It provides the mathematical operations required to flatten nested JSON into CSV-compatible dot-notation dictionaries and restore them back to hierarchical structures, handling both streaming and batch processing scenarios.

How does jsontool.py handle JSON arrays versus objects?

During the restoration phase, the is_array_index function analyzes the keys of each group to determine whether they form consecutive integer indexes. If the keys are integers (or string representations thereof), from_leaf instantiates a Python list; otherwise, it creates a dictionary. This automatic detection happens at every level of the reconstruction process.

Can jsontool.py process extremely large JSON files without loading everything into memory?

Yes, the convert_json function supports streaming processing by accepting file-like objects and processing JSON objects line-by-line. When reading newline-delimited JSON (NDJSON), it transforms each object individually and writes results immediately to the output stream, maintaining constant memory regardless of file size.

What happens if my JSON keys contain dots when using the default separator?

Without safety mode, dots within keys would be interpreted as path separators during restoration, corrupting the data structure. By passing safe=True to expand and restore, the module uses encode_safe_key to escape separator characters before flattening and decode_safe_key to restore them during unflattening, ensuring perfect round-trip fidelity.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}