How jsoncsv Handles Line-Delimited JSON: Stream Processing Explained

jsoncsv treats line-delimited JSON as a stream of independent objects, parsing each line individually with json.loads and yielding results through a generator to minimize memory overhead.

The jsoncsv library provides bidirectional conversion between hierarchical JSON documents and flat representations suitable for CSV export. Understanding how jsoncsv handles line-delimited JSON reveals its memory-efficient architecture for processing large datasets without loading entire files into RAM.

Stream-Based Processing Architecture

The core logic resides in jsoncsv/jsontool.py, specifically within the convert_json function. By default, the tool assumes input consists of one complete JSON object per line, commonly known as JSON Lines (JSONL) format.

The gen_objs Generator

When the --array flag is omitted (default behavior), convert_json instantiates an inner generator named gen_objs that iterates over the input stream line by line:

def gen_objs() -> Iterator[JsonType]:
    for line in fin:
        obj = json.loads(line)   # ← parses one JSON object per line

        yield obj

This generator is assigned to the objs variable and consumed lazily in a subsequent loop. Source: jsoncsv/jsontool.py lines 40-45.

Per-Object Conversion and Output

For each object yielded by the generator, convert_json applies the selected transformation function and serializes the result back to line-delimited format:

for obj in objs:
    new = func(obj, separator=separator, safe=safe)
    content = json.dumps(new, ensure_ascii=False)
    fout.write(content)
    fout.write("\n")

The func parameter refers to either expand (flattening nested structures) or restore (reconstructing nested objects). Source: jsoncsv/jsontool.py lines 57-61.

Array Mode vs. Line-Delimited Mode

jsoncsv supports two distinct input parsing strategies controlled by the -A or --array command-line flag.

Default Line-Delimited Behavior

Without the --array flag, jsoncsv processes files as JSON Lines. This approach maintains constant memory usage regardless of file size, as only one JSON object resides in memory at any given time. The gen_objs generator reads from fin (the input file handle) and yields parsed objects individually.

Bulk Array Processing

When the user supplies the -A/--array option, convert_json bypasses the line-by-line generator in favor of gen_objs_from_array, which loads the entire input as a single JSON array via json.load(fin):

if array:
    objs = gen_objs_from_array(fin)
else:
    objs = gen_objs()

This mode is suitable for smaller files where the input is formatted as a standard JSON array rather than newline-separated objects. Source: jsoncsv/jsontool.py lines 48-55.

Practical Usage Examples

Expand a Line-Delimited JSON File

Given an input file with one object per line:

# input.txt

{"id":1,"name":"Alice"}
{"id":2,"name":"Bob"}

Execute the expansion (flattening) operation:

cat input.txt | python -m jsoncsv jsoncsv -e - -

Output remains line-delimited, with each line containing the expanded version of the corresponding input object.

Restore Flattened Structures

To reverse the expansion process and reconstruct nested objects:

cat expanded.txt | python -m jsoncsv jsoncsv --restore - -

Process Standard JSON Arrays

For files containing a JSON array rather than line-delimited objects:

# array.json

[ {"id":1,"info":{"age":30}},
  {"id":2,"info":{"age":25}} ]

cat array.json | python -m jsoncsv jsoncsv -A -e - -

The -A flag instructs jsoncsv to parse the entire file as a single JSON array before processing individual elements.

CLI Integration

The command-line entry point in jsoncsv/main.py (lines 46-50) handles argument parsing and stream initialization before delegating to convert_json. It opens input and output file handles and passes them directly to the conversion logic, ensuring that the streaming architecture works transparently whether reading from files or Unix pipes.

Summary

• jsoncsv processes line-delimited JSON by streaming individual lines through the gen_objs generator in jsontool.py, maintaining low memory footprint for large files.
• Each line is parsed independently using json.loads, transformed via expand or restore, and written back with a trailing newline to preserve the line-delimited format.
• The --array / -A flag switches to bulk mode, loading the entire input as a JSON array via json.load() instead of streaming line by line.
• Default behavior optimizes for JSON Lines format (one object per line), while array mode accommodates traditional JSON array structures.

Frequently Asked Questions

Does jsoncsv load the entire line-delimited file into memory?

No. By default, jsoncsv uses a generator-based approach in convert_json that yields one object at a time. Only the current line is parsed into memory using json.loads, making it suitable for processing gigabyte-scale line-delimited JSON files on modest hardware.

How do I process a standard JSON array instead of line-delimited objects?

Use the -A or --array flag when invoking jsoncsv. This switches the input parser from the line-by-line gen_objs generator to gen_objs_from_array, which calls json.load(fin) to read the entire file as a single JSON array, then iterates over its elements.

What conversion functions are applied to each line?

Each parsed object passes through either the expand function (which flattens nested dictionaries using dot notation) or the restore function (which reconstructs nested structures from flattened keys). The active function is determined by command-line arguments and passed as the func parameter to convert_json.

Where is the line-delimited processing logic implemented?

The primary implementation resides in jsoncsv/jsontool.py within the convert_json function (lines 40-61). This module defines the gen_objs generator for line-by-line streaming and handles the output serialization that maintains newline separation between objects.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}