# How jsoncsv Handles Line-Delimited JSON: Stream Processing Explained

> Discover how jsoncsv handles line-delimited JSON using efficient stream processing. Learn to parse independent JSON objects with minimal memory.

- Repository: [alingse/jsoncsv](https://github.com/alingse/jsoncsv)
- Tags: internals
- Published: 2026-02-24

---

**jsoncsv treats line-delimited JSON as a stream of independent objects, parsing each line individually with `json.loads` and yielding results through a generator to minimize memory overhead.**

The `jsoncsv` library provides bidirectional conversion between hierarchical JSON documents and flat representations suitable for CSV export. Understanding **how jsoncsv handles line-delimited JSON** reveals its memory-efficient architecture for processing large datasets without loading entire files into RAM.

## Stream-Based Processing Architecture

The core logic resides in **[`jsoncsv/jsontool.py`](https://github.com/alingse/jsoncsv/blob/main/jsoncsv/jsontool.py)**, specifically within the `convert_json` function. By default, the tool assumes input consists of one complete JSON object per line, commonly known as JSON Lines (JSONL) format.

### The gen_objs Generator

When the `--array` flag is omitted (default behavior), `convert_json` instantiates an inner generator named `gen_objs` that iterates over the input stream line by line:

```python
def gen_objs() -> Iterator[JsonType]:
    for line in fin:
        obj = json.loads(line)   # ← parses one JSON object per line

        yield obj

```

This generator is assigned to the `objs` variable and consumed lazily in a subsequent loop. *Source:* [`jsoncsv/jsontool.py`](https://github.com/alingse/jsoncsv/blob/main/jsoncsv/jsontool.py) lines 40-45.

### Per-Object Conversion and Output

For each object yielded by the generator, `convert_json` applies the selected transformation function and serializes the result back to line-delimited format:

```python
for obj in objs:
    new = func(obj, separator=separator, safe=safe)
    content = json.dumps(new, ensure_ascii=False)
    fout.write(content)
    fout.write("\n")

```

The `func` parameter refers to either `expand` (flattening nested structures) or `restore` (reconstructing nested objects). *Source:* [`jsoncsv/jsontool.py`](https://github.com/alingse/jsoncsv/blob/main/jsoncsv/jsontool.py) lines 57-61.

## Array Mode vs. Line-Delimited Mode

jsoncsv supports two distinct input parsing strategies controlled by the `-A` or `--array` command-line flag.

### Default Line-Delimited Behavior

Without the `--array` flag, jsoncsv processes files as JSON Lines. This approach maintains constant memory usage regardless of file size, as only one JSON object resides in memory at any given time. The `gen_objs` generator reads from `fin` (the input file handle) and yields parsed objects individually.

### Bulk Array Processing

When the user supplies the `-A/--array` option, `convert_json` bypasses the line-by-line generator in favor of `gen_objs_from_array`, which loads the entire input as a single JSON array via `json.load(fin)`:

```python
if array:
    objs = gen_objs_from_array(fin)
else:
    objs = gen_objs()

```

This mode is suitable for smaller files where the input is formatted as a standard JSON array rather than newline-separated objects. *Source:* [`jsoncsv/jsontool.py`](https://github.com/alingse/jsoncsv/blob/main/jsoncsv/jsontool.py) lines 48-55.

## Practical Usage Examples

### Expand a Line-Delimited JSON File

Given an input file with one object per line:

```bash

# input.txt

{"id":1,"name":"Alice"}
{"id":2,"name":"Bob"}

```

Execute the expansion (flattening) operation:

```bash
cat input.txt | python -m jsoncsv jsoncsv -e - -

```

Output remains line-delimited, with each line containing the expanded version of the corresponding input object.

### Restore Flattened Structures

To reverse the expansion process and reconstruct nested objects:

```bash
cat expanded.txt | python -m jsoncsv jsoncsv --restore - -

```

### Process Standard JSON Arrays

For files containing a JSON array rather than line-delimited objects:

```bash

# array.json

[ {"id":1,"info":{"age":30}},
  {"id":2,"info":{"age":25}} ]

cat array.json | python -m jsoncsv jsoncsv -A -e - -

```

The `-A` flag instructs jsoncsv to parse the entire file as a single JSON array before processing individual elements.

## CLI Integration

The command-line entry point in **[`jsoncsv/main.py`](https://github.com/alingse/jsoncsv/blob/main/jsoncsv/main.py)** (lines 46-50) handles argument parsing and stream initialization before delegating to `convert_json`. It opens input and output file handles and passes them directly to the conversion logic, ensuring that the streaming architecture works transparently whether reading from files or Unix pipes.

## Summary

- **jsoncsv** processes line-delimited JSON by streaming individual lines through the `gen_objs` generator in [`jsontool.py`](https://github.com/alingse/jsoncsv/blob/main/jsontool.py), maintaining low memory footprint for large files.
- Each line is parsed independently using `json.loads`, transformed via `expand` or `restore`, and written back with a trailing newline to preserve the line-delimited format.
- The `--array` / `-A` flag switches to bulk mode, loading the entire input as a JSON array via `json.load()` instead of streaming line by line.
- Default behavior optimizes for JSON Lines format (one object per line), while array mode accommodates traditional JSON array structures.

## Frequently Asked Questions

### Does jsoncsv load the entire line-delimited file into memory?

No. By default, jsoncsv uses a generator-based approach in `convert_json` that yields one object at a time. Only the current line is parsed into memory using `json.loads`, making it suitable for processing gigabyte-scale line-delimited JSON files on modest hardware.

### How do I process a standard JSON array instead of line-delimited objects?

Use the `-A` or `--array` flag when invoking jsoncsv. This switches the input parser from the line-by-line `gen_objs` generator to `gen_objs_from_array`, which calls `json.load(fin)` to read the entire file as a single JSON array, then iterates over its elements.

### What conversion functions are applied to each line?

Each parsed object passes through either the `expand` function (which flattens nested dictionaries using dot notation) or the `restore` function (which reconstructs nested structures from flattened keys). The active function is determined by command-line arguments and passed as the `func` parameter to `convert_json`.

### Where is the line-delimited processing logic implemented?

The primary implementation resides in **[`jsoncsv/jsontool.py`](https://github.com/alingse/jsoncsv/blob/main/jsoncsv/jsontool.py)** within the `convert_json` function (lines 40-61). This module defines the `gen_objs` generator for line-by-line streaming and handles the output serialization that maintains newline separation between objects.