Architecture of the jsoncsv Python Tool: A 4-Layer Deep Dive

The jsoncsv Python tool follows a modular four-layer architecture comprising a CLI dispatcher, a core JSON processor for flattening and restoring nested data, a pluggable dump engine for CSV/XLS serialization, and a utility layer for type safety and key encoding.

The jsoncsv package is a lightweight open-source command-line utility that transforms nested JSON into flat CSV or Excel tables and vice versa. As implemented in the alingse/jsoncsv repository, its architecture demonstrates clean separation of concerns across four distinct logical layers. Understanding this structure helps developers extend the tool with new output formats or integrate its functionality into data processing pipelines.

CLI Layer: Command Dispatch (jsoncsv/main.py)

The entry point defines two Click commands that parse arguments and dispatch work to the appropriate internal functions.

The jsoncsv Command

This command handles JSON expansion and restoration. It validates options like --expand, --restore, --safe, and --array, then calls jsontool.convert_json with the selected processing function. The command explicitly closes input and output streams to ensure deterministic I/O behavior.

The mkexcel Command

This command creates CSV or XLS files from JSON streams. It selects the appropriate dumper class—either DumpCSV or DumpXLS—and invokes dumptool.dump_excel to handle the serialization. Both commands reside in jsoncsv/main.py and serve as thin wrappers around the core processing logic.

Core JSON Processor: Flattening Logic (jsoncsv/jsontool.py)

The jsoncsv/jsontool.py module contains the tree-walking algorithms that convert nested JSON into flat key-value maps and back again.

Key Processing Functions

gen_leaf recursively traverses JSON objects and yields (path, value) tuples for every leaf node. is_array_index detects whether a collection of path segments represents a list by checking if all indices are numeric.

expand flattens a JSON object into a dictionary where keys are dot-separated paths (e.g., user.address.zip). restore performs the inverse operation, reconstructing nested structures from flat representations using from_leaf, which reassembles objects from path tuples.

convert_json streams JSON lines from an input file, applies either expand or restore to each object, and writes the results to an output file. It handles the --array flag for processing entire JSON arrays instead of line-delimited objects.

Dump Engine: CSV and XLS Output (jsoncsv/dumptool.py)

The dump subsystem in jsoncsv/dumptool.py uses an abstract-base-class pattern to support multiple output formats through a common interface.

Class Hierarchy

Dump serves as the abstract skeleton defining initialize, prepare, dump_file, and on_finish methods. ReadHeadersMixin provides static utility methods that scan the first N rows to collect all possible keys and optionally sort them.

DumpExcel combines the abstract Dump class with ReadHeadersMixin, storing headers and data before writing. Concrete implementations DumpCSV and DumpXLS inherit from this hierarchy. DumpCSV writes data using Python’s csv.DictWriter, while DumpXLS creates spreadsheets via xlwt.

The dump_excel function validates the selected dumper class and executes the generic workflow: instantiate the dumper, prepare headers, and stream rows to the output file.

Utility Layer: Safe Key Handling (jsoncsv/utils.py)

The jsoncsv/utils.py module provides shared infrastructure for type safety and key encoding.

Type aliases like JsonType, PathType, and LeafType enable static typing throughout the codebase. When users enable --safe mode to handle keys containing the separator character, the module's encode_safe_key and decode_safe_key functions escape dots with backslashes (e.g., key\.name).

This prevents key collisions when flattening JSON objects where field names themselves contain dot characters.

Practical Code Examples

Expanding Nested JSON to Flat Structure

python -m jsoncsv jsoncsv data.json expanded.json

This processes data.json line-by-line, converting nested objects into flat dictionaries with dot-separated keys like address.street.

Restoring Flat JSON to Nested Structure

python -m jsoncsv jsoncsv --restore expanded.json restored.json

This reverses the flattening operation, reconstructing the original hierarchical JSON using the restore function.

Using Safe Mode for Keys with Separators

python -m jsoncsv jsoncsv --safe data.json safe_expanded.json

Keys containing . are escaped as \. using encode_safe_key, preventing ambiguity during the flattening process.

Converting JSON to CSV

python -m jsoncsv mkexcel data.json table.csv

Behind the scenes, DumpCSV reads the first lines to infer headers via ReadHeadersMixin, then streams each object as a CSV row.

Creating Excel Files with Sorted Headers

python -m jsoncsv mkexcel -t xls --sort data.json table.xls

DumpXLS creates an xlwt.Workbook, writes sorted headers in the first row, and fills subsequent rows with data.

Summary

The architecture of the jsoncsv Python tool separates concerns into four distinct layers:

• CLI Layer (jsoncsv/main.py): Parses command-line arguments via Click and dispatches to core functions
• Core Processor (jsoncsv/jsontool.py): Implements tree-walking algorithms (gen_leaf, expand, restore) for JSON flattening and reconstruction
• Dump Engine (jsoncsv/dumptool.py): Provides an extensible framework with abstract Dump classes and concrete DumpCSV/DumpXLS implementations
• Utility Layer (jsoncsv/utils.py): Supplies type definitions and safe-key encoding to handle edge cases in key names

This modular design allows developers to add new output formats by extending the Dump class hierarchy, while keeping the command-line interface lightweight and predictable.

Frequently Asked Questions

How does jsoncsv handle deeply nested JSON objects?

The gen_leaf function in jsoncsv/jsontool.py recursively traverses the JSON tree, yielding (path, value) tuples for every leaf node regardless of nesting depth. The expand function accumulates these paths into dot-separated keys, preserving the full hierarchical context in the flat output.

What is the difference between the expand and restore functions in jsoncsv?

expand flattens a nested JSON object into a dictionary with dot-separated keys (e.g., {"user": {"name": "John"}} becomes {"user.name": "John"}). restore performs the inverse operation, using from_leaf to rebuild nested structures from flat key-value pairs by splitting paths and reconstructing objects and arrays.

How can I add a new output format to jsoncsv?

Create a new class inheriting from Dump (or DumpExcel if you need header reading) in jsoncsv/dumptool.py. Implement the write_headers and write_obj methods, then register your class in the CLI's mkexcel command in jsoncsv/main.py. The dump_excel function handles the generic workflow, so your implementation only needs to manage format-specific serialization.

What does the --safe mode do in jsoncsv?

The --safe flag activates key encoding in jsoncsv/utils.py. When enabled, encode_safe_key escapes separator characters (dots) with backslashes, allowing JSON keys that contain dots to be flattened unambiguously. The decode_safe_key function reverses this process during restoration, ensuring data integrity for keys like version.1.0.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}