How code-review-graph Handles Parsing for Jupyter and Databricks Notebooks

code-review-graph treats notebook files as first-class source types by detecting .ipynb extensions and Databricks Python exports, extracting executable code cells while filtering magic commands, and feeding normalized source into language-specific Tree-sitter parsers to generate unified dependency graphs.

The code-review-graph repository provides a unified parsing pipeline that handles both standard Jupyter notebooks and Databricks notebook exports as native source files. By treating .ipynb JSON files and Databricks .py scripts as parseable code artifacts, the system enables dependency analysis and code review across polyglot notebook environments. This article examines the implementation details of how the parser detects, processes, and converts notebook cells into graph structures.

File Type Detection and Routing

The parsing workflow begins with extension-based language detection defined in code_review_graph/parser.py. At line 689, the extension mapping registers .ipynb as the language "notebook", triggering specialized handling for JSON-based Jupyter files.

When the parse() method encounters a file whose detected language is "notebook", it routes the input to the _parse_notebook method (lines 2470–2471). For Databricks exports saved as .py files, the parser checks the first line for the header # Databricks notebook source and diverts to _parse_databricks_py_notebook instead (lines 2471–2473).

Jupyter Notebook Parsing Pipeline

Kernel Detection and Cell Extraction

The _parse_notebook method loads the JSON payload and extracts the kernel language from the notebook metadata, defaulting to Python when unspecified (lines 2915–2920). Currently, the parser only supports Python and R kernels (lines 2423–2426).

The method iterates over the cells array, keeping only code cells while discarding markdown and raw cells. For each code cell, it examines magic directives to determine the effective programming language.

Magic Command Processing

Cell-level language detection handles Jupyter magic commands through pattern matching. The parser looks for magic directives including %python, %sql, and %r to override the kernel language for that specific cell (lines 2828–2835). Conversely, it drops cells containing skip magics such as %scala, %md, or %sh.

For Python and R cells, the parser strips lines beginning with % or ! (lines 3262–3268), removing shell-style commands that should not be parsed as executable code. The cleaned source of each valid cell is encapsulated in a CellInfo object (lines 2744–2749), preserving the original cell index and language metadata.

Databricks Python Export Parsing

Header Detection and Cell Separation

Databricks notebooks exported as .py files use a distinct format recognized by the header comment # Databricks notebook source. When detected, the parser invokes _parse_databricks_py_notebook (lines 3125–3130), strips the header, and splits the file on # COMMAND ---------- delimiters to reconstruct logical cells (lines 3137–3143).

MAGIC Directive Handling

Each extracted cell may contain # MAGIC prefixes indicating the cell's language, such as # MAGIC %sql or # MAGIC %r. The parser ignores cells marked with # MAGIC %md or # MAGIC %sh. For Python cells, it removes the # MAGIC prefix and stores the remaining source as a CellInfo with language "python" (lines 4445–4470).

After processing, the collected cells feed into the shared parsing pipeline, and the resulting File node receives the extra metadata field notebook_format="databricks_py" (lines 3215–3219).

Unified Cell Processing and Graph Generation

Language Grouping and AST Parsing

Both Jupyter and Databricks paths converge on _parse_notebook_cells, which groups cells by language (python, r, sql). For Python and R groups, the concatenated source strings are submitted to the Tree-sitter parser for that language. Line numbers are remapped so that each language group starts at line 1, with each cell's offset recorded to maintain accurate source references (lines 2810–2830).

SQL cells undergo lightweight analysis using regex patterns that extract table names and generate IMPORTS_FROM edges (lines 3220–3227), bypassing the full AST parsing used for general-purpose languages.

Node Construction and Notebook Metadata

The routine returns NodeInfo and EdgeInfo objects representing classes, functions, imports, and calls, augmented with notebook-specific metadata such as cell_index (lines 3010–3031). The final File node records the original notebook path, marks test files when appropriate, and attaches format-specific extras, enabling the dependency analysis pipeline to treat notebooks identically to standard source files.

Practical Usage Example

from code_review_graph.parser import CodeParser
from pathlib import Path

# Parse a Jupyter notebook

parser = CodeParser()
nodes, edges = parser.parse(Path("example.ipynb"))

# Each node includes language, line numbers, and cell metadata

for n in nodes:
    print(n.kind, n.name, n.language, n.extra.get("cell_index"))

# Parse a Databricks .py notebook export

db_nodes, db_edges = parser.parse(Path("export.py"))

# The File node carries notebook_format="databricks_py"

print(db_nodes[0].extra["notebook_format"])

The parse method internally chains _detect_language → language-specific dispatch → _parse_notebook or _parse_databricks_py_notebook → _parse_notebook_cells, producing a unified graph representation regardless of the original format.

Summary

• code-review-graph registers .ipynb files as the "notebook" language type in code_review_graph/parser.py (line 689) and detects Databricks exports via header inspection (lines 2471–2473).

• Jupyter parsing extracts Python and R code cells while filtering magic commands (%scala, %md, %sh) and shell prefixes to produce clean CellInfo objects (lines 2744–2749).

• Databricks .py exports split on # COMMAND ---------- delimiters and process # MAGIC directives to normalize cells before shared processing (lines 3137–3143).

• The unified _parse_notebook_cells routine groups code by language, remaps line numbers, and generates NodeInfo/EdgeInfo objects compatible with the standard graph pipeline (lines 3010–3031).

Frequently Asked Questions

What notebook formats does code-review-graph support?

The parser supports standard Jupyter notebooks (.ipynb files) and Databricks notebook exports saved as .py files. For Jupyter files, it handles JSON structures with Python and R kernels (lines 2423–2426). For Databricks, it recognizes the # Databricks notebook source header and COMMAND delimiters (lines 2471–2473).

How does the parser handle magic commands like %sql or %md?

Magic commands are processed differently based on type. Active magics like %python, %sql, or %r override the cell language, while skip magics including %md, %scala, and %sh cause the cell to be excluded from parsing (lines 2828–2835). In Databricks exports, the # MAGIC prefix is stripped before language detection (lines 4445–4470).

Can code-review-graph parse notebooks with mixed languages (Python and R)?

Yes. The parser supports polyglot notebooks by grouping cells by detected language after processing magic directives. Python and R cells are fed to their respective Tree-sitter parsers, while SQL cells receive specialized regex handling (lines 2810–2830). Each cell retains its cell_index metadata to preserve notebook structure.

How are line numbers preserved when parsing notebook cells?

The parser remaps line numbers so that each language group starts at line 1, while recording each cell's offset within the original notebook (lines 2810–2830). This allows accurate cross-referencing between the generated graph nodes and the source notebook locations, even when cells are concatenated for language-specific parsing.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}