How Incremental Parsing Keeps the Code-Review-Graph Updated Efficiently

The incremental parsing mechanism in code-review-graph updates the graph by re-parsing only the files that have actually changed plus any files that depend on those changes, avoiding the cost of full repository rebuilds.

The tirth8205/code-review-graph repository implements a sophisticated incremental parsing system that maintains an up-to-date code dependency graph without requiring complete re-parsing after every commit. By leveraging git integration and graph traversal algorithms, the system identifies exactly which files need reprocessing and limits parser execution to only those specific files and their dependents. This approach dramatically reduces CPU and I/O overhead for large repositories while keeping the graph accurately synchronized with the source code.

The Seven-Stage Incremental Parsing Pipeline

The core workflow is orchestrated in code_review_graph/incremental.py and consists of seven distinct stages that minimize unnecessary work while ensuring graph consistency.

1. Detect Changed Files via Git Integration

The process begins with get_changed_files(), which executes git diff --name-status -z (or SVN equivalents) to obtain the list of modified, added, renamed, or deleted paths. The helper _decode_name_status_paths() normalizes the output so both old and new names of a renamed file are returned, ensuring no changes slip through during rename operations. This stage precisely identifies the initial set of files requiring attention.

2. Reconcile Stale Files

Files that no longer exist or are now ignored are removed from the graph using _reconcile_stale_files(). This prevents orphaned nodes from persisting across updates and ensures the graph structure accurately reflects the current filesystem state.

3. Find Dependent Files Through Graph Traversal

For each changed file, find_dependents() walks the graph edges—including imports, calls, and inheritance relationships—up to a configurable hop limit defined by _MAX_DEPENDENT_HOPS. This yields the complete set of files whose semantics may be affected by the original changes, ensuring that indirect impacts are captured without re-parsing the entire codebase.

4. Assemble the Work-List with Deduplication

The system combines changed_files ∪ dependent_files into a unified all_files set. Files that are ignored, binary, or whose language is not recognized are filtered out. A cheap SHA-256 hash check then skips parsing of files whose content has not changed since the last successful parse, eliminating redundant work even when files appear in the diff due to timestamp changes.

5. Selective Parsing with Parallel Execution

The remaining files are parsed either serially (for tiny repositories or when CRG_SERIAL_PARSE=1 is set) or in parallel using a ThreadPoolExecutor or ProcessPoolExecutor selected by _select_executor_kind(). The actual parsing work is performed by _parse_single_file(), which reuses a thread-local CodeParser instance to avoid the overhead of fresh grammar initialization for each file.

6. Atomic Storage of Results

Parsed nodes and edges are written to the SQLite-backed GraphStore via store.store_file_nodes_edges(). After all files are processed, deletion of missing files through store.remove_files_permanently and metadata updates (including last_updated, last_build_type, and VCS info) are committed in a single transaction, ensuring database consistency.

7. Run Selective Resolvers

Language-specific post-processing—such as resolving Python imports or Spring dependency injection calls—is re-run only when at least one file of the corresponding language changed. This resolver gating mechanism avoids unnecessary whole-graph passes that would otherwise occur during every update.

Incremental Parsing vs. Full Rebuilds

While incremental parsing handles the common case of small changes efficiently, the system gracefully falls back to a full rebuild when the stored C++ identity version changes or when the stored commit SHA is no longer a valid diff base, detected via resolve_incremental_base().

The following example demonstrates the typical workflow:

from pathlib import Path
from code_review_graph.graph import GraphStore
from code_review_graph.incremental import full_build, incremental_update

repo_root = Path("/path/to/repo")
store = GraphStore(repo_root / ".code-review-graph/graph.db")

# -------------------------------------------------

# 1️⃣ Full initial build (once per repository)

# -------------------------------------------------

full_stats = full_build(repo_root, store)
print("Full build parsed", full_stats["files_parsed"], "files")

# -------------------------------------------------

# 2️⃣ Incremental update after a commit

# -------------------------------------------------

inc_stats = incremental_update(repo_root, store)   # defaults to HEAD~1 diff base

print("Incremental updated", inc_stats["files_updated"], "files")

Running the above after a git commit touches only the files reported by git diff and any files that import from them, dramatically reducing CPU and I/O compared with a full rebuild.

Key Implementation Files

The incremental parsing system spans several critical modules:

• code_review_graph/incremental.py – Core incremental parsing logic, change detection, dependent discovery, and selective resolver orchestration.
• code_review_graph/graph.py – SQLite-backed GraphStore implementation that persists nodes, edges, and metadata.
• code_review_graph/parser.py – CodeParser abstraction that handles language detection and delegates to language-specific parsers (Tree-Sitter).
• code_review_graph/registry.py – Per-repository configuration registry used by incremental functions.

Summary

• Change Detection: Uses git diff --name-status -z via get_changed_files() to identify exactly what changed in the working directory.
• Dependency Tracking: find_dependents() traverses graph edges up to _MAX_DEPENDENT_HOPS to capture indirectly affected files.
• Work Deduplication: SHA-256 hash checks prevent re-parsing files whose content remains unchanged despite appearing in diffs.
• Parallel Execution: _select_executor_kind() chooses between thread and process pools to optimize parsing throughput.
• Atomic Updates: All database writes occur in a single transaction to maintain graph consistency.
• Graceful Degradation: Falls back to full rebuilds when the parser version changes or the baseline commit becomes invalid.

Frequently Asked Questions

What triggers a full rebuild instead of incremental parsing?

The system initiates a full rebuild when the stored C++ identity version changes (indicating parser logic updates) or when resolve_incremental_base() determines the stored commit SHA is no longer a valid diff base, such as after a history rewrite or when the database is corrupted.

How does incremental parsing handle renamed files in git?

The _decode_name_status_paths() helper processes the output of git diff --name-status to capture both the old and new path names during rename operations. This ensures the graph correctly removes the old file reference while adding the new one, maintaining accurate dependency links.

What limits the scope of dependent file discovery?

The find_dependents() function respects _MAX_DEPENDENT_HOPS, a configurable limit that controls how many levels of graph edges (imports, calls, inheritance) to traverse when finding files affected by a change. This prevents the incremental update from expanding to the entire repository while still capturing meaningful indirect dependencies.

How does incremental parsing optimize for different repository sizes?

Small repositories or environments with CRG_SERIAL_PARSE=1 use serial parsing to avoid process overhead, while larger codebases utilize ThreadPoolExecutor or ProcessPoolExecutor selected by _select_executor_kind(). Additionally, thread-local CodeParser instances prevent repeated grammar initialization costs during parallel processing.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}