# How Incremental Parsing Keeps the Code-Review-Graph Updated Efficiently > Discover how incremental parsing efficiently updates the code-review-graph by re-parsing only changed files and their dependencies. Avoid costly full rebuilds and streamline your workflow. - Repository: [Tirth Kanani/code-review-graph](https://github.com/tirth8205/code-review-graph) - Tags: internals - Published: 2026-08-09 --- **The incremental parsing mechanism in code-review-graph updates the graph by re-parsing only the files that have actually changed plus any files that depend on those changes, avoiding the cost of full repository rebuilds.** The `tirth8205/code-review-graph` repository implements a sophisticated incremental parsing system that maintains an up-to-date code dependency graph without requiring complete re-parsing after every commit. By leveraging git integration and graph traversal algorithms, the system identifies exactly which files need reprocessing and limits parser execution to only those specific files and their dependents. This approach dramatically reduces CPU and I/O overhead for large repositories while keeping the graph accurately synchronized with the source code. ## The Seven-Stage Incremental Parsing Pipeline The core workflow is orchestrated in [`code_review_graph/incremental.py`](https://github.com/tirth8205/code-review-graph/blob/main/code_review_graph/incremental.py) and consists of seven distinct stages that minimize unnecessary work while ensuring graph consistency. ### 1. Detect Changed Files via Git Integration The process begins with `get_changed_files()`, which executes `git diff --name-status -z` (or SVN equivalents) to obtain the list of modified, added, renamed, or deleted paths. The helper `_decode_name_status_paths()` normalizes the output so both old and new names of a renamed file are returned, ensuring no changes slip through during rename operations. This stage precisely identifies the initial set of files requiring attention. ### 2. Reconcile Stale Files Files that no longer exist or are now ignored are removed from the graph using `_reconcile_stale_files()`. This prevents orphaned nodes from persisting across updates and ensures the graph structure accurately reflects the current filesystem state. ### 3. Find Dependent Files Through Graph Traversal For each changed file, `find_dependents()` walks the graph edges—including imports, calls, and inheritance relationships—up to a configurable hop limit defined by `_MAX_DEPENDENT_HOPS`. This yields the complete set of files whose semantics may be affected by the original changes, ensuring that indirect impacts are captured without re-parsing the entire codebase. ### 4. Assemble the Work-List with Deduplication The system combines `changed_files ∪ dependent_files` into a unified *all_files* set. Files that are ignored, binary, or whose language is not recognized are filtered out. A cheap SHA-256 hash check then skips parsing of files whose content has not changed since the last successful parse, eliminating redundant work even when files appear in the diff due to timestamp changes. ### 5. Selective Parsing with Parallel Execution The remaining files are parsed either serially (for tiny repositories or when `CRG_SERIAL_PARSE=1` is set) or in parallel using a `ThreadPoolExecutor` or `ProcessPoolExecutor` selected by `_select_executor_kind()`. The actual parsing work is performed by `_parse_single_file()`, which reuses a thread-local `CodeParser` instance to avoid the overhead of fresh grammar initialization for each file. ### 6. Atomic Storage of Results Parsed nodes and edges are written to the SQLite-backed `GraphStore` via `store.store_file_nodes_edges()`. After all files are processed, deletion of missing files through `store.remove_files_permanently` and metadata updates (including `last_updated`, `last_build_type`, and VCS info) are committed in a single transaction, ensuring database consistency. ### 7. Run Selective Resolvers Language-specific post-processing—such as resolving Python imports or Spring dependency injection calls—is re-run only when at least one file of the corresponding language changed. This resolver gating mechanism avoids unnecessary whole-graph passes that would otherwise occur during every update. ## Incremental Parsing vs. Full Rebuilds While incremental parsing handles the common case of small changes efficiently, the system gracefully falls back to a full rebuild when the stored C++ identity version changes or when the stored commit SHA is no longer a valid diff base, detected via `resolve_incremental_base()`. The following example demonstrates the typical workflow: ```python from pathlib import Path from code_review_graph.graph import GraphStore from code_review_graph.incremental import full_build, incremental_update repo_root = Path("/path/to/repo") store = GraphStore(repo_root / ".code-review-graph/graph.db") # ------------------------------------------------- # 1️⃣ Full initial build (once per repository) # ------------------------------------------------- full_stats = full_build(repo_root, store) print("Full build parsed", full_stats["files_parsed"], "files") # ------------------------------------------------- # 2️⃣ Incremental update after a commit # ------------------------------------------------- inc_stats = incremental_update(repo_root, store) # defaults to HEAD~1 diff base print("Incremental updated", inc_stats["files_updated"], "files") ``` Running the above after a git commit touches only the files reported by `git diff` and any files that import from them, dramatically reducing CPU and I/O compared with a full rebuild. ## Key Implementation Files The incremental parsing system spans several critical modules: - [`code_review_graph/incremental.py`](https://github.com/tirth8205/code-review-graph/blob/main/code_review_graph/incremental.py) – Core incremental parsing logic, change detection, dependent discovery, and selective resolver orchestration. - [`code_review_graph/graph.py`](https://github.com/tirth8205/code-review-graph/blob/main/code_review_graph/graph.py) – SQLite-backed `GraphStore` implementation that persists nodes, edges, and metadata. - [`code_review_graph/parser.py`](https://github.com/tirth8205/code-review-graph/blob/main/code_review_graph/parser.py) – `CodeParser` abstraction that handles language detection and delegates to language-specific parsers (Tree-Sitter). - [`code_review_graph/registry.py`](https://github.com/tirth8205/code-review-graph/blob/main/code_review_graph/registry.py) – Per-repository configuration registry used by incremental functions. ## Summary - **Change Detection**: Uses `git diff --name-status -z` via `get_changed_files()` to identify exactly what changed in the working directory. - **Dependency Tracking**: `find_dependents()` traverses graph edges up to `_MAX_DEPENDENT_HOPS` to capture indirectly affected files. - **Work Deduplication**: SHA-256 hash checks prevent re-parsing files whose content remains unchanged despite appearing in diffs. - **Parallel Execution**: `_select_executor_kind()` chooses between thread and process pools to optimize parsing throughput. - **Atomic Updates**: All database writes occur in a single transaction to maintain graph consistency. - **Graceful Degradation**: Falls back to full rebuilds when the parser version changes or the baseline commit becomes invalid. ## Frequently Asked Questions ### What triggers a full rebuild instead of incremental parsing? The system initiates a full rebuild when the stored C++ identity version changes (indicating parser logic updates) or when `resolve_incremental_base()` determines the stored commit SHA is no longer a valid diff base, such as after a history rewrite or when the database is corrupted. ### How does incremental parsing handle renamed files in git? The `_decode_name_status_paths()` helper processes the output of `git diff --name-status` to capture both the old and new path names during rename operations. This ensures the graph correctly removes the old file reference while adding the new one, maintaining accurate dependency links. ### What limits the scope of dependent file discovery? The `find_dependents()` function respects `_MAX_DEPENDENT_HOPS`, a configurable limit that controls how many levels of graph edges (imports, calls, inheritance) to traverse when finding files affected by a change. This prevents the incremental update from expanding to the entire repository while still capturing meaningful indirect dependencies. ### How does incremental parsing optimize for different repository sizes? Small repositories or environments with `CRG_SERIAL_PARSE=1` use serial parsing to avoid process overhead, while larger codebases utilize `ThreadPoolExecutor` or `ProcessPoolExecutor` selected by `_select_executor_kind()`. Additionally, thread-local `CodeParser` instances prevent repeated grammar initialization costs during parallel processing.