Incremental Graph Updating in code-review-graph: Architecture and Performance

Incremental graph updating re-parses only modified files and their transitive dependents, using SHA-256 content hashing and parallel processing to update the knowledge graph in under a second for typical development cycles.

The tirth8205/code-review-graph project constructs a knowledge graph of code repositories by parsing source files into nodes and edges. For large codebases, full rebuilds are prohibitively expensive, so the system implements incremental graph updating to maintain near-real-time accuracy. This approach processes only files that changed since the last update plus any files that depend on them, dramatically reducing computation time while ensuring graph consistency.

How Incremental Graph Updating Works

Detecting File Changes via VCS

The process begins in code_review_graph/incremental.py where the incremental_update function calls get_changed_files to execute a VCS diff. For Git repositories, this runs git diff --name-only -z to identify added, modified, or deleted files since the base revision. When the changed_files parameter is not explicitly supplied, the system automatically detects changes by comparing against stored VCS metadata.

Traversing the Dependency Graph

For each changed file, the find_dependents function queries the graph's edge tables to locate files that import from or otherwise reference the modified file. The search traverses up to _MAX_DEPENDENT_HOPS hops through the dependency graph, returning a DependentList that may be truncated if the set exceeds _MAX_DEPENDENT_FILES. This ensures semantic changes propagate to all affected downstream code without re-parsing the entire repository.

Content Hashing and Short-Circuiting

Before parsing, the system filters the work set through two mechanisms. First, _should_ignore checks files against repository-wide ignore patterns and validates language detection via parser.detect_language. Deleted files are immediately removed from the graph using store.remove_file_data. For remaining files, the system computes a SHA-256 hash of current contents and compares it against the hash stored in the GraphStore. If the hashes match, the file is skipped entirely, avoiding redundant parsing even if the file appeared in the dependency search.

Parallel Parsing and Atomic Storage

Files requiring parsing are processed either serially (when the environment variable CRG_SERIAL_PARSE=1 is set) or in parallel using a ProcessPoolExecutor or ThreadPoolExecutor selected by _select_executor_kind. The _parse_single_file function handles Tree-sitter parsing, returning nodes, edges, and the new file hash. With up to _MAX_PARSE_WORKERS (default 8) concurrent workers, CPU-bound parsing scales linearly with available cores. All mutations are wrapped in a single SQLite transaction via store.commit to maintain consistency.

Selective Language Resolver Execution

After the primary update, the system conditionally executes language-specific resolvers for ReScript, Spring, Temporal, and HCL only if files of those languages changed. This avoids unnecessary computational work when updates are limited to other languages.

How Fast Is Incremental Graph Updating?

The performance advantages of incremental graph updating stem from three architectural optimizations:

• Work-set reduction: Instead of parsing all tracked files via collect_all_files, the incremental path processes only changed files plus their dependents. In typical development cycles, this reduces the work set from thousands of files to a handful.

• Hash-based deduplication: The SHA-256 content check eliminates parsing for dependent files whose contents remain unchanged, further shrinking the computational load regardless of dependency graph complexity.

• Parallel execution: When the work set exceeds eight files, parsing distributes across up to eight workers, achieving near-linear speedup on multi-core machines where Tree-sitter parsing releases the GIL.

Empirical benchmarks in code_review_graph/token_benchmark.py demonstrate that incremental updates complete in under one second for a few dozen changed files on modern 8-core hardware. In contrast, a full rebuild of a 10,000-file repository takes tens of seconds, making incremental updating essential for interactive development workflows.

Running Incremental Updates

You can trigger an incremental update programmatically or via the CLI.

Using Python:

from pathlib import Path
from code_review_graph.incremental import incremental_update
from code_review_graph.graph import GraphStore

repo_root = Path("/path/to/your/repo")
store = GraphStore()                     # Open the SQLite graph store

result = incremental_update(repo_root, store)
print("Updated files:", result["files_updated"])
print("Changed files:", result["changed_files"])
print("Dependent files:", result["dependent_files"])

Using the command line:

$ crg incremental --repo /path/to/repo

The CLI command invokes incremental_update with the default base of HEAD~1, processing changes since the previous commit.

Summary

• Incremental graph updating in tirth8205/code-review-graph limits re-parsing to changed files and their transitive dependents, implemented primarily in code_review_graph/incremental.py.
• The system uses VCS diffs (git diff --name-only) to detect changes and traverses dependency edges via find_dependents to locate affected files up to configurable hop and count limits.
• SHA-256 content hashing skips files that haven't changed, while parallel parsing with up to 8 workers accelerates CPU-bound operations.
• Language-specific resolvers run only when needed, and all updates occur within a single SQLite transaction for consistency.
• Typical updates complete in under a second compared to tens of seconds for full repository rebuilds.

Frequently Asked Questions

What triggers an incremental graph update in code-review-graph?

An incremental update triggers when you call incremental_update() with a repository path, or run the crg incremental CLI command. By default, the system compares the current working directory against HEAD~1 using get_changed_files, though you can supply a specific changed_files list to bypass VCS detection and manually specify the work set.

How does code-review-graph handle deleted files during incremental updates?

During the filtering phase in incremental.py, the system checks file existence on disk. Files that no longer exist are removed from the graph using store.remove_file_data before the parsing phase begins, ensuring the knowledge graph stays synchronized with the repository state without retaining stale nodes.

What is the maximum number of dependent files processed in an incremental update?

The find_dependents function respects two limits defined in incremental.py: _MAX_DEPENDENT_HOPS controls the traversal depth through the dependency graph, while _MAX_DEPENDENT_FILES caps the total number of dependents returned. If the dependent set exceeds this threshold, the list is truncated to prevent unbounded growth during updates.

Can incremental graph updating run in serial mode?

Yes. Setting the environment variable CRG_SERIAL_PARSE=1 forces the system to use serial parsing instead of parallel execution. This is useful for debugging, very small repositories where process overhead exceeds parallel benefits, or environments with limited resources where process pool creation is undesirable.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}