How the Incremental Engine in code-review-graph Detects File Changes Using SHA-256 Hashes

The incremental engine combines VCS diffing with SHA-256 checksums to skip parsing files whose byte content remains identical, avoiding redundant analysis work.

The code-review-graph repository implements a high-performance incremental analysis system that minimizes re-parsing overhead in large codebases. When processing repository updates, the engine located in code_review_graph/incremental.py employs a two-stage filtering mechanism that first identifies candidate files through version control, then verifies actual content changes using cryptographic hashing before invoking the parser.

Two-Stage Change Detection Process

The incremental update workflow relies on distinct phases to narrow the scope of files requiring re-analysis.

Stage 1: VCS-Based Candidate Selection

The engine begins by querying the version control system to establish an initial set of potentially modified files. The get_changed_files() function executes git diff --name-only -z (or SVN equivalent commands) to retrieve files changed since a reference commit (default HEAD~1).

Additionally, the system calculates dependent files—modules that import from changed files—to ensure structural relationship updates propagate through the graph. This logic appears in incremental.py lines 1035-1044:

dependent_files: set[str] = set()
for rel_path in changed_files:
    full_path = str(repo_root / rel_path)
    deps = find_dependents(store, full_path)
    for d in deps:
        try:
            dependent_files.add(str(Path(d).relative_to(repo_root)))
        except ValueError:
            pass

Stage 2: SHA-256 Content Verification

For every candidate file (changed files plus dependents), the engine performs a quick hash check to eliminate false positives. As implemented in incremental.py lines 1045-1051, the system reads raw bytes, computes a SHA-256 digest, and compares it against the stored hash in the graph database:

# Quick hash check to skip unchanged files

try:
    raw = abs_path.read_bytes()
    fhash = hashlib.sha256(raw).hexdigest()
    existing_nodes = store.get_nodes_by_file(str(abs_path))
    if existing_nodes and existing_nodes[0].file_hash == fhash:
        continue
except (OSError, PermissionError):
    pass
to_parse.append(rel_path)

When the computed fhash matches existing_nodes[0].file_hash, the file is skipped entirely. Only files with mismatched hashes proceed to to_parse for actual AST analysis.

SHA-256 Implementation Details

Core Hashing Logic in incremental.py

The inline hashing uses Python's standard hashlib module for deterministic, platform-independent checksums. The hexdigest() output provides a 64-character string suitable for SQLite storage and comparison. This approach executes for every candidate file during the incremental_update() workflow.

Reusable Utility in parser.py

The codebase also exposes a dedicated utility function in code_review_graph/parser.py lines 1980-1982 for standalone hash generation:

def file_hash(path: Path) -> str:
    """SHA-256 hash of file contents."""
    return hashlib.sha256(path.read_bytes()).hexdigest()

This function serves other components requiring content-addressed caching or duplicate detection outside the incremental engine.

Database Integration and Storage

The file_hash value persists in the graph store's nodes table, associated with each parsed file node. When store_file_nodes_edges() commits new analysis results during both full builds and incremental updates, it stores the current SHA-256 value alongside AST metadata. The store.get_nodes_by_file() method retrieves these historical hashes for comparison during subsequent runs.

Practical Performance Benefits

This dual-filter architecture delivers significant optimizations:

• VCS noise reduction: File permission changes, timestamp updates, or rename operations that preserve content no longer trigger expensive re-parsing.
• Dependency accuracy: The system correctly handles cases where dependent files remain semantically unchanged despite imports being modified.
• Cryptographic certainty: SHA-256 collisions are computationally infeasible, ensuring the engine never incorrectly skips genuinely modified files.

Summary

• The incremental engine in code_review_graph/incremental.py uses SHA-256 hashing to verify file content changes after initial VCS diffing.
• The hash comparison occurs at lines 1045-1051, using hashlib.sha256(raw).hexdigest() and checking against existing_nodes[0].file_hash.
• A reusable file_hash() utility exists in code_review_graph/parser.py for consistent checksum generation.
• Only files with mismatched SHA-256 hashes enter the to_parse queue, eliminating redundant AST construction.
• The mechanism handles VCS false positives, permission changes, and content-preserving renames without unnecessary processing overhead.

Frequently Asked Questions

Why does the engine use SHA-256 instead of faster hashes like MD5?

SHA-256 provides collision resistance required for build system reliability. While MD5 offers marginally faster computation, the incremental engine prioritizes correctness guarantees over micro-optimizations, ensuring that two different file contents never produce identical hash values that would cause the system to skip necessary re-parsing.

What happens when a file exists in the repository but has no stored hash?

The engine treats missing database entries as changed files requiring parsing. If store.get_nodes_by_file() returns an empty list (existing_nodes is falsy), the hash comparison is bypassed and the file automatically proceeds to the to_parse list for initial AST analysis and hash storage.

Does the SHA-256 check handle binary files or permission errors?

The implementation gracefully handles non-parseable files through exception handling. The try/except block catching OSError and PermissionError ensures that unreadable files (including binaries or those with restricted permissions) do not crash the incremental update, though they typically would not appear in Python source analysis candidates.

How does hashing impact performance on large repositories?

The overhead is negligible compared to AST parsing. Reading file bytes and computing SHA-256 hashes occurs in milliseconds, while parsing Python source into abstract syntax trees requires orders of magnitude more processing. The time saved by skipping unchanged files far outweighs the hash computation cost, particularly in monorepos with thousands of modules.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}