Risk Scoring System for PR Reviews: How Levels Are Determined in code-review-graph

The code-review-graph repository assigns every changed symbol a risk score between 0.0 and 1.0 based on seven weighted factors, then maps that score to one of four levels (critical, high, medium, low) using fixed thresholds defined in the PR comment renderer.

The risk scoring system for PR reviews evaluates code changes by analyzing symbol-level impact within a codebase graph. According to the tirth8205/code-review-graph source code, the system combines structural graph metrics, test coverage data, and semantic keyword matching to produce a normalized risk value. This score drives automated gating decisions and reviewer prioritization in CI/CD workflows.

How Risk Scores Are Calculated

The core calculation happens in code_review_graph/changes.py inside the compute_risk_score function (lines 311-327). This function aggregates seven distinct risk factors, each weighted and capped to prevent any single metric from dominating the final result.

Flow Participation Metrics

The algorithm considers how integrated a symbol is within execution flows:

• Flow participation: Adds 0.05 for each flow the node belongs to, capped at 0.25
• Criticality-weighted flow: Uses the explicit criticality value of flows instead of raw counts when available, still capped at 0.25

These metrics ensure that widely-connected functions or those sitting on critical paths receive appropriately higher risk weighting.

Cross-Boundary and Caller Impact

Two factors measure API exposure and architectural coupling:

• Community crossing: Adds 0.05 for each caller located in a different community (module/cluster), capped at 0.15. This identifies changes that affect multiple organizational domains.
• Caller count: Calculates caller_count / 20, capped at 0.10. Highly-utilized utilities receive modest risk elevation due to their broad blast radius.

Test Coverage Modifier

Test density reduces perceived risk through a baseline-subtraction model:

• Baseline starts at 0.30
• Subtracts 0.05 for every 5 reachable tests covering the symbol
• Hard floor at 0.05 ensures some risk remains even for well-tested code

Security Sensitivity Boost

The system applies a +0.20 flat boost when a symbol's name or qualified name matches any keyword in SECURITY_KEYWORDS. These keywords—defined in code_review_graph/constants.py (lines 33-38)—include terms like auth, login, token, password, and encrypt. This ensures credential-handling code receives elevated scrutiny regardless of graph topology.

Optional Change Frequency (Churn)

When historical commit data is supplied via the churn_counts parameter, the system adds:

commit_count / 10 × 0.15 (capped at 0.15)

This identifies frequently-modified "hot" files that historically correlate with defects.

Final Normalization

After summing contributions, the function clamps the total to the 0.0–1.0 interval and rounds to four decimal places:

def compute_risk_score(store, node, churn_counts=None) -> float:
    """Compute a risk score (0.0 - 1.0) for a single node."""
    # ... factor calculations ...

    return round(min(max(score, 0.0), 1.0), 4)

Translating Scores to Risk Levels

Once individual symbols receive numeric scores, the PR comment renderer in scripts/render_pr_comment.py converts these into human-readable levels using the risk_level() function (lines 38-55). The thresholds are hardcoded in the RISK_THRESHOLDS dictionary:

Level	Minimum Score
critical	0.85
high	0.70
medium	0.40
low	< 0.40

The renderer calculates an overall PR risk score by averaging the top-ranked changed symbols, then displays both the numeric value and its corresponding level:

**Overall risk: 0.72 (HIGH)**

When using the --fail-on-risk CLI flag, the script exits with code 3 if the overall level meets or exceeds the specified threshold (e.g., high or critical), enabling CI pipelines to block merges automatically.

Practical Usage Examples

Calculate a single node's risk and map it to a level:

from code_review_graph.changes import compute_risk_score
from scripts.render_pr_comment import risk_level
from code_review_graph.graph import GraphStore, GraphNode

# Assuming `store` is a populated GraphStore and `node` is a GraphNode

score = compute_risk_score(store, node)          # Returns 0.0-1.0 float

level = risk_level(score)                        # Returns "low", "medium", "high", or "critical"

print(f"{node.qualified_name} → score={score:.2f}, level={level}")

Generate a PR comment with risk-based gating:

# Detect changes against base commit

code-review-graph detect-changes --base HEAD~1 > changes.json

# Render comment and fail CI if risk is high or critical

python -m scripts.render_pr_comment \
  --input changes.json \
  --fail-on-risk high

Summary

• The risk scoring system for PR reviews in code-review-graph evaluates every changed symbol using seven weighted factors including flow participation, community crossing, test coverage, and security keyword matching.
• Scores range from 0.0 to 1.0 and are computed in code_review_graph/changes.py by the compute_risk_score function, which clamps and rounds the final result to four decimal places.
• Security-sensitive symbols receive a mandatory +0.20 boost when their names match entries in SECURITY_KEYWORDS (defined in code_review_graph/constants.py).
• The risk_level() function in scripts/render_pr_comment.py maps scores to four discrete levels: critical (≥0.85), high (≥0.70), medium (≥0.40), and low (<0.40).
• CI integration supports --fail-on-risk flags to automatically block merges when overall PR risk exceeds configurable thresholds.

Frequently Asked Questions

What is the maximum possible risk score for a single changed symbol?

The maximum theoretical risk score is 1.0, achieved by clamping in compute_risk_score. In practice, reaching exactly 1.0 requires nearly maximum contributions from flow participation (0.25), community crossing (0.15), security keywords (0.20), caller count (0.10), churn (0.15), and minimal test coverage reduction (0.05 floor), totaling approximately 0.90 before security boosts push it toward the cap.

How does the system handle symbols with zero test coverage?

Symbols with no reachable tests receive the full 0.30 test coverage baseline with no reduction applied. This represents the highest risk contribution from the coverage factor. As test coverage increases, this component decreases by 0.05 per 5 tests until hitting the minimum floor of 0.05.

Can I customize the risk thresholds for my organization?

Currently, the thresholds (0.85, 0.70, 0.40) are hardcoded as constants in scripts/render_pr_comment.py at lines 38-42. To customize levels, you must modify the RISK_THRESHOLDS dictionary in the source code or fork the repository. There is no runtime configuration file for threshold adjustment in the current implementation.

Why are security keywords checked by name rather than by data flow analysis?

The system applies the +0.20 security boost through simple substring matching against SECURITY_KEYWORDS because it provides deterministic, fast classification without requiring expensive static analysis or taint tracking. While data flow analysis might offer precision, the name-based heuristic captures the vast majority of credential-handling functions (authentication, token generation, encryption utilities) with minimal computational overhead during graph traversal.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}