How to Debug Suboptimal Review Quality or Misplaced Comments in Open Code Review

Enable verbose mode with --debug or OCR_DEBUG=1 to inspect the JSON pipeline from LLM output through the positioning module to the final POST payload, tracing whether issues stem from raw generation, metadata sanitization, diff positioning, or routing policy.

When using Alibaba Open Code Review (OCR), a system that combines deterministic engineering with an LLM-driven agent to generate line-accurate review comments, you may encounter scenarios where comments appear on the wrong lines, get routed to the summary instead of inline, or seem to miss defects entirely. Understanding how to debug suboptimal review quality or misplaced comments requires tracing the data flow through OCR's multi-layered architecture to isolate whether the root cause lies in the LLM output, the diff positioner, or the routing policy.

Understanding the Review Pipeline Architecture

OCR processes every code review through a deterministic pipeline with seven distinct layers. Each layer transforms the data and passes it to the next, creating specific failure points you can inspect.

• File/Change Selection (src/main.go): Guarantees every changed file matching a rule is fed to the agent, preventing coverage gaps that cause missing defects.

• Rule Matching & Bundling (examples/gitlab_ci/post_review.py): Applies per-file rules (category, severity) and groups related files into a bundle so the agent receives stable context.

• Comment Generation (Agent): The LLM produces raw comments containing optional fields like category, severity, path, start_line, and end_line.

• Positioning Module (internal/tool/comment_positioner.go): Maps LLM output to exact locations in the diff. When mapping fails, comments fall back to summary notes with the NO_LINE_REASON tag.

• Routing / Summary Fallback (examples/gitlab_ci/post_review.py::route_comment): Uses the policy configuration (route_by_severity, severity_rank, route_by_category) to decide whether a comment posts inline or routes to the summary.

• Folding / Deduplication (examples/gerrit_ci/post_review.py::fold_comments): Collapses duplicate or overlapping comments before posting.

• Posting (examples/*_ci/post_review.py::post): Sends the final ReviewInput (inline plus summary) to the CI platform.

Common Root Causes of Misplaced or Missing Comments

Incorrect Metadata from the LLM

Malformed category or severity strings from the agent can cause routing failures. The sanitizing step (sanitize_metadata) in examples/gitlab_ci/post_review.py (lines 24-28) normalizes these values before the routing logic evaluates them. If the LLM omits these fields or uses non-standard values, the comment may bypass inline posting rules and default to the summary.

Positioning Failures in the Diff Parser

When the positioning module cannot locate the line referenced by the LLM, it tags the comment with NO_LINE_REASON and routes it to the summary. This occurs in examples/gitlab_ci/post_review.py (lines 262-268) when the diff parser cannot reconcile the LLM-provided start_line/end_line with the actual git diff output. Verify your git diff flags and parser version if you see this tag in debug output.

Aggressive Routing Policy Configuration

The routing policy constructed in post_review.py (lines 11-16) determines which comments deserve inline placement versus summary routing. If policy.route_by_severity is enabled with a permissive severity_rank, or if route_by_category filters out your target categories, high volumes of comments will appear in the MR summary tab instead of on specific lines.

Step-by-Step Debug Workflow

Follow this sequence to isolate the failure layer:

1. Enable Verbose Mode: Run OCR_DEBUG=1 ocr review or add --debug to print intermediate JSON from the LLM and the final ReviewInput before posting.

2. Capture Raw LLM Output: Execute ocr review --format json > /tmp/review.json to inspect the agent's raw output. Verify that path, start_line, end_line, category, and severity fields are present and correctly typed.

3. Validate Routing Logic: Examine the policy dictionary in examples/gitlab_ci/post_review.py. Ensure severity_rank matches your desired cutoff and that categories contains the specific strings the LLM emits.

4. Check Positioning Results: Search the JSON output for NO_LINE_REASON to identify which diffs failed line mapping. This indicates positioning module issues in internal/tool/comment_positioner.go.

5. Test Deduplication in Isolation: Call fold_comments(review_input) directly (from examples/gerrit_ci/post_review.py) in a Python script to verify duplicate detection works when comments share identical path and start_line values.

6. Inspect the Final Payload: Run the posting script with --dry-run to view the exact ReviewInput dictionary that will be sent to GitLab or Gerrit without actually submitting it.

Quick-Start Debug Commands

Generate and inspect review data without posting to your CI platform:

# Generate raw review JSON without posting

ocr review --format json > /tmp/review.json

# Run GitLab posting script in dry-run mode to see the payload

python3 examples/gitlab_ci/post_review.py --dry-run /tmp/review.json

# Enable full debug output via environment variable

OCR_DEBUG=1 ocr review

Code Examples for Manual Debugging

Inspecting Raw LLM Output

Use this Python snippet to validate that the LLM produced valid line references and metadata before the routing layer processes them:

import json

data = json.load(open("/tmp/review.json"))
for c in data.get("comments", []):
    print(f"{c.get('path')}:{c.get('start_line')}-{c.get('end_line')} → "
          f"{c.get('severity')} / {c.get('category')}")

Running Routing Decisions Manually

Test how your policy configuration affects specific comments without running the full pipeline:

from examples.gitlab_ci import post_review as pr

policy = {
    "route_by_severity": True,
    "severity_rank": pr.SEVERITY_RANK["low"],
    "route_by_category": False,
    "categories": set(),
}

comment = {
    "category": "style",
    "severity": "low",
    "path": "a.py",
    "start_line": 10
}

# Returns routing decision: inline vs summary

print(pr.route_comment(comment, policy))

Testing Comment Folding

Verify that deduplication logic correctly collapses duplicates when comment IDs differ but content matches:

from examples.gerrit_ci import post_review as gpr

review_input = {
    "comments": {
        "a.py": [
            {"path": "a.py", "start_line": 5, "content": "foo"},
            {"path": "a.py", "start_line": 5, "content": "foo"}
        ]
    }
}

folded = gpr.fold_comments(review_input)
print(folded["comments"])

Summary

• Enable OCR_DEBUG=1 to expose the intermediate JSON pipeline and identify whether comments are malformed at generation or routing time.
• Inspect sanitize_metadata in examples/gitlab_ci/post_review.py when comments have malformed category or severity strings.
• Search for NO_LINE_REASON in debug output to catch positioning failures from internal/tool/comment_positioner.go.
• Adjust the policy (route_by_severity, severity_rank) in post_review.py to control whether comments appear inline or in the summary.
• Use --dry-run to preview the final ReviewInput payload before it reaches GitLab or Gerrit.

Frequently Asked Questions

Why do all my comments appear in the MR summary instead of inline?

Your routing policy is likely configured to send low-severity or specific categories to the summary. Check the policy dictionary in examples/gitlab_ci/post_review.py and verify that route_by_severity is not set too aggressively or that severity_rank is not too permissive. You may also see this behavior if the positioning module tags comments with NO_LINE_REASON due to line mapping failures.

How can I verify if the LLM is generating incorrect line numbers?

Run ocr review --format json and inspect the start_line and end_line fields in the raw output. If these values do not align with the actual diff, the positioning module (internal/tool/comment_positioner.go) will fail to map them, triggering a fallback to summary comments. Compare the LLM output against your git diff to identify parser version mismatches.

What causes duplicate comments to appear in the review?

Duplicates occur when the fold_comments function in examples/gerrit_ci/post_review.py cannot identify identical comments as matches. Ensure that comments intended for deduplication share the same path and start_line fields, as the function uses these keys for set-based deduplication. If comment IDs differ but content matches, the folding logic should still collapse them if the structural keys align.

How do I debug missing defects that should have been caught?

First, verify file selection in src/main.go to ensure the changed files matched your rules and were fed to the agent. Then check the raw LLM JSON output for empty comments arrays or missing entries. If the LLM generated the comments but they are missing from the final review, inspect the route_comment logic to see if they were filtered by category or severity before posting.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}