# How to Debug Suboptimal Review Quality or Misplaced Comments in Open Code Review

> Debug suboptimal review quality or misplaced comments in Open Code Review by enabling debug mode. Trace issues from LLM output to final payload to identify generation, sanitization, or positioning problems.

- Repository: [Alibaba/open-code-review](https://github.com/alibaba/open-code-review)
- Tags: how-to-guide
- Published: 2026-08-06

---

**Enable verbose mode with `--debug` or `OCR_DEBUG=1` to inspect the JSON pipeline from LLM output through the positioning module to the final POST payload, tracing whether issues stem from raw generation, metadata sanitization, diff positioning, or routing policy.**

When using **Alibaba Open Code Review (OCR)**, a system that combines deterministic engineering with an LLM-driven agent to generate line-accurate review comments, you may encounter scenarios where comments appear on the wrong lines, get routed to the summary instead of inline, or seem to miss defects entirely. Understanding how to debug suboptimal review quality or misplaced comments requires tracing the data flow through OCR's multi-layered architecture to isolate whether the root cause lies in the LLM output, the diff positioner, or the routing policy.

## Understanding the Review Pipeline Architecture

OCR processes every code review through a deterministic pipeline with seven distinct layers. Each layer transforms the data and passes it to the next, creating specific failure points you can inspect.

- **File/Change Selection** ([`src/main.go`](https://github.com/alibaba/open-code-review/blob/main/src/main.go)): Guarantees every changed file matching a rule is fed to the agent, preventing coverage gaps that cause missing defects.

- **Rule Matching & Bundling** ([`examples/gitlab_ci/post_review.py`](https://github.com/alibaba/open-code-review/blob/main/examples/gitlab_ci/post_review.py)): Applies per-file rules (category, severity) and groups related files into a bundle so the agent receives stable context.

- **Comment Generation (Agent)**: The LLM produces raw comments containing optional fields like `category`, `severity`, `path`, `start_line`, and `end_line`.

- **Positioning Module** ([`internal/tool/comment_positioner.go`](https://github.com/alibaba/open-code-review/blob/main/internal/tool/comment_positioner.go)): Maps LLM output to exact locations in the diff. When mapping fails, comments fall back to summary notes with the `NO_LINE_REASON` tag.

- **Routing / Summary Fallback** (`examples/gitlab_ci/post_review.py::route_comment`): Uses the policy configuration (`route_by_severity`, `severity_rank`, `route_by_category`) to decide whether a comment posts inline or routes to the summary.

- **Folding / Deduplication** (`examples/gerrit_ci/post_review.py::fold_comments`): Collapses duplicate or overlapping comments before posting.

- **Posting** (`examples/*_ci/post_review.py::post`): Sends the final `ReviewInput` (inline plus summary) to the CI platform.

## Common Root Causes of Misplaced or Missing Comments

### Incorrect Metadata from the LLM

Malformed `category` or `severity` strings from the agent can cause routing failures. The **sanitizing step** (`sanitize_metadata`) in [`examples/gitlab_ci/post_review.py`](https://github.com/alibaba/open-code-review/blob/main/examples/gitlab_ci/post_review.py) (lines 24-28) normalizes these values before the routing logic evaluates them. If the LLM omits these fields or uses non-standard values, the comment may bypass inline posting rules and default to the summary.

### Positioning Failures in the Diff Parser

When the positioning module cannot locate the line referenced by the LLM, it tags the comment with `NO_LINE_REASON` and routes it to the summary. This occurs in [`examples/gitlab_ci/post_review.py`](https://github.com/alibaba/open-code-review/blob/main/examples/gitlab_ci/post_review.py) (lines 262-268) when the diff parser cannot reconcile the LLM-provided `start_line`/`end_line` with the actual git diff output. Verify your `git diff` flags and parser version if you see this tag in debug output.

### Aggressive Routing Policy Configuration

The **routing policy** constructed in [`post_review.py`](https://github.com/alibaba/open-code-review/blob/main/post_review.py) (lines 11-16) determines which comments deserve inline placement versus summary routing. If `policy.route_by_severity` is enabled with a permissive `severity_rank`, or if `route_by_category` filters out your target categories, high volumes of comments will appear in the MR summary tab instead of on specific lines.

## Step-by-Step Debug Workflow

Follow this sequence to isolate the failure layer:

1. **Enable Verbose Mode**: Run `OCR_DEBUG=1 ocr review` or add `--debug` to print intermediate JSON from the LLM and the final `ReviewInput` before posting.

2. **Capture Raw LLM Output**: Execute `ocr review --format json > /tmp/review.json` to inspect the agent's raw output. Verify that `path`, `start_line`, `end_line`, `category`, and `severity` fields are present and correctly typed.

3. **Validate Routing Logic**: Examine the policy dictionary in [`examples/gitlab_ci/post_review.py`](https://github.com/alibaba/open-code-review/blob/main/examples/gitlab_ci/post_review.py). Ensure `severity_rank` matches your desired cutoff and that `categories` contains the specific strings the LLM emits.

4. **Check Positioning Results**: Search the JSON output for `NO_LINE_REASON` to identify which diffs failed line mapping. This indicates positioning module issues in [`internal/tool/comment_positioner.go`](https://github.com/alibaba/open-code-review/blob/main/internal/tool/comment_positioner.go).

5. **Test Deduplication in Isolation**: Call `fold_comments(review_input)` directly (from [`examples/gerrit_ci/post_review.py`](https://github.com/alibaba/open-code-review/blob/main/examples/gerrit_ci/post_review.py)) in a Python script to verify duplicate detection works when comments share identical `path` and `start_line` values.

6. **Inspect the Final Payload**: Run the posting script with `--dry-run` to view the exact `ReviewInput` dictionary that will be sent to GitLab or Gerrit without actually submitting it.

## Quick-Start Debug Commands

Generate and inspect review data without posting to your CI platform:

```bash

# Generate raw review JSON without posting

ocr review --format json > /tmp/review.json

# Run GitLab posting script in dry-run mode to see the payload

python3 examples/gitlab_ci/post_review.py --dry-run /tmp/review.json

# Enable full debug output via environment variable

OCR_DEBUG=1 ocr review

```

## Code Examples for Manual Debugging

### Inspecting Raw LLM Output

Use this Python snippet to validate that the LLM produced valid line references and metadata before the routing layer processes them:

```python
import json

data = json.load(open("/tmp/review.json"))
for c in data.get("comments", []):
    print(f"{c.get('path')}:{c.get('start_line')}-{c.get('end_line')} → "
          f"{c.get('severity')} / {c.get('category')}")

```

### Running Routing Decisions Manually

Test how your policy configuration affects specific comments without running the full pipeline:

```python
from examples.gitlab_ci import post_review as pr

policy = {
    "route_by_severity": True,
    "severity_rank": pr.SEVERITY_RANK["low"],
    "route_by_category": False,
    "categories": set(),
}

comment = {
    "category": "style",
    "severity": "low",
    "path": "a.py",
    "start_line": 10
}

# Returns routing decision: inline vs summary

print(pr.route_comment(comment, policy))

```

### Testing Comment Folding

Verify that deduplication logic correctly collapses duplicates when comment IDs differ but content matches:

```python
from examples.gerrit_ci import post_review as gpr

review_input = {
    "comments": {
        "a.py": [
            {"path": "a.py", "start_line": 5, "content": "foo"},
            {"path": "a.py", "start_line": 5, "content": "foo"}
        ]
    }
}

folded = gpr.fold_comments(review_input)
print(folded["comments"])

```

## Summary

- **Enable `OCR_DEBUG=1`** to expose the intermediate JSON pipeline and identify whether comments are malformed at generation or routing time.
- **Inspect `sanitize_metadata`** in [`examples/gitlab_ci/post_review.py`](https://github.com/alibaba/open-code-review/blob/main/examples/gitlab_ci/post_review.py) when comments have malformed `category` or `severity` strings.
- **Search for `NO_LINE_REASON`** in debug output to catch positioning failures from [`internal/tool/comment_positioner.go`](https://github.com/alibaba/open-code-review/blob/main/internal/tool/comment_positioner.go).
- **Adjust the policy** (`route_by_severity`, `severity_rank`) in [`post_review.py`](https://github.com/alibaba/open-code-review/blob/main/post_review.py) to control whether comments appear inline or in the summary.
- **Use `--dry-run`** to preview the final `ReviewInput` payload before it reaches GitLab or Gerrit.

## Frequently Asked Questions

### Why do all my comments appear in the MR summary instead of inline?

Your routing policy is likely configured to send low-severity or specific categories to the summary. Check the `policy` dictionary in [`examples/gitlab_ci/post_review.py`](https://github.com/alibaba/open-code-review/blob/main/examples/gitlab_ci/post_review.py) and verify that `route_by_severity` is not set too aggressively or that `severity_rank` is not too permissive. You may also see this behavior if the positioning module tags comments with `NO_LINE_REASON` due to line mapping failures.

### How can I verify if the LLM is generating incorrect line numbers?

Run `ocr review --format json` and inspect the `start_line` and `end_line` fields in the raw output. If these values do not align with the actual diff, the **positioning module** ([`internal/tool/comment_positioner.go`](https://github.com/alibaba/open-code-review/blob/main/internal/tool/comment_positioner.go)) will fail to map them, triggering a fallback to summary comments. Compare the LLM output against your `git diff` to identify parser version mismatches.

### What causes duplicate comments to appear in the review?

Duplicates occur when the `fold_comments` function in [`examples/gerrit_ci/post_review.py`](https://github.com/alibaba/open-code-review/blob/main/examples/gerrit_ci/post_review.py) cannot identify identical comments as matches. Ensure that comments intended for deduplication share the same `path` and `start_line` fields, as the function uses these keys for set-based deduplication. If comment IDs differ but content matches, the folding logic should still collapse them if the structural keys align.

### How do I debug missing defects that should have been caught?

First, verify **file selection** in [`src/main.go`](https://github.com/alibaba/open-code-review/blob/main/src/main.go) to ensure the changed files matched your rules and were fed to the agent. Then check the raw LLM JSON output for empty `comments` arrays or missing entries. If the LLM generated the comments but they are missing from the final review, inspect the `route_comment` logic to see if they were filtered by category or severity before posting.