# How the LLM Pool and Compression Optimization Works in Open-Code-Review

> Discover how OpenCodeReview's LLM pool and compression optimization uses goroutine workers and memory compression for efficient asynchronous comment processing and token management.

- Repository: [Alibaba/open-code-review](https://github.com/alibaba/open-code-review)
- Tags: internals
- Published: 2026-08-03

---

**The LLM pool and compression optimization in Open-Code-Review uses a fixed-size goroutine worker pool for asynchronous comment processing and a three-zone memory compression system to keep token usage under model limits.**

Open-Code-Review implements two complementary mechanisms to maintain performance during code reviews: a **Comment Worker Pool** that prevents blocking in the main LLM loop, and **three-zone memory compression** that ensures conversations stay within token constraints. This article examines both systems as implemented in the `alibaba/open-code-review` repository, referencing actual source files from `internal/llmloop/`.

## Comment Worker Pool: Asynchronous Post-Processing

The `CommentWorkerPool` in [`internal/llmloop/pool.go`](https://github.com/alibaba/open-code-review/blob/main/internal/llmloop/pool.go) handles expensive comment-validation work without stalling the main LLM loop.

### Pool Structure and Concurrency Control

The pool uses a **semaphore pattern** to limit concurrent goroutines:

- `semaphore chan struct{}` — controls maximum concurrent workers
- `sync.WaitGroup` — tracks submitted jobs for clean shutdown
- Thread-safe result slice — stores `[]model.LlmComment` outputs

```go
// Create a comment-worker pool (8 workers by default)
pool := llmloop.NewCommentWorkerPool(0)

// Submit a post-processing task for a file "main.go"
pool.SubmitFor("main.go", func() ([]model.LlmComment, error) {
    // … heavy validation logic: line-range tracking, reflection, suggestion checks …
    return []model.LlmComment{comment}, nil
})

// Later, wait for all tasks of that file to finish
pool.AwaitKey("main.go")

```

### Submit Patterns and Result Collection

The pool supports two submission modes:

- **`Submit`** — general task without file association
- **`SubmitFor`** — task tied to a specific file path for keyed result retrieval

Results are collected via:
- `Await()` — blocks until all jobs complete, returns all comments
- `AwaitKey(key string)` — blocks until jobs for a specific file finish

Each job runs in its own goroutine, recovers from panics, and writes results safely to the shared slice.

## Three-Zone Memory Compression: Token Management

The compression system in [`internal/llmloop/compression.go`](https://github.com/alibaba/open-code-review/blob/main/internal/llmloop/compression.go) prevents conversations from exceeding the LLM's `MaxTokens` limit through strategic summarization.

### Zone Architecture

`partitionMessages` splits the conversation into three distinct zones:

| Zone | Content | Compression Behavior |
|------|---------|----------------------|
| **Frozen** | First two messages | Never compressed; preserved as anchor context |
| **Compress** | Older conversation rounds | Summarized by the LLM when thresholds trigger |
| **Active** | Most recent rounds | Kept intact to maintain current context |

### Dual Threshold Triggering

Two configured percentages of `Template.MaxTokens` control compression behavior:

- **Soft limit (60%)** — triggers `triggerAsyncCompression()` for background summarization
- **Warning limit (80%)** — forces `runCompression()` for immediate synchronous compression

### Compression Loop Integration

The `Runner.addNextMessage` function in [`internal/llmloop/loop.go`](https://github.com/alibaba/open-code-review/blob/main/internal/llmloop/loop.go) orchestrates the full flow:

1. Compute `softLimit` and `warnLimit` from `Template.MaxTokens`
2. Apply any pending async compression via `tryApplyPendingCompression`
3. If already over warning threshold, force synchronous `runCompression`
4. Append new assistant/tool messages to the conversation
5. Final token check — run another synchronous compression if still over warning limit
6. If between soft and warning limits, launch async compression for next iteration

```go
// Manual compression trigger (rarely needed — the loop handles this automatically)
runner := llmloop.NewRunner(deps)               // deps contain LLM client, template, etc.
msgs   := []llm.Message{ /* existing conversation */ }
compressed, err := runner.runCompression(context.Background(), msgs, "example.go")
if err != nil {
    fmt.Printf("compression failed: %v\n", err)
} else {
    fmt.Printf("new message count: %d\n", len(compressed))
}

```

### Compression Execution Details

The `runCompression` function:
- Encodes the **compress zone** as XML
- Calls the LLM with the `MemoryCompressionTask` prompt
- Receives a concise summary
- Reassembles messages as: **frozen + summary + active**

If the summary is empty or the LLM call fails, the original messages are preserved unchanged. The `compressionState` struct tracks ongoing async jobs and their results for thread-safe merging.

## How Pool and Compression Work Together

These two systems operate at different layers of the LLM loop:

| System | Location | Purpose |
|--------|----------|---------|
| Comment Worker Pool | Post-generation | Offload CPU-intensive comment validation from the main loop |
| Memory Compression | Pre-generation | Reduce token count before adding new messages |

The pool ensures that comment processing parallelism doesn't bottleneck the per-file loop, while compression guarantees that accumulated conversation history never violates model context limits. Both use goroutine-based concurrency — the pool for bounded parallelism, compression for non-blocking background summarization.

## Summary

- **Comment Worker Pool** ([`pool.go`](https://github.com/alibaba/open-code-review/blob/main/pool.go)): Fixed-size goroutine pool with semaphore-based concurrency control, defaulting to 8 workers; supports keyed submission and result collection via `SubmitFor` and `AwaitKey`
- **Three-Zone Compression** ([`compression.go`](https://github.com/alibaba/open-code-review/blob/main/compression.go)): Splits conversation into frozen, compressible, and active zones; triggers at 60% (async) and 80% (sync) of `MaxTokens`
- **Loop Orchestration** ([`loop.go`](https://github.com/alibaba/open-code-review/blob/main/loop.go)): `addNextMessage` coordinates token checks, pending compression application, and threshold-based trigger decisions
- **Failure Handling**: Empty summaries or failed compression calls fall back to original messages; worker pool jobs recover from panics without crashing the main loop

## Frequently Asked Questions

### What happens if compression fails in Open-Code-Review?

The system preserves the original conversation unchanged. In `runCompression`, if the LLM returns an empty summary or the call errors, the function returns the original `messages` slice without modification. This prevents data loss at the cost of potentially exceeding token limits on the next iteration.

### Why does the worker pool default to 8 goroutines?

The `NewCommentWorkerPool` constructor uses 8 as a sensible default when `workerCount <= 0`, balancing parallelism against resource contention during comment post-processing. This number can be tuned via the constructor parameter for deployments with different CPU or I/O characteristics.

### How does the three-zone compression preserve context?

The **frozen** zone anchors the conversation with the system prompt and initial context, while the **active** zone keeps recent tool results and assistant responses intact. Only middle history is summarized, ensuring the LLM retains both original intent and current execution state.

### Can async and sync compression run simultaneously?

The `compressionState` struct prevents overlap through its `running` flag and `pendingResult` field. When a synchronous compression is forced at the 80% warning threshold, it runs immediately and blocks the loop. The 60% soft trigger only launches when no compression is currently active, with `tryApplyPendingCompression` merging results on subsequent `addNextMessage` calls.