How the LLM Pool and Compression Optimization Works in Open-Code-Review

The LLM pool and compression optimization in Open-Code-Review uses a fixed-size goroutine worker pool for asynchronous comment processing and a three-zone memory compression system to keep token usage under model limits.

Open-Code-Review implements two complementary mechanisms to maintain performance during code reviews: a Comment Worker Pool that prevents blocking in the main LLM loop, and three-zone memory compression that ensures conversations stay within token constraints. This article examines both systems as implemented in the alibaba/open-code-review repository, referencing actual source files from internal/llmloop/.

Comment Worker Pool: Asynchronous Post-Processing

The CommentWorkerPool in internal/llmloop/pool.go handles expensive comment-validation work without stalling the main LLM loop.

Pool Structure and Concurrency Control

The pool uses a semaphore pattern to limit concurrent goroutines:

• semaphore chan struct{} — controls maximum concurrent workers
• sync.WaitGroup — tracks submitted jobs for clean shutdown
• Thread-safe result slice — stores []model.LlmComment outputs

// Create a comment-worker pool (8 workers by default)
pool := llmloop.NewCommentWorkerPool(0)

// Submit a post-processing task for a file "main.go"
pool.SubmitFor("main.go", func() ([]model.LlmComment, error) {
    // … heavy validation logic: line-range tracking, reflection, suggestion checks …
    return []model.LlmComment{comment}, nil
})

// Later, wait for all tasks of that file to finish
pool.AwaitKey("main.go")

Submit Patterns and Result Collection

The pool supports two submission modes:

• Submit — general task without file association
• SubmitFor — task tied to a specific file path for keyed result retrieval

Results are collected via:

• Await() — blocks until all jobs complete, returns all comments
• AwaitKey(key string) — blocks until jobs for a specific file finish

Each job runs in its own goroutine, recovers from panics, and writes results safely to the shared slice.

Three-Zone Memory Compression: Token Management

The compression system in internal/llmloop/compression.go prevents conversations from exceeding the LLM's MaxTokens limit through strategic summarization.

Zone Architecture

partitionMessages splits the conversation into three distinct zones:

Zone	Content	Compression Behavior
Frozen	First two messages	Never compressed; preserved as anchor context
Compress	Older conversation rounds	Summarized by the LLM when thresholds trigger
Active	Most recent rounds	Kept intact to maintain current context

Dual Threshold Triggering

Two configured percentages of Template.MaxTokens control compression behavior:

• Soft limit (60%) — triggers triggerAsyncCompression() for background summarization
• Warning limit (80%) — forces runCompression() for immediate synchronous compression

Compression Loop Integration

The Runner.addNextMessage function in internal/llmloop/loop.go orchestrates the full flow:

1. Compute softLimit and warnLimit from Template.MaxTokens
2. Apply any pending async compression via tryApplyPendingCompression
3. If already over warning threshold, force synchronous runCompression
4. Append new assistant/tool messages to the conversation
5. Final token check — run another synchronous compression if still over warning limit
6. If between soft and warning limits, launch async compression for next iteration

// Manual compression trigger (rarely needed — the loop handles this automatically)
runner := llmloop.NewRunner(deps)               // deps contain LLM client, template, etc.
msgs   := []llm.Message{ /* existing conversation */ }
compressed, err := runner.runCompression(context.Background(), msgs, "example.go")
if err != nil {
    fmt.Printf("compression failed: %v\n", err)
} else {
    fmt.Printf("new message count: %d\n", len(compressed))
}

Compression Execution Details

The runCompression function:

• Encodes the compress zone as XML
• Calls the LLM with the MemoryCompressionTask prompt
• Receives a concise summary
• Reassembles messages as: frozen + summary + active

If the summary is empty or the LLM call fails, the original messages are preserved unchanged. The compressionState struct tracks ongoing async jobs and their results for thread-safe merging.

How Pool and Compression Work Together

These two systems operate at different layers of the LLM loop:

System	Location	Purpose
Comment Worker Pool	Post-generation	Offload CPU-intensive comment validation from the main loop
Memory Compression	Pre-generation	Reduce token count before adding new messages

The pool ensures that comment processing parallelism doesn't bottleneck the per-file loop, while compression guarantees that accumulated conversation history never violates model context limits. Both use goroutine-based concurrency — the pool for bounded parallelism, compression for non-blocking background summarization.

Summary

• Comment Worker Pool (pool.go): Fixed-size goroutine pool with semaphore-based concurrency control, defaulting to 8 workers; supports keyed submission and result collection via SubmitFor and AwaitKey
• Three-Zone Compression (compression.go): Splits conversation into frozen, compressible, and active zones; triggers at 60% (async) and 80% (sync) of MaxTokens
• Loop Orchestration (loop.go): addNextMessage coordinates token checks, pending compression application, and threshold-based trigger decisions
• Failure Handling: Empty summaries or failed compression calls fall back to original messages; worker pool jobs recover from panics without crashing the main loop

Frequently Asked Questions

What happens if compression fails in Open-Code-Review?

The system preserves the original conversation unchanged. In runCompression, if the LLM returns an empty summary or the call errors, the function returns the original messages slice without modification. This prevents data loss at the cost of potentially exceeding token limits on the next iteration.

Why does the worker pool default to 8 goroutines?

The NewCommentWorkerPool constructor uses 8 as a sensible default when workerCount <= 0, balancing parallelism against resource contention during comment post-processing. This number can be tuned via the constructor parameter for deployments with different CPU or I/O characteristics.

How does the three-zone compression preserve context?

The frozen zone anchors the conversation with the system prompt and initial context, while the active zone keeps recent tool results and assistant responses intact. Only middle history is summarized, ensuring the LLM retains both original intent and current execution state.

Can async and sync compression run simultaneously?

The compressionState struct prevents overlap through its running flag and pendingResult field. When a synchronous compression is forced at the 80% warning threshold, it runs immediately and blocks the loop. The 60% soft trigger only launches when no compression is currently active, with tryApplyPendingCompression merging results on subsequent addNextMessage calls.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}