How the RTK + Caveman Compression Pipeline Achieves Token Savings in OmniRoute

The RTK + Caveman compression pipeline achieves token savings by stacking a coarse-grained semantic condenser (Caveman) with a fine-grained token-aware trimmer (RTK), yielding 30–70% reductions through deduplication, smart truncation, and specialized rendering of structured data.

The OmniRoute repository implements a sophisticated request-level prompt compression system designed to minimize token consumption when sending requests to upstream LLM providers. The RTK + Caveman compression pipeline combines two distinct engines—Caveman for coarse rule-based condensation and RTK (Rule-based Token-Killer) for precision trimming—to preserve semantic meaning while dramatically reducing payload size.

Pipeline Architecture: Stacking Caveman and RTK

Strategy Selection and Orchestration

When aggressive or "ultra" compression modes are activated, the strategy selector located in open-sse/services/compression/strategySelector.ts constructs a stacked execution plan. This plan intentionally runs the Caveman engine first, followed by the RTK engine in open-sse/services/compression/engines/rtk/index.ts.

Execution Order and Rationale

The ordering matters: Caveman performs high-level semantic condensation first, removing duplicate system prompts and collapsing whitespace. The partially compressed output then flows into RTK, which applies fine-grained, content-aware trimming such as line filtering and smart truncation. This sequential approach ensures that coarse redundancies are eliminated before token-level optimization begins.

Stage 1: The Caveman Engine

Rule-Based Semantic Condensation

The Caveman engine (open-sse/services/compression/caveman.ts) applies handcrafted rules defined in open-sse/services/compression/cavemanRules.ts. These rules remove duplicate system prompts, strip irrelevant sections, and collapse whitespace while applying preservation heuristics to retain critical information like code snippets and tool results.

Deduplication and Hedging

Caveman deduplicates repeated messages across multi-turn conversations using a Set-based tracking mechanism. The engine also maintains a small "hedge" of original content for potential re-injection, ensuring that aggressive compression does not permanently lose context that might be needed later.

// Inside the Caveman engine – deduplicating system prompts
export function dedupSystemPrompt(messages: Message[]): Message[] {
  const seen = new Set<string>();
  return messages.filter(m => {
    if (m.role === 'system') {
      if (seen.has(m.content)) return false;
      seen.add(m.content);
    }
    return true;
  });
}

Stage 2: The RTK Engine

Line Filtering and Smart Truncation

The RTK engine processes Caveman's output through several specialized modules. The line filter (open-sse/services/compression/engines/rtk/lineFilter.ts) eliminates blank lines, comment lines, and trivial markup. The smart truncation module (open-sse/services/compression/engines/rtk/smartTruncate.ts) detects the longest meaningful prefix of large blocks—such as lengthy logs or diffs—and discards the tail while preserving syntactically valid fragments.

Code Stripping and Deduplication

The code stripper (open-sse/services/compression/engines/rtk/codeStriper.ts) removes language-specific comment syntaxes and redundant boilerplate. The deduplicator (open-sse/services/compression/engines/rtk/deduplicator.ts) further collapses repeated lines across the Caveman stage output, ensuring no redundancy persists into the final payload.

Specialized Renderers for Structured Data

RTK includes domain-specific renderers in open-sse/services/compression/engines/rtk/renderers/ that convert complex structures into token-efficient text. For example, terraformPlan.ts extracts only resource changes, while structuredTable.ts preserves header rows with minimal sample data, drastically reducing token count for structured outputs.

// Inside the RTK engine – smart truncation of long log blocks
export function smartTruncate(text: string, maxTokens: number): string {
  const tokens = tokenizer.encode(text);
  if (tokens.length <= maxTokens) return text;
  // Keep the first N tokens that preserve a syntactically valid block
  const prefix = tokens.slice(0, maxTokens);
  return tokenizer.decode(prefix);
}

Performance Optimization and Caching

Result Memoization

After RTK completes processing, the result memo module (open-sse/services/compression/resultMemo.ts) caches the compressed result. Subsequent calls within the same session reuse this payload without re-processing, reducing CPU overhead and maintaining consistent token counts.

Token Savings Metrics

The stats module (open-sse/services/compression/stats.ts) records original versus compressed token counts and calculates percentage savings. Unit tests in rtk-smart-truncate.test.ts and caveman-preservation.test.ts verify that the pipeline consistently achieves 30–70% token reductions for realistic workloads including tool-generated diffs, long logs, and multi-turn chat histories.

// Example: Using the stacked "aggressive" mode
import { resolveCompressionPlan } from '@/open-sse/services/compression/resolveCompressionPlan';
import { CompressionMode } from '@/open-sse/services/compression/types';

const plan = await resolveCompressionPlan({
  mode: CompressionMode.Aggressive,   // Triggers Caveman → RTK stacking
  requestBody: incomingPrompt,
});

const compressed = await plan.run();   // Runs Caveman then RTK

Summary

• Two-stage architecture: Caveman handles coarse semantic condensation while RTK performs fine-grained token optimization.
• Strategic ordering: Running Caveman before RTK eliminates high-level redundancy before line-level trimming occurs.
• Comprehensive deduplication: Both engines remove duplicates at different granularities—Caveman at the message level and RTK at the line level.
• Smart truncation: RTK preserves syntactically valid prefixes when truncating large content blocks rather than cutting arbitrarily.
• Specialized renderers: Domain-specific parsers in open-sse/services/compression/engines/rtk/renderers/ convert structured data (Terraform plans, tables, diffs) into minimal token representations.
• Proven savings: The pipeline consistently achieves 30–70% token reduction across production workloads, verified by the stats module and comprehensive unit tests.

Frequently Asked Questions

How does the ordering of Caveman and RTK affect compression quality?

Caveman runs before RTK to ensure that high-level semantic redundancies—such as duplicate system prompts and repeated conversation turns—are eliminated first. This prevents RTK from wasting token budget on trimming content that should have been removed entirely, allowing RTK to focus its fine-grained heuristics on preserving the semantic density of the remaining content.

What types of content benefit most from the RTK + Caveman pipeline?

Structured data and repetitive conversations see the highest savings. Terraform plans processed by terraformPlan.ts, long log files handled by smartTruncate.ts, and multi-turn chat histories with repeated system prompts benefit most—often hitting the upper range of 70% token reduction according to the stats module in open-sse/services/compression/stats.ts.

How does result memoization improve performance beyond token savings?

The result memo module (open-sse/services/compression/resultMemo.ts) caches the final compressed output after RTK processing. This eliminates redundant CPU cycles for identical subsequent requests in the same session, ensuring that compression overhead remains minimal while keeping token counts consistent across repeated calls.

What token reduction rates can be expected in production workloads?

According to unit tests in caveman-preservation.test.ts and rtk-smart-truncate.test.ts, the RTK + Caveman compression pipeline consistently delivers 30–70% token savings. The actual percentage depends on content redundancy—highly repetitive tool outputs (like build logs or infrastructure diffs) typically achieve higher savings than dense, unique prose.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}