How the RTK + Caveman Token Compression Pipeline Works in OmniRoute

The RTK + Caveman token compression pipeline is a two-stage system that first applies rule-based semantic condensation via the Caveman engine, then performs token-aware fine-tuning via the RTK engine to reduce prompt sizes by 30-70% while preserving semantic integrity.

The diegosouzapw/OmniRoute repository implements an aggressive request-level compression system designed to minimize upstream LLM costs. This pipeline combines coarse-grained rule matching with specialized content trimming to shrink multi-turn conversations, code diffs, and structured logs before they reach the provider.

Pipeline Orchestration and Strategy Selection

When a request triggers aggressive or ultra compression modes, the strategy selector (open-sse/services/compression/strategySelector.ts) constructs a stacked execution plan. This orchestrator intentionally runs the Caveman engine first, then passes the intermediate result to the RTK engine (open-sse/services/compression/engines/rtk/index.ts).

This ordering maximizes efficiency: Caveman performs high-level semantic condensation, removing structural redundancy, while RTK applies low-level, content-aware optimizations that require the noise already removed.

Stage 1: The Caveman Engine

Located in open-sse/services/compression/caveman.ts, the Caveman engine executes rule-based semantic condensation defined in cavemanRules.ts. It operates through three primary mechanisms:

• Duplicate elimination: Removes repeated system prompts and redundant messages across multi-turn conversations using deduplication heuristics.
• Whitespace normalization: Collapses unnecessary whitespace and formatting characters that consume tokens without adding meaning.
• Hedged preservation: Retains a small "hedge" of original content for potential re-injection, ensuring critical information like tool results or code snippets survive the initial pass.

// Inside open-sse/services/compression/caveman.ts
export function dedupSystemPrompt(messages: Message[]): Message[] {
  const seen = new Set<string>();
  return messages.filter(m => {
    if (m.role === 'system') {
      if (seen.has(m.content)) return false;
      seen.add(m.content);
    }
    return true;
  });
}

Stage 2: The RTK Engine

The RTK (Rule-based Token-Killer) engine, implemented in open-sse/services/compression/engines/rtk/index.ts, receives the Caveman-processed content and applies fine-grained, token-specific optimizations through several specialized sub-modules.

Line Filtering and Cleaning

The lineFilter.ts module eliminates blank lines, comment-only lines, and trivial markup that passed through the initial stage. This targets vertical whitespace and non-semantic formatting common in logs and diffs.

Smart Truncation for Large Blocks

smartTruncate.ts detects the longest meaningful prefix of large content blocks—such as extensive logs or Git diffs—and discards the tail while preserving syntactically valid fragments. It uses the tokenizer to ensure truncation occurs at token boundaries rather than character boundaries.

// Inside open-sse/services/compression/engines/rtk/smartTruncate.ts
export function smartTruncate(text: string, maxTokens: number): string {
  const tokens = tokenizer.encode(text);
  if (tokens.length <= maxTokens) return text;
  // keep the first N tokens that preserve a syntactically valid block
  const prefix = tokens.slice(0, maxTokens);
  return tokenizer.decode(prefix);
}

Code-Specific Optimization

codeStriper.ts removes language-specific comment syntaxes and redundant boilerplate code. It identifies common patterns like Python docstrings or JavaScript inline comments and strips them without affecting executable logic.

Cross-Block Deduplication

The deduplicator.ts module performs a second-pass deduplication across the entire output, catching repeated lines that may have originated from different sources but converged after the Caveman stage.

Specialized Renderers

The renderers/*.ts collection converts complex structured data into concise textual representations. For example, terraformPlan.ts extracts only resource changes (add, modify, delete), while structuredTable.ts preserves header rows with a limited sample of data rows. These renderers transform verbose machine outputs into token-efficient summaries.

Result Memoization and Metrics

After RTK processing completes, resultMemo.ts caches the compressed payload. Subsequent calls within the same session reuse this cached result, eliminating redundant CPU cycles and ensuring consistent token counts across repeated requests.

The stats.ts module tracks compression efficacy, logging original token counts, compressed counts, and percentage savings. Unit tests such as rtk-smart-truncate.test.ts and caveman-preservation.test.ts verify that realistic workloads—including tool-generated code diffs and long chat histories—consistently achieve 30-70% token reduction.

Implementation Example

To invoke the stacked pipeline in your application:

import { resolveCompressionPlan } from '@/open-sse/services/compression/resolveCompressionPlan';
import { CompressionMode } from '@/open-sse/services/compression/types';

const plan = await resolveCompressionPlan({
  mode: CompressionMode.Aggressive,   // triggers Caveman → RTK stacking
  requestBody: incomingPrompt,
});

const compressed = await plan.run();   // runs Caveman then RTK

Summary

• The RTK + Caveman token compression pipeline uses a two-stage architecture: Caveman for coarse semantic condensation, RTK for fine-grained token optimization.
• Caveman (caveman.ts) removes duplicate system prompts, collapses whitespace, and applies preservation heuristics via rules defined in cavemanRules.ts.
• RTK (engines/rtk/index.ts) performs line filtering, smart truncation, code stripping, and specialized rendering through modular sub-components.
• Result memoization (resultMemo.ts) caches outputs to improve performance, while stats (stats.ts) tracks 30-70% token savings.
• The pipeline is triggered via CompressionMode.Aggressive or CompressionMode.Ultra through the strategy selector (strategySelector.ts).

Frequently Asked Questions

What is the difference between the Caveman and RTK engines in the compression pipeline?

Caveman performs high-level, rule-based semantic condensation such as deduplicating system prompts and normalizing whitespace, while RTK executes fine-grained, content-aware optimizations like token-level truncation and code-comment stripping. The pipeline intentionally runs Caveman first to remove structural noise before RTK applies its computationally heavier heuristics.

How much token reduction can I expect when using the RTK + Caveman pipeline?

According to the test suites rtk-smart-truncate.test.ts and caveman-preservation.test.ts, the pipeline consistently achieves 30-70% token reduction for realistic workloads including multi-turn chat histories, tool-generated code diffs, and verbose structured logs.

Does the compression pipeline preserve code syntax and semantic meaning?

Yes. The Caveman engine uses preservation heuristics to retain critical content like code snippets and tool results, while the RTK engine's smartTruncate.ts ensures truncation occurs at syntactically valid boundaries. The codeStriper.ts module specifically targets comment syntax rather than executable code, maintaining semantic integrity.

How does OmniRoute optimize performance when compressing multiple requests?

The resultMemo.ts module implements result memoization, storing the compressed output after the first RTK pass. Subsequent requests within the same session retrieve the cached payload instead of reprocessing the original content, significantly reducing CPU overhead and ensuring consistent token counts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}