How RTK and Caveman Compression Reduce Token Usage in OmniRoute

RTK and Caveman compression in OmniRoute work as a stacked pipeline—RTK applies command-aware filtering first (≈80% token reduction), then Caveman applies prose-aware compression (additional ≈46% reduction)—achieving 78%–95% overall token savings before text reaches the LLM.

OmniRoute's dual-engine compression system targets the single largest cost driver in LLM-powered workflows: token volume. By chaining two specialized engines—RTK for machine-generated output and Caveman for narrative condensation—the system multiplicatively shrinks payloads while preserving critical semantic information. This article explains the architecture, configuration, and measurable impact of how RTK and Caveman compression reduce token usage in production deployments.

RTK Compression: Command-Aware First Pass

The RTK (Run-Time-Kit) engine handles the initial heavy lifting on noisy, structured output from developer tools.

Architecture and Pipeline

Located at open-sse/services/compression/engines/rtk/index.ts, the RTK engine implements a seven-stage pipeline:

1. Command detection — commandDetector.ts classifies incoming content by tool type (git, npm, Docker, test runners, etc.)
2. Filter loading — Priority-ordered filter resolution: project-level (.rtk/filters.*), then global (DATA_DIR/rtk/filters.*), then built-in defaults in open-sse/services/compression/engines/rtk/filters/
3. DSL rule application — Filters execute ANSI stripping, pattern replacement, match-output extraction, include/drop logic, and truncation per filterSchema.ts
4. Per-filter deduplication — rules.deduplicate collapses repeated lines within individual filter scopes
5. Engine-wide deduplication — Final pass with deduplicateThreshold (default 3 identical consecutive lines)
6. Optional grouping — groupSimilarLines collapses near-identical adjacent lines when rtkConfig.enableGrouping is enabled
7. Intensity-based truncation — Head/tail limits vary by intensity level

Intensity Configuration

Level	Head/Tail Lines	Use Case
minimal	24 lines	Maximum fidelity, audit scenarios
standard (default)	20 lines	Balanced compression
aggressive	16 lines	Cost-critical, high-volume pipelines

RTK never transmits raw output to downstream LLMs unless rawOutputRetention is explicitly configured for debugging.

Measured Savings

Per the RTK benchmark documentation in docs/compression/RTK_COMPRESSION.md, the engine achieves 60%–90% token reduction depending on input verbosity. OmniRoute adopts ≈80% as the operational average.

Caveman Compression: Prose-Aware Second Pass

The Caveman engine receives RTK-compressed output and applies heuristic-based narrative compression.

Core Implementation

The engine entry point at open-sse/services/compression/caveman.ts orchestrates rules defined in open-sse/services/compression/cavemanRules.ts:

• Line deduplication — Tuned threshold for prose rather than structured logs
• Hedging removal — Strips filler phrases ("as far as I can tell", "it seems that")
• Structural pruning — Collapses repetitive stack traces, long log sequences
• Error preservation — Guarantees survival of stack traces, assertion failures, and critical error metadata

Engine-Wide Final Pass

After per-rule application, Caveman runs identical deduplication to RTK: collapsing runs of ≥3 identical consecutive lines. This ensures consistency across the stacked pipeline.

Incremental Savings

Caveman contributes ≈46% additional reduction on the already-RTK-compressed payload. The multiplicative effect is what drives total savings toward the upper bound.

Stacked Compression: Calculating Total Token Reduction

When engines operate in sequence (rtk → caveman), savings compound rather than add:

Stacked saving = 1 − (1 − RTK_save) × (1 − Caveman_save)
               = 1 − (1 − 0.80) × (1 − 0.46)
               ≈ 89% overall token reduction

Accounting for RTK's reported variability (60%–90%), the full system achieves 78%–95% total token reduction before LLM ingestion.

Practical Configuration Examples

Preview Stacked Compression Pipeline

POST /api/compression/preview
{
  "mode": "stacked",
  "messages": [
    {
      "role": "tool",
      "content": "FAIL tests/example.test.ts\nAssertionError: expected true\nTest Files 1 failed (1 total)\nTests 1 failed (2 total)\nDuration 1.23s"
    }
  ],
  "config": {
    "rtkConfig": { "intensity": "standard" },
    "cavemanConfig": { "intensity": "full" }
  }
}

The response includes CompressionStats with rtkSavedPct and cavemanSavedPct fields, plus the compressed payload typically measuring ~10% of original token count.

Single-Engine RTK Endpoint

POST /api/context/rtk/test
{
  "command": "npm test",
  "text": "FAIL tests/example.test.ts\nAssertionError: expected true\n    at Object.<anonymous> (/project/tests/example.test.ts:15:15)",
  "config": { "intensity": "aggressive" }
}

Returns compressed text (~20% of original) and per-engine statistics.

Enable Raw Output Retention

PUT /api/context/rtk/config
{
  "rawOutputRetention": "always",
  "maxCharsPerResult": 12000
}

Raw originals are stored to DATA_DIR/rtk/raw-output/ and retrievable via:

GET /api/context/rtk/raw-output/<request_id>

Programmatic Intensity Adjustment

import { updateEngineConfig } from "@omniroute/open-sse/services/compression/engines/registry";

// Maximum compression for cost optimization
updateEngineConfig("rtk", { intensity: "aggressive" });
updateEngineConfig("caveman", { intensity: "full" });

This pushes total reduction toward the ~95% upper bound by truncating more aggressively (16 lines/section) and applying full Caveman heuristics.

Key Source Files and Components

Component	Path	Purpose
RTK documentation	docs/compression/RTK_COMPRESSION.md	Savings methodology, intensity definitions
RTK engine	open-sse/services/compression/engines/rtk/index.ts	Core pipeline implementation
Command detector	open-sse/services/compression/engines/rtk/commandDetector.ts	Tool output classification
Filter schema	open-sse/services/compression/engines/rtk/filterSchema.ts	Zod validation for filter DSL
Filters directory	open-sse/services/compression/engines/rtk/filters/	Built-in filter definitions
Caveman engine	open-sse/services/compression/caveman.ts	Second-stage compression orchestration
Caveman rules	open-sse/services/compression/cavemanRules.ts	Heuristic rule definitions
Compression API	src/app/api/v1/compression/preview/route.ts	HTTP endpoint for preview/stacked modes
Configuration storage	src/lib/db/compression.ts	SQLite key_value persistence for settings
Regression tests	tests/golden-set/compression-caveman-v2.test.ts	Validation of advertised savings percentages

Summary

• RTK compression achieves ~80% token reduction through command detection, filter DSL rules, deduplication, and intensity-controlled truncation
• Caveman compression adds ~46% incremental reduction via prose heuristics, hedging removal, and error-preserving structural pruning
• Stacked operation produces 78%–95% total savings through multiplicative (not additive) composition
• Configuration granularity spans per-request, per-engine, and global persistence layers
• Observability includes raw output retention, preview endpoints, and statistical reporting for cost attribution

Frequently Asked Questions

How does OmniRoute ensure critical error information survives compression?

Both engines implement error-preservation heuristics. RTK's filters in commandDetector.ts recognize test failure patterns and apply less aggressive truncation. Caveman's rules in cavemanRules.ts explicitly whitelist stack traces, assertion messages, and error metadata during structural pruning. The tests/golden-set/compression-caveman-v2.test.ts suite validates that essential diagnostic content persists through the full pipeline.

Can I disable Caveman and use only RTK compression?

Yes. The /api/compression/preview endpoint accepts "mode": "rtk" to execute single-engine compression. However, this sacrifices the additional ~46% savings Caveman provides. For maximum cost efficiency, stacked mode is recommended unless profiling reveals Caveman overhead exceeds its benefit for your specific workload patterns.

What is the latency cost of stacked compression?

The RTK engine operates in sub-millisecond time for typical CLI output (<10KB). Caveman adds comparable overhead for prose processing. Combined, the pipeline typically adds <5ms per request—negligible compared to LLM API round-trip times (100ms–2000ms). The token savings translate directly to reduced LLM latency and cost, yielding net performance improvement.

How does rawOutputRetention impact storage costs?

When enabled, raw outputs are written to DATA_DIR/rtk/raw-output/ with automatic rotation based on maxCharsPerResult (default 12,000 characters). For high-throughput systems, configure retention to "errors" to store only failed compressions, or implement external cleanup policies on the raw-output directory. Storage growth is predictable: uncompressed retention consumes ~5× the compressed payload size.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}