OmniRoute Token Compression: How the Multi-Engine Pipeline Reduces LLM Costs

OmniRoute's token compression feature rewrites prompts through a configurable pipeline of specialized engines (Caveman, RTK, Stacked, etc.) to reduce token consumption by 15–95%, controlled globally or per-request via the x-omniroute-compression header.

The OmniRoute token compression system in the diegosouzapw/OmniRoute repository is a modular, plug-in pipeline that shrinks prompts before they reach language models. By implementing multiple compression engines—from lightweight URL stripping to aggressive summarization—it allows the same logical requests to fit within smaller token windows while consuming fewer provider-quota tokens.

Compression Modes and Configuration Options

OmniRoute offers seven distinct compression modes that trade token savings against content preservation. The mode determines which engine or stack of engines processes the request.

Mode	Description	Typical Token Savings
off	No compression applied; raw prompt forwarded.	0%
lite	Strips obvious image URLs and removes empty blocks.	Low
standard	Uses the Caveman engine for aggressive trimming, deduplication, and summarization.	15–60%
aggressive	Enhanced Caveman execution with stricter truncation.	40–80%
ultra	Maximum aggressive trimming for largest possible reduction.	60–95%
rtk	Retrieval-augmented compression using the RTK engine and stored snippets.	Variable
stacked	Runs a sequenced stack (Caveman → RTK → …) to maximize savings.	Highest

Global activation is controlled via the OMNI_COMPRESSION_WORKERS environment variable and database configuration, while per-request overrides use the x-omniroute-compression header documented in src/app/api/reference/API_REFERENCE.md.

The Compression Pipeline Architecture

The system processes every chat completion request through a structured flow from entry to compressed payload.

Request Entry and Routing

The handler for chat completions in open-sse/handlers/chatCore.ts inspects both the global compression switch and the incoming request headers. If the x-omniroute-compression header is present, it takes precedence over the global setting. The validation layer defined in open-sse/services/compression/validation.ts ensures the requested mode is supported before execution begins.

Plan Resolution and Engine Selection

Once a mode is determined, open-sse/services/compression/resolveCompressionPlan.ts builds a concrete compression plan enriched with the runtime engine catalog from open-sse/services/compression/engineCatalog.ts. This catalog registers all available implementations including Caveman, RTK, and Stacked variants. The open-sse/services/compression/strategySelector.ts module then maps the plan to a concrete engine instance.

Execution and Worker Pool

Heavy CPU work is delegated to a worker pool managed by open-sse/services/compression/compressionWorkerPool.ts to keep the main event loop responsive. The public entry point applyCompressionAsync—exposed at src/app/api/compression/preview/route.ts and consumed internally by src/app/api/internal/codex-responses-ws/compression.ts—hands the request body to the selected engine for processing.

Built-in Compression Engines

Each engine implements a specific strategy for reducing token count while preserving semantic intent.

Caveman Engine (Standard and Aggressive Modes)

The open-sse/services/compression/caveman.ts file implements the primary compression logic used in standard and aggressive modes. It performs aggressive trimming, deduplication of repeated phrases, and summarization of verbose content sections. This engine yields the bulk of token savings for general-purpose prompts.

RTK Engine (Retrieval-Augmented Compression)

For rtk mode, the system leverages open-sse/services/compression/engines/rtk/smartTruncate.ts and related files in the RTK directory. This engine performs smart truncation based on stored knowledge snippets, deduplication against retrieved context, and line-filtering to remove redundant technical details while preserving domain-specific terminology.

Stacked Pipeline Orchestration

When stacked mode is requested, open-sse/services/compression/stackedStepCore.ts coordinates the execution of multiple engines in sequence—typically running Caveman followed by RTK—to achieve maximum token reduction. The output of each stage feeds into the next, compounding savings while allowing each engine to handle the content it optimizes best.

Safety Mechanisms

The pipeline includes protective steps to prevent critical information loss. open-sse/services/compression/riskGate/riskGate.ts aborts compression entirely if risky patterns (such as specific code signatures or legal clauses) are detected. Additionally, open-sse/services/compression/prefixFreeze.ts protects frequently-observed system-prompt prefixes from being trimmed, ensuring behavioral instructions remain intact.

Configuring and Using Token Compression

Developers can interact with the compression system through global settings, HTTP headers, or preview endpoints.

Global Configuration

Enable compression across all requests by setting the environment variable OMNI_COMPRESSION_WORKERS to a positive integer indicating the number of worker threads. Database-level configuration can further tune default modes and engine parameters.

Per-Request Header Overrides

Force a specific compression mode for individual requests using the x-omniroute-compression header:

POST /api/chat/completions HTTP/1.1
Content-Type: application/json
x-omniroute-compression: ultra

{
  "model": "gpt-4o-mini",
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": "Explain quantum computing in detail..." }
  ]
}

From a TypeScript client:

import { fetchChat } from "omniroute-client";

await fetchChat({
  model: "gpt-4o-mini",
  messages,
  headers: { "x-omniroute-compression": "rtk" }
});

Previewing Compression Results

Inspect how a specific prompt will be transformed before sending it to the model by calling the preview endpoint implemented in src/app/api/compression/preview/route.ts:

POST /api/compression/preview HTTP/1.1
Content-Type: application/json
x-omniroute-compression: stacked

{
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": "Explain token compression in detail..." }
  ],
  "max_output_tokens": 1024
}

The endpoint returns the compressed payload and token statistics without forwarding the request to a language model provider.

Analytics and Observability

OmniRoute tracks compression performance through a dedicated analytics subsystem.

Database Persistence

Token counts before and after compression are persisted to the compression_analytics table, created by the migration runner in src/lib/db/migrationRunner.ts. This enables historical analysis of savings per mode and engine.

Analytics Endpoint

Query aggregated statistics via the endpoint served by open-sse/services/compression/stats.ts:

GET /api/analytics/compression HTTP/1.1
Authorization: Bearer <admin-token>

This returns JSON showing total tokens saved, compression ratios, and mode distribution:

{
  "totalTokensBefore": 125000,
  "totalTokensAfter": 72000,
  "tokensSaved": 53000,
  "modeDistribution": { "lite": 12, "standard": 34, "rtk": 7 }
}

Summary

• OmniRoute token compression reduces prompt size through a modular pipeline of specialized engines including Caveman, RTK, and Stacked modes, achieving 15–95% token savings.
• The system is controlled globally via OMNI_COMPRESSION_WORKERS or per-request using the x-omniroute-compression header, with validation handled in open-sse/services/compression/validation.ts.
• Plan resolution occurs in open-sse/services/compression/resolveCompressionPlan.ts, while engine selection is managed by open-sse/services/compression/strategySelector.ts.
• CPU-intensive compression runs in a worker pool (open-sse/services/compression/compressionWorkerPool.ts) to maintain API responsiveness.
• Safety mechanisms including risk gates (open-sse/services/compression/riskGate/riskGate.ts) and prefix freezing (open-sse/services/compression/prefixFreeze.ts) prevent loss of critical context.
• Operators can preview compression via src/app/api/compression/preview/route.ts and monitor savings through the compression_analytics table and /api/analytics/compression endpoint.

Frequently Asked Questions

What compression mode should I use for production?

Start with standard mode (x-omniroute-compression: standard), which uses the Caveman engine in open-sse/services/compression/caveman.ts to achieve 15–60% savings with minimal semantic loss. If you operate with large context windows or repetitive technical documentation, experiment with rtk mode for retrieval-augmented compression, or stacked mode for maximum efficiency when latency is acceptable.

Can I disable compression for specific requests?

Yes. Send the header x-omniroute-compression: off to bypass the pipeline for individual requests, or set the global environment variable to disable workers entirely. The request handler in open-sse/handlers/chatCore.ts checks this header before invoking applyCompressionAsync, allowing fine-grained control over which prompts get processed.

How does OmniRoute ensure important system instructions aren't lost during compression?

The pipeline implements prefix freezing via open-sse/services/compression/prefixFreeze.ts, which protects frequently-observed system-prompt prefixes from trimming. Additionally, the risk gate in open-sse/services/compression/riskGate/riskGate.ts scans for risky patterns before compression begins and aborts the process if sensitive content structures are detected, ensuring critical instructions remain intact.

Where does OmniRoute store compression statistics?

Token counts before and after compression are written to the compression_analytics table, created by src/lib/db/migrationRunner.ts. Administrators can query aggregated metrics through the /api/analytics/compression endpoint implemented in open-sse/services/compression/stats.ts, which returns JSON containing total savings and mode distribution across the fleet.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}