How to Optimize Token Usage with Prompt Compression Budgets in OmniRoute

OmniRoute reduces upstream LLM token costs by enforcing configurable compression budgets that automatically fall back to less aggressive modes when token limits are exceeded.

OmniRoute implements a sophisticated prompt-compression pipeline that helps organizations optimize token usage with prompt compression budgets. The system automatically selects the most efficient compression strategy while respecting hard token limits configured per combo. According to the OmniRoute source code, this budget-aware architecture ensures optimal cost control without requiring manual prompt engineering for every request.

How the Compression Pipeline Works

The compression service lives in open-sse/services/compression and processes every request through a series of budget-aware stages. When a request arrives, the engine first resolves a compression plan via resolveCompressionPlan.ts. This plan evaluates combo-level overrides and checks the budget gate (budgetGate.ts and its helper hardBudget.ts) to determine whether the projected token count exceeds the configured allowance.

If the request would exceed the budget, the pipeline automatically falls back through a progression of less aggressive modes: lite → standard → off. This guarantees that no request exceeds your configured token ceiling while still attempting to provide some level of compression.

Strategy Selection Based on Budget

The strategySelector.ts file contains the logic for choosing between available compression engines—including lite, caveman, RTK, and stacked—based on the request parameters, combo overrides, and current budget state. The selector consults the budget gate before finalizing the engine choice, ensuring the selected strategy fits within the remaining token allowance.

Configuring Compression Budgets

Compression settings are validated by Zod schemas defined in src/shared/validation/compressionConfigSchemas.ts. Per-combo budgets persist in SQLite via src/lib/db/compressionContextBudget.ts (re-exported through src/lib/localDb.ts), allowing fine-grained control over token allocation per use case.

To set a hard budget for a specific combo, send a PUT request to the settings endpoint:

curl -X PUT https://omniroute.example.com/api/settings/compression \
  -H "Authorization: Bearer $OMNIRoute_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "compressionComboId": "combo-42",
        "budget": {
          "maxTokens": 8192,
          "hard": true
        }
      }'

Setting "hard": true invokes the enforcement logic in hardBudget.ts, which prevents the pipeline from exceeding the specified maxTokens and triggers the fallback mechanism if necessary.

Request-Level Compression Control

While budgets constrain the system globally, individual requests can request specific compression modes via the X-OmniRoute-Compression header. The header handling logic in open-sse/utils/compressionHeaderEcho.ts processes these overrides, though the budget gate may still downgrade the mode if the request exceeds the combo's token limit.

Force a specific compression mode for a single request:

curl https://omniroute.example.com/v1/chat/completions \
  -H "Authorization: Bearer $OMNIRoute_API_KEY" \
  -H "X-OmniRoute-Compression: caveman" \
  -d '{
        "model": "gpt-4o",
        "messages": [{"role":"user","content":"<very long prompt>"}]
      }'

For programmatic usage, the Node.js client supports passing a compression preference that respects the budget constraints:

import { Omniglot } from "@omniroute/client";

const client = new Omniglot({ apiKey: process.env.OMNIRoute_API_KEY });

await client.chat.completions.create({
  model: "anthropic/claude-3.5-sonnet",
  messages: [{ role: "user", content: longPrompt }],
  compression: "lite",
});

Monitoring Token Usage and Budget Hits

After execution, open-sse/services/compression/stats.ts records telemetry showing tokens saved, the final compression mode applied, and whether the budget was exceeded. This data feeds into analytics dashboards and can be accessed programmatically.

Inspect the latest compression analytics to verify budget compliance:

import { getLatestCompressionAnalyticsRun } from "@/lib/db/compressionAnalytics";

async function showLastRun() {
  const run = await getLatestCompressionAnalyticsRun();
  console.log(`Combo ${run.comboId} used ${run.tokensBefore} → ${run.tokensAfter} tokens`);
  console.log(`Budget hit: ${run.budgetExceeded}`);
}
showLastRun();

Real-time budget telemetry streams through the WebSocket endpoint defined in src/server/ws/liveServer.ts (lines 363-409), enabling live monitoring of token savings and budget enforcement events.

Summary

• Budget gates (budgetGate.ts and hardBudget.ts) enforce token limits by automatically falling back to less aggressive compression modes when thresholds approach.
• Per-combo budgets stored in src/lib/db/compressionContextBudget.ts provide fine-grained control over token allocation via validated JSON configurations.
• Request headers (X-OmniRoute-Compression) allow client-side hints, but the system may override them to respect hard budget constraints.
• Telemetry collection via stats.ts and live WebSocket feeds enables real-time monitoring of token savings and budget hits.

Frequently Asked Questions

What happens when a request exceeds the compression budget?

When the projected token count exceeds the configured budget, the budget gate (budgetGate.ts) intercepts the request and triggers the fallback logic in hardBudget.ts. The system automatically downgrades the compression mode from lite to standard or ultimately to off, ensuring the request stays within the maxTokens limit while still completing successfully.

Can I set different budget limits for different API endpoints?

Yes. OmniRoute stores budgets per compression combo in src/lib/db/compressionContextBudget.ts. Each combo can have its own token ceiling and hard/soft enforcement settings, allowing you to allocate larger budgets for critical production endpoints while keeping development or testing environments strictly limited.

How do I know which compression mode was actually applied?

The open-sse/utils/compressionHeaderEcho.ts utility echoes the final compression mode back in the response headers. Additionally, the stats.ts service records the final mode used, the token count before and after compression, and whether the budget was hit—accessible via the analytics API or the live WebSocket stream in src/server/ws/liveServer.ts.

What is the difference between hard and soft budget enforcement?

A hard budget ("hard": true) activates the enforcement logic in hardBudget.ts, which prevents any compression strategy that would exceed the token limit and forces a fallback to a cheaper mode or disables compression entirely. A soft budget allows the system to exceed the limit slightly if no viable fallback exists, though OmniRoute primarily recommends hard limits for cost-sensitive production environments.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}