How Mako Manages Context and Token Budgets for LLM Interactions: A Deep Dive into the Runtime Budget System

Mako applies a functional context-budget policy to its immutable Runtime Event Log through three stages: token estimation, stale-tool-result pruning, and history compaction, producing a projection that preserves semantics while keeping requests within provider limits.

The Apache Mako runtime (from the apache/maka repository) treats context and token budget management as a pure-functional transformation over an immutable event history. Rather than mutating the persisted ledger, the runtime creates temporary views of the interaction history that fit within configured limits. This design ensures replay correctness while preventing LLM requests from exceeding provider-imposed token ceilings.

The Three-Stage Budget Pipeline

Mako's budget system operates in distinct phases, each implemented in specific source files with clear responsibilities.

Stage 1: Token Estimation

Before any pruning occurs, Mako estimates token consumption using a configurable character-to-token ratio. The default assumes 4 characters per token, matching common LLM tokenization behavior.

The estimation logic lives in packages/runtime/src/context-budget-helpers.ts:

// estimateTokens @ L35-L38
export function estimateTokens(text: string, charsPerToken: number): number {
  return Math.ceil(utf8ByteLength(text) / charsPerToken);
}

This lightweight estimation enables the runtime to make budget decisions without invoking expensive tokenizers for every projection.

Stage 2: Stale Tool Result Pruning

Tool results from previous turns often consume substantial tokens without contributing to the current conversation semantics. Mako's staleToolResultPrune policy removes these oversized artifacts.

The policy configuration appears in packages/runtime/src/context-budget.ts at lines 74-76:

interface ContextBudgetPolicy {
  staleToolResultPrune?: {
    enabled: boolean;
    maxChars: number;  // tool results exceeding this are pruned
  };
}

Critical architectural note: This pruning is replay-only—it never mutates the persisted Runtime Event Log. The original events remain intact for recovery and audit purposes.

Stage 3: History Compaction

When simple pruning insufficient, Mako applies history compaction via applyRuntimeEventHistoryCompactNarrow. This configurable policy replaces old events with semantically equivalent placeholders, bounding log size while preserving turn continuity.

The Core Budget Function: applyRuntimeEventContextBudget

The orchestrating function applyRuntimeEventContextBudget (lines 13-18 in context-budget.ts) implements the complete pipeline:

export function applyRuntimeEventContextBudget(
  events: readonly RuntimeEvent[],
  policy: ContextBudgetPolicy | undefined,
): BudgetedRuntimeContext | undefined {
  // ①  Determine which optimizations are active
  const pruneEnabled = policy?.staleToolResultPrune?.enabled === true;
  const historyCompactEnabled = policy?.historyCompact?.enabled === true;
  if (!pruneEnabled && !historyCompactEnabled) return undefined;

  // ②  Pre-compute token estimate for diagnostics
  const charsPerToken = policy?.charsPerToken ?? 4;
  const estimatedTokensBefore = estimateRuntimeEventsTokens(events, charsPerToken);

  // ③  Apply history compaction (stale pruning handled upstream)
  const compacted = applyRuntimeEventHistoryCompactNarrow(
    events,
    policy?.historyCompact,
    charsPerToken,
  );

  // ④  Build comprehensive diagnostic record
  const diagnostic: ContextBudgetDiagnostic = {
    enabled: true,
    policyName: policy?.name,
    estimatedTokensBefore,
    estimatedTokensAfter: estimateRuntimeEventsTokens(compacted.events, charsPerToken),
    keptTurns: new Set(compacted.events.map(runtimeEventTurnKey)).size,
    droppedTurns: /* calculated from difference */,
    keptEvents: compacted.events.length,
    droppedEvents: events.length - compacted.events.length,
    ...compacted.diagnosticPatch,
  };

  return { events: compacted.events, diagnostic, /* ... */ };
}

The function returns budgeted events (the pruned, compacted projection) plus diagnostics for observability.

From Budgeted Events to Provider Messages

The events array from applyRuntimeEventContextBudget feeds directly into provider-specific message construction. In packages/runtime/src/ai-sdk-backend.ts, the backend:

1. Receives the budgeted RuntimeEvent array
2. Projects events into provider-native message formats
3. Submits the resulting request to the LLM provider

This separation of concerns—budget logic in context-budget.ts, provider adaptation in ai-sdk-backend.ts—enables Mako to support multiple LLM backends with consistent budget semantics.

Configuring Context Budget Policies

Clients supply ContextBudgetPolicy objects when initializing turns. A typical configuration balancing quality and cost:

import { ContextBudgetPolicy } from '@maka/runtime';

const budgetPolicy: ContextBudgetPolicy = {
  name: "production-budget",
  charsPerToken: 4,
  staleToolResultPrune: { 
    enabled: true, 
    maxChars: 20000      // Remove tool results >20KB characters
  },
  activeToolResultPrune: { 
    enabled: true, 
    maxTokens: 2048      // Cap active results at ~2K tokens
  },
  historyCompact: { 
    enabled: true, 
    maxEvents: 500        // Compact when history exceeds 500 events
  },
};

The runtime kernel—entry point at runtime-kernel.ts—passes this policy to applyRuntimeEventContextBudget at turn start.

Why Mako's Budget Is Advisory, Not Absolute

As noted in comments at lines 68-71 of ContextBudgetPolicy, the provider remains the ultimate authority on token limits. Mako's estimates approximate actual tokenization; providers may reject requests that appear within budget. When this occurs, the runtime can apply additional trimming or fallback to smaller context windows.

This design acknowledges a fundamental reality: different providers tokenize differently, and runtime-side budgets must balance proactive optimization against defensive flexibility.

Observability Through Diagnostics

Every budget application produces a ContextBudgetDiagnostic (see minimalContextBudgetDiagnostic in context-budget.ts). These records enable operators to tune policies based on empirical data:

// Example diagnostic output
{
  enabled: true,
  policyName: "production-budget",
  estimatedTokensBefore: 15420,
  estimatedTokensAfter: 8934,
  keptTurns: 12,
  droppedTurns: 8,
  keptEvents: 127,
  droppedEvents: 73
}

Monitoring these diagnostics reveals whether budgets are too aggressive (quality loss) or too conservative (unnecessary cost).

Practical Implementation Example

Complete workflow integrating budget application with provider communication:

import { 
  applyRuntimeEventContextBudget, 
  ContextBudgetPolicy 
} from '@maka/runtime';

// 1. Define budget constraints
const budgetPolicy: ContextBudgetPolicy = {
  name: "tight-budget",
  charsPerToken: 4,
  activeToolResultPrune: { enabled: true, maxTokens: 2048 },
  historyCompact: { enabled: true, maxEvents: 300 },
};

// 2. Apply to current event stream (typically inside backend adapter)
const budgeted = applyRuntimeEventContextBudget(currentEvents, budgetPolicy);

if (budgeted) {
  // 3. Project events to provider-specific messages
  const providerMessages = buildProviderMessages(budgeted.events);
  
  // 4. Log diagnostics for monitoring
  console.info("Context budget applied:", budgeted.diagnostic);
  
  // 5. Execute LLM call with constrained payload
  const response = await llmClient.complete(providerMessages);
}

Key Source Files and Responsibilities

File	Purpose	Critical Functions/Types
packages/runtime/src/context-budget.ts	Core budgeting logic	applyRuntimeEventContextBudget, applyRuntimeEventHistoryCompactNarrow, ContextBudgetPolicy, ContextBudgetDiagnostic
packages/runtime/src/context-budget-helpers.ts	Estimation utilities	estimateTokens, utf8ByteLength, estimateRuntimeEventsTokens
packages/runtime/src/ai-sdk-backend.ts	Provider message construction	Converts budgeted RuntimeEvent[] to provider-native format
packages/runtime/src/runtime-kernel.ts	Turn orchestration	Entry point invoking budget application
packages/runtime/src/__tests__/context-budget.test.ts	Behavioral verification	Unit tests for compaction, pruning, estimation accuracy

Summary

• Mako manages context and token budgets through pure-functional projection of the immutable Runtime Event Log, never mutating persisted state.

• Three pipeline stages—token estimation, stale tool result pruning, and history compaction—progressively reduce payload size while preserving semantic correctness.

• applyRuntimeEventContextBudget in packages/runtime/src/context-budget.ts orchestrates the complete workflow, returning both constrained events and diagnostic metadata.

• Budget policies are client-configurable via ContextBudgetPolicy, specifying character-per-token ratios, pruning thresholds, and compaction limits.

• Provider authority remains ultimate; Mako's estimates optimize proactively but cannot guarantee acceptance, requiring runtime fallback strategies.

• Comprehensive diagnostics enable data-driven policy tuning through ContextBudgetDiagnostic records logged on every request.

Frequently Asked Questions

How does Mako estimate tokens without calling an LLM tokenizer?

Mako uses a configurable character-to-token ratio (default 4 characters/token) implemented in estimateTokens within context-budget-helpers.ts. This approximation avoids tokenizer latency while providing sufficient accuracy for budget decisions. The charsPerToken policy field allows adjustment for different model families with varying tokenization densities.

Can stale tool result pruning cause information loss during replay?

No. Stale tool result pruning is a pure projection applied only to the temporary request view. The original RuntimeEvent containing the full tool result remains in the persisted log. According to the architecture documentation, this guarantees that replay and recovery operations see complete, unmodified history regardless of budget constraints applied to active requests.

What happens when a provider rejects a request that Mako believed was within budget?

The runtime implements defensive fallback strategies. Since provider tokenization may differ from Mako's estimates (noted in ContextBudgetPolicy comments L68-71), the system can apply additional compaction, reduce maxEvents thresholds, or switch to smaller context windows. These adjustments occur at the AiSdkBackend layer before retrying the request.

How can operators monitor whether their budget policies are effective?

Every budget application produces a ContextBudgetDiagnostic object containing beforE/after token estimates, kept/dropped turn counts, and event retention metrics. Operators should aggregate these diagnostics—particularly estimatedTokensBefore, estimatedTokensAfter, and droppedTurns—to identify policies that are too aggressive (excessive turn loss) or too conservative (insufficient savings).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}