How Stored Responses and Session Compaction Work in the Grok Gateway

The Grok gateway maintains an in-memory LRU cache of response payloads per user session and executes compaction when detecting a compaction_trigger request type, merging cached entries to eliminate duplicates and refresh TTLs.

The chenyme/grok2api project implements a high-performance gateway that buffers AI responses for reuse within active sessions. Understanding how stored responses and session compaction operate is essential for optimizing memory usage and ensuring low-latency retrieval. The system combines a time-bound result cache with an explicit compaction protocol triggered by specific client requests.

Detecting a Compaction Request

The gateway identifies a compaction request by inspecting the JSON payload for a specific trigger value. In backend/internal/application/gateway/responses_compaction.go, the isResponsesCompactionRequest function unmarshals the request body and scans the input array for an entry where the type field equals "compaction_trigger".

// backend/internal/application/gateway/responses_compaction.go
func isResponsesCompactionRequest(body []byte) bool {
    var payload struct {
        Input []struct {
            Type string `json:"type"`
        } `json:"input"`
    }
    if json.Unmarshal(body, &payload) != nil {
        return false
    }
    for _, item := range payload.Input {
        if strings.EqualFold(strings.TrimSpace(item.Type), "compaction_trigger") {
            return true
        }
    }
    return false
}

When this function returns true, the gateway routes the request to the compaction handler instead of processing it as a standard inference call.

The Result Cache Architecture

Every response passing through the gateway is written to a result cache implemented in backend/internal/pkg/resultcache/cache.go. The cache stores three critical pieces of metadata alongside the response bytes:

• storedAt: A timestamp recording when the entry was created
• expiresAt: The absolute time when the entry becomes stale
• TTL: The duration supplied by the caller during the Set operation

The cache employs an LRU-style eviction policy: when the cache reaches its configured capacity, it removes the oldest entries based on the storedAt timestamp.

// Storing a response with a 5-minute TTL
cache.Set(ctx, "grok", "sess123", [][]byte{responseBytes}, time.Now().Add(5*time.Minute))

Session-specific routing logic in backend/internal/application/gateway/selector_session.go generates the unique session key (e.g., "sess123") used to isolate cached data between different users.

The Session Compaction Flow

When a compaction request is detected, the gateway executes a multi-step compaction routine:

1. Retrieval: The handler fetches all cached responses associated with the session key using cache.Get.
2. Merging: The system consolidates the payloads, removes duplicate or obsolete items, and produces a single compacted byte stream.
3. Replacement: The compacted payload is written back to the cache with a refreshed TTL, replacing the original fragmented entries.

// Retrieving and compacting a session’s responses
stored, ok, err := cache.Get(ctx, "grok", "sess123", time.Now(), time.Hour)
if ok && err == nil {
    compacted := compactResponses(stored) // merges & dedupes
    cache.Set(ctx, "grok", "sess123", [][]byte{compacted}, time.Now().Add(5*time.Minute))
}

Following compaction, the gateway invokes responseMediaAudit from backend/internal/application/gateway/response_media_audit.go to record telemetry. This audit captures the number of items compacted, the total byte size before and after compaction, and any media assets that must be retained downstream.

TTL Management and Sliding Windows

The gateway implements a sliding-window TTL policy to keep active sessions alive while cleaning up idle ones. As implemented in backend/internal/infra/runtime/memory/reasoning_replay.go, the system evaluates the storedAt timestamp whenever accessing a cached entry. For active sessions, the gateway extends the expiresAt time, effectively refreshing the TTL. Idle sessions that exceed their TTL without access are subject to LRU eviction during the next cache write operation.

Summary

• Compaction Detection: The isResponsesCompactionRequest function in responses_compaction.go scans for "compaction_trigger" in the request payload to initiate the process.
• Storage Mechanism: The resultcache/cache.go implementation provides an in-memory LRU cache keyed by session identifiers, tracking storedAt and expiresAt for each entry.
• Data Optimization: The compaction flow merges cached responses, eliminates duplicates, and replaces the original set with a compacted payload featuring a refreshed TTL.
• Audit Trail: responseMediaAudit emits metadata about compaction results, including size deltas and retained media assets, via HTTP response headers.
• Lifecycle Management: A sliding-window TTL policy in reasoning_replay.go extends expiration times for active sessions while allowing LRU eviction for stale data.

Frequently Asked Questions

What triggers a session compaction in the Grok gateway?

A client request containing a JSON payload with {"input":[{"type":"compaction_trigger"}]} triggers the compaction flow. The gateway detects this via isResponsesCompactionRequest and treats the request as a command to consolidate the session's cached responses rather than a standard inference query.

How does the gateway decide which cached responses to evict?

The result cache implements an LRU (Least Recently Used) eviction policy based on the storedAt timestamp. When the cache reaches capacity, it removes the oldest entries first. Additionally, entries whose expiresAt time has passed are considered stale and eligible for removal during cleanup cycles.

What happens to the TTL during the compaction process?

During compaction, the merged payload is written back to the cache with a new TTL calculated from the current time. This effectively resets the sliding window for session expiration, ensuring that active sessions retain their data while preventing indefinite growth of the cache for abandoned sessions.

Where is the compaction audit data stored?

The responseMediaAudit function in backend/internal/application/gateway/response_media_audit.go generates audit metadata that is attached to the HTTP response. This includes metrics such as the number of items compacted, byte size reductions, and references to media assets that downstream services must preserve.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}