How Grok2API Handles Session Sticky with a Custom TTL: Implementation Guide

Grok2API implements session sticky by binding each request to a specific credential ID stored with a configurable time-to-live (TTL), ensuring consistent routing across the infrastructure until the entry expires or the credential is revoked.

Grok2API, the open-source API gateway for Grok AI services, uses a sophisticated session sticky mechanism to maintain request affinity with specific credentials. This feature routes user requests consistently to the same account for a configurable duration, preventing unnecessary credential switching and optimizing cache utilization. The implementation supports both in-memory and Redis backends, with TTL enforcement handled at the repository level according to the chenyme/grok2api source code.

Sticky Session Architecture Overview

The sticky session system centers on a repository pattern that abstracts storage operations behind a clean interface, allowing operators to choose between ephemeral memory storage or persistent Redis clusters.

The StickySessionRepository Interface

The core contract is defined in backend/internal/repository/runtime.go. The StickySessionRepository interface specifies three essential operations: Set to store a binding with expiration, Get to retrieve active bindings, and DeleteByAccount to invalidate all entries for a specific account.

Two concrete implementations provide this functionality:

• memory.NewStickyStore – In-memory map with manual expiry tracking
• redis.NewStickyStore – Redis-backed storage utilizing native TTL capabilities

TTL Configuration in RoutingConfig

The sticky duration is configurable via the YAML configuration. In backend/internal/infra/config/config.go, the RoutingConfig struct exposes a StickyTTL field parsed as a Duration type. The selector accesses this value through cfg.Routing.StickyTTL.Value(), passing it during initialization to control how long credentials remain bound to specific request signatures.

How the Selector Implements Sticky Logic

The credential selection logic in backend/internal/application/gateway/selector.go orchestrates the sticky session flow during the request lifecycle.

Sticky Key Generation and Lookup

When processing a request, the selector generates a sticky key using the promptCacheStickyKey function, which derives a unique identifier from the prompt-cache identity. The selector then queries the sticky store:

// Inside selector.go selection flow (lines 135-273)
stickyKey := promptCacheStickyKey(promptCacheKey)
storedID, ok := sticky.Get(ctx, stickyKey, now)

If ok returns true and the stored credential ID matches the candidate, the selector immediately uses that credential without recomputing rankings, ensuring session continuity.

TTL Enforcement and Updates

Upon successful credential selection, the selector updates the sticky entry with a fresh expiration timestamp:

err := sticky.Set(ctx, stickyKey, candidate.Credential.ID, currentTime.Add(stickyTTL))

Both storage backends validate expiration during the Get operation. The selector passes the current time (now) to the store; if the entry has expired, the store returns ok == false, triggering a fresh selection cycle.

Cleanup on Credential Revocation

When credentials are removed or revoked, the selector cleans up orphaned sticky bindings:

sticky.DeleteByAccount(ctx, revokedAccountID)

This ensures that revoked accounts immediately stop receiving traffic, regardless of remaining TTL.

Storage Backends: Memory vs Redis

Grok2API provides two production-ready implementations of the sticky session store, each handling TTL enforcement differently.

In-Memory Store with Expiry Tracking

The memory implementation in backend/internal/infra/runtime/memory/store.go uses a stickyBinding struct containing an expiresAt field. During Get and Set operations, the store filters expired entries by comparing expiresAt against the provided now timestamp, ensuring stale bindings never influence routing decisions.

Redis Implementation with Lua Scripts

The Redis backend in backend/internal/infra/runtime/redis/store.go leverages Redis native TTL for automatic expiration. For account deletion operations, it uses a Lua script (deleteStickyByAccountScript) that iterates through bindings and verifies expiration timestamps before removal, maintaining consistency across distributed instances.

Practical Implementation Examples

Below are concrete implementations demonstrating how to configure and interact with the sticky session system.

Initialize a selector with a 5-minute sticky TTL:

selector := gateway.NewSelector(
    accountRepo,
    concurrencyLimiter,
    stickyStore,               // memory.NewStickyStore() or redis.NewStickyStore()
    providerRegistry,
    5*time.Minute,             // stickyTTL
    1*time.Second,             // cooldownBase
    10*time.Second,            // cooldownMax
    0,                         // optional capacity wait
)

The selector automatically applies sticky logic during request processing:

candidate, err := selector.Select(ctx, request)
// If a sticky entry exists for the request's prompt-cache key, the same
// credential ID is returned until the TTL expires.

Manually set a sticky session for advanced use cases:

stickyKey := promptCacheStickyKey(promptCacheKey) // builds "sticky:<hash>"
err := stickyStore.Set(ctx, stickyKey, credentialID, time.Now().Add(5*time.Minute))
if err != nil {
    // handle error
}

Clear sticky bindings when revoking credentials:

_ = stickyStore.DeleteByAccount(ctx, revokedAccountID)

Summary

• Grok2API implements session sticky through the StickySessionRepository interface defined in backend/internal/repository/runtime.go, with concrete implementations in memory and Redis.
• The TTL is configurable via RoutingConfig.StickyTTL in backend/internal/infra/config/config.go and enforced during the selection process.
• The selector in backend/internal/application/gateway/selector.go generates sticky keys from prompt-cache identities, retrieves existing bindings, and updates expiration timestamps after successful requests.
• Both storage backends validate expiration during Get operations: the memory store checks expiresAt fields, while Redis relies on native key TTL with Lua script verification for account deletion.
• Sticky bindings are automatically cleared when credentials are revoked via DeleteByAccount.

Frequently Asked Questions

What is the default TTL for sticky sessions in Grok2API?

Grok2API does not hardcode a default TTL; the value is entirely configuration-driven through the routing.stickyTTL YAML setting. The application parses this as a Duration struct and injects it into the selector at startup, requiring operators to explicitly define the sticky window based on their traffic patterns.

How does Grok2API handle expired sticky sessions?

When the selector calls sticky.Get(ctx, stickyKey, now), the underlying store checks the expiration timestamp against the provided now parameter. If the entry has expired, the store returns ok == false, causing the selector to fall back to normal credential ranking and potentially select a new account for subsequent requests.

Can I disable session sticky in Grok2API?

While there is no explicit "disable" flag in the interface, you can effectively disable sticky sessions by setting the stickyTTL configuration to zero or a negative duration. This causes all sticky entries to expire immediately upon creation, forcing the selector to re-evaluate credentials for every request.

What happens to sticky sessions when a credential is revoked?

The selector invokes sticky.DeleteByAccount(ctx, accountID) whenever a credential is removed or revoked. This operation, implemented in both memory and Redis stores, immediately purges all sticky bindings associated with that account ID, ensuring no further requests route to the invalid credential regardless of remaining TTL.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}