How Cost Guard and Rate Limiting Protect LLM Traffic in MemoryProxy: Full Breakdown

MemoryProxy enforces a two-stage shield — Rate Limiting throttles traffic volume per memory instance while Cost Guard rewrites requests at the routing layer — together preventing runaway costs and protecting upstream LLM providers from burst traffic.

According to the TencentCloud/TencentDB-Agent-Memory repository, MemoryProxy is the request gateway that sits between client LLM calls and the upstream model service. It adds configurable safety and efficiency layers that keep the system stable, protect downstream resources, and give operators fine-grained control over traffic. Two of the most critical layers are the Cost Guard and the per-memory-instance Rate Limiting, both defined in the MemoryProxy/src/types.ts source file.

What Is the Cost Guard in MemoryProxy?

The Cost Guard is a routing layer implemented in the MemoryProxy source that can rewrite the request body and/or authentication headers before a request is forwarded to the upstream model. As defined in the [CostGuardConfig interface in MemoryProxy/src/types.ts](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/feat/server_team/MemoryProxy/src/types.ts), this guard gives operators several independent controls:

• enabled — Toggles the entire cost guard on or off globally.
• markerOptIn — Requires an explicit /cost-guard URL marker before the guard applies, meaning only marked traffic is transformed.
• agentProfile — Pins a specific agent profile, allowing the guard to auto-detect which policy rules apply per request.
• anthropicUpstream — Overrides the default Anthropic upstream endpoint, letting operators route traffic to a different provider or a cheaper fallback.

When the cost guard is enabled, every request reaching a primary handler passes through this router first. This ensures that cost-related policies, such as limiting expensive models or applying private API keys, are enforced consistently across all traffic rather than relying on developers to remember them per call.

How Per-Instance Rate Limiting Protects the Upstream Service

While the cost guard handles what the request looks like, Rate Limiting in MemoryProxy handles how much traffic is allowed. The [RateLimitConfig interface in MemoryProxy/src/types.ts](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/feat/server_team/MemoryProxy/src/types.ts) limits the number of input tokens per minute (TPM) and requests per minute (QPM) a single memory instance, identified by the combination of space ID × model, may consume within a rolling minute window.

Rate Limiter Configuration Fields

• tpm — Maximum input tokens per minute per space/model. Set to 0 to disable the token limit entirely.
• qpm — Maximum requests per minute per space/model. Set to 0 to disable request throttling.

The implementation uses a sliding-window counter stored in Redis, applied before the request is forwarded upstream. This protects the upstream model service from burst traffic and prevents a single memory space from consuming the entire token quota.

How Mutually Help They Work Together: The Two-Stage Shield

The interaction between Rate Limiting and Cost Guard creates a defense-in-depth strategy:

1. Incoming request → The handler extracts the spaceId and model name from the request context.
2. Rate-limit check — A Redis-backed sliding window counter compares current usage against RateLimitConfig. If the request exceeds the configured TPM or QPM, it is rejected early, avoiding unnecessary upstream HTTP calls.
3. Cost Guard routing — If enabled, the cost guard router runs next. It may rewrite the request body, inject private authentication headers, or enforce the /cost-guard marker opt-in logic.

The result is a two-stage shield: rate limiting throttles traffic volume to protect the upstream service from spikes, while the cost guard enforces policy-driven request transformation, giving operators control over which models are used, which credentials are distributed to what traffic is affected.

Code Example: Enabling Cost Guard and Rate Limiting in Configuration

The following YAML snippet, based on representative configuration from the MemoryProxy/config.example.yaml sample, shows how to enable both features:

costGuard:
  enabled: true                # Turn the guard on

  markerOptIn: false           # Apply to all traffic (no explicit marker needed)

  agentProfile: "auto"         # Auto-detect profile from request headers

  anthropicUpstream:
    url: "https://anthropic.example.com/v1"

rateLimit:
  tpm: 500_000                 # Max 500k input tokens per minute per space/model

  qpm: 1_000                   # Max 1k requests per minute per space/model

Applying the Logic in Node.js

Implementing the pipeline might look like the following, which mirrors the handler logic allowed in MemoryProxy/src/handler.ts:

import { RateLimitConfig, CostGuardConfig } from "./types";

// From config: const rateLimit: RateLimitConfig = config.rateLimit;
// From config: const costGuard: CostGuardConfig = config.costGuard;

if (rateLimit.tpm > 0 || rateLimit.qpm > 0) {
  // Rate-limit check (sliding window counter in Redis)
  await redisRateLimiter.check(spaceId, model, rateLimit);
}

// Cost guard processing
if (costGuard.enabled) {
  const adaptedRequest = applyCostGuard(request, costGuard);
  forwardToUpstream(adaptedRequest);
}

Within the MemoryProxy codebase, comments near the primary handler in MemoryProxy/src/handler.ts reference precisely where these two stages are invoked, so developers can trace the enforcement path through the source.

Comparing the Two Guard Layers

Component	Purpose	What It Protects	Where Defined
Rate Limiting	Token/request throttling per space/model	Upstream service from runaway traffic and cost surprises	RateLimitConfig in MemoryProxy/src/types.ts
Cost Guard	Request body and auth header rewriting	Cost policy compliance (model selection, credentials)	CostGuardConfig in MemoryProxy/src/types.ts

Summary

• Rate Limiting enforces per-memory-instance TPM and QPM limits using a Redis-backed sliding window counter, protecting the upstream model from burst traffic.
• Cost Guard acts as a request router that can rewrite the body or add authentication headers before upstream forwarding, enabling consistent cost policy enforcement.
• Configuration flexibility: operators can independently toggle either feature, require explicit /cost-guard opt-in markers, or direct traffic to a custom Anthropic endpoint.
• Overall the two stages combine approach a complete traffic management loop: one controls flow, the other controls request content — giving the system both operational stability and financial governance.

Frequently Asked Questions

What is the cost guard in MemoryProxy?

The cost guard is a routing layer defined in MemoryProxy/src/types.ts as CostGuardConfig. When enabled, it rewrites the request body or authentication headers before a request is forwarded upstream, allowing operators to enforce model-selection policies, inject private keys, and optionally require an explicit URL opt-in marker.

How does rate limiting work in the MemoryProxy?

Rate limiting limits both input tokens per minute (TPM) and requests per minute (QPM) per memory instance, where the instance is identified by the combination of spaceId and model. The limiter uses a Redis-backed sliding-window counter and rejects traffic that exceeds the configured thresholds, with a value of 0 disabling the limit entirely.

Can you disable rate limiting but keep the cost guard enabled?

Yes. In RateLimitConfig, setting both tpm and qpm to 0 disables the rate limiter while the independent CostGuardConfig ensures the guard continues to rewrite requests. The two features are fully independent and can be toggled separately in the MemoryProxy configuration.

Where are the cost guard and rate limiting defined in the source code?

Both configurations are defined in MemoryProxy/src/types.ts. The CostGuardConfig interface is declared first, containing the enabled, markerOptIn, agentProfile, and anthropicUpstream fields, while the RateLimitConfig interface follows with the tpm and qpm limit fields. The enforcement itself happens in MemoryProxy/src/handler.ts inside the main request pipeline.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}