# How Cost Guard and Rate Limiting Protect LLM Traffic in MemoryProxy: Full Breakdown

> Discover how MemoryProxy’s rate limiting and cost guard shield LLM traffic, prevent runaway costs, and protect upstream providers from burst traffic.

- Repository: [Tencent Cloud/TencentDB-Agent-Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory)
- Tags: deep-dive
- Published: 2026-08-22

---

**MemoryProxy enforces a two-stage shield — Rate Limiting throttles traffic volume per memory instance while Cost Guard rewrites requests at the routing layer — together preventing runaway costs and protecting upstream LLM providers from burst traffic.**

According to the [TencentCloud/TencentDB-Agent-Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory) repository, MemoryProxy is the request gateway that sits between client LLM calls and the upstream model service. It adds configurable safety and efficiency layers that keep the system stable, protect downstream resources, and give operators fine-grained control over traffic. Two of the most critical layers are the **Cost Guard** and the **per-memory-instance Rate Limiting**, both defined in the [`MemoryProxy/src/types.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryProxy/src/types.ts) source file.

## What Is the Cost Guard in MemoryProxy?

The **Cost Guard** is a routing layer implemented in the MemoryProxy source that can rewrite the request body and/or authentication headers before a request is forwarded to the upstream model. As defined in the [`CostGuardConfig` interface in [`MemoryProxy/src/types.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryProxy/src/types.ts)](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/feat/server_team/MemoryProxy/src/types.ts), this guard gives operators several independent controls:

- **`enabled`** — Toggles the entire cost guard on or off globally.
- **`markerOptIn`** — Requires an explicit `/cost-guard` URL marker before the guard applies, meaning only marked traffic is transformed.
- **`agentProfile`** — Pins a specific agent profile, allowing the guard to auto-detect which policy rules apply per request.
- **`anthropicUpstream`** — Overrides the default Anthropic upstream endpoint, letting operators route traffic to a different provider or a cheaper fallback.

When the cost guard is enabled, every request reaching a primary handler passes through this router first. This ensures that cost-related policies, such as limiting expensive models or applying private API keys, are enforced consistently across all traffic rather than relying on developers to remember them per call.

## How Per-Instance Rate Limiting Protects the Upstream Service

While the cost guard handles *what* the request looks like, **Rate Limiting** in MemoryProxy handles *how much* traffic is allowed. The [`RateLimitConfig` interface in [`MemoryProxy/src/types.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryProxy/src/types.ts)](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/feat/server_team/MemoryProxy/src/types.ts) limits the number of **input tokens per minute (TPM)** and **requests per minute (QPM)** a single memory instance, identified by the combination of **space ID × model**, may consume within a rolling minute window.

### Rate Limiter Configuration Fields

- **`tpm`** — Maximum input tokens per minute per space/model. Set to `0` to disable the token limit entirely.
- **`qpm`** — Maximum requests per minute per space/model. Set to `0` to disable request throttling.

The implementation uses a sliding-window counter stored in Redis, applied before the request is forwarded upstream. This protects the upstream model service from burst traffic and prevents a single memory space from consuming the entire token quota.

## How Mutually Help They Work Together: The Two-Stage Shield

The interaction between Rate Limiting and Cost Guard creates a defense-in-depth strategy:

1. **Incoming request** → The handler extracts the `spaceId` and model name from the request context.
2. **Rate-limit check** — A Redis-backed sliding window counter compares current usage against `RateLimitConfig`. If the request exceeds the configured TPM or QPM, it is rejected early, avoiding unnecessary upstream HTTP calls.
3. **Cost Guard routing** — If enabled, the cost guard router runs next. It may rewrite the request body, inject private authentication headers, or enforce the `/cost-guard` marker opt-in logic.

The result is a **two-stage shield**: rate limiting throttles traffic volume to protect the upstream service from spikes, while the cost guard enforces policy-driven request transformation, giving operators control over which models are used, which credentials are distributed to what traffic is affected.

## Code Example: Enabling Cost Guard and Rate Limiting in Configuration

The following YAML snippet, based on representative configuration from the [`MemoryProxy/config.example.yaml`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryProxy/config.example.yaml) sample, shows how to enable both features:

```yaml
costGuard:
  enabled: true                # Turn the guard on

  markerOptIn: false           # Apply to all traffic (no explicit marker needed)

  agentProfile: "auto"         # Auto-detect profile from request headers

  anthropicUpstream:
    url: "https://anthropic.example.com/v1"

rateLimit:
  tpm: 500_000                 # Max 500k input tokens per minute per space/model

  qpm: 1_000                   # Max 1k requests per minute per space/model

```

### Applying the Logic in Node.js

Implementing the pipeline might look like the following, which mirrors the handler logic allowed in MemoryProxy/src/handler.ts:

```ts
import { RateLimitConfig, CostGuardConfig } from "./types";

// From config: const rateLimit: RateLimitConfig = config.rateLimit;
// From config: const costGuard: CostGuardConfig = config.costGuard;

if (rateLimit.tpm > 0 || rateLimit.qpm > 0) {
  // Rate-limit check (sliding window counter in Redis)
  await redisRateLimiter.check(spaceId, model, rateLimit);
}

// Cost guard processing
if (costGuard.enabled) {
  const adaptedRequest = applyCostGuard(request, costGuard);
  forwardToUpstream(adaptedRequest);
}

```

Within the MemoryProxy codebase, comments near the primary handler in [`MemoryProxy/src/handler.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryProxy/src/handler.ts) reference precisely where these two stages are invoked, so developers can trace the enforcement path through the source.

## Comparing the Two Guard Layers

| **Component** | **Purpose** | **What It Protects** | **Where Defined** |
|---|---|---|---|
| **Rate Limiting** | Token/request throttling per space/model | Upstream service from runaway traffic and cost surprises | `RateLimitConfig` in [`MemoryProxy/src/types.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryProxy/src/types.ts) |
| **Cost Guard** | Request body and auth header rewriting | Cost policy compliance (model selection, credentials) | `CostGuardConfig` in [`MemoryProxy/src/types.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryProxy/src/types.ts) |

## Summary

- **Rate Limiting** enforces per-memory-instance TPM and QPM limits using a Redis-backed sliding window counter, protecting the upstream model from burst traffic.
- **Cost Guard** acts as a request router that can rewrite the body or add authentication headers before upstream forwarding, enabling consistent cost policy enforcement.
- **Configuration flexibility**: operators can independently toggle either feature, require explicit `/cost-guard` opt-in markers, or direct traffic to a custom Anthropic endpoint.
- **Overall** the two stages combine approach a complete traffic management loop: one controls flow, the other controls request content — giving the system both operational stability and financial governance.


## Frequently Asked Questions

### What is the cost guard in MemoryProxy?

The cost guard is a routing layer defined in [`MemoryProxy/src/types.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryProxy/src/types.ts) as `CostGuardConfig`. When enabled, it rewrites the request body or authentication headers before a request is forwarded upstream, allowing operators to enforce model-selection policies, inject private keys, and optionally require an explicit URL opt-in marker.

### How does rate limiting work in the MemoryProxy?

Rate limiting limits both input tokens per minute (TPM) and requests per minute (QPM) per memory instance, where the instance is identified by the combination of `spaceId` and model. The limiter uses a Redis-backed sliding-window counter and rejects traffic that exceeds the configured thresholds, with a value of `0` disabling the limit entirely.

### Can you disable rate limiting but keep the cost guard enabled?

Yes. In `RateLimitConfig`, setting both `tpm` and `qpm` to `0` disables the rate limiter while the independent `CostGuardConfig` ensures the guard continues to rewrite requests. The two features are fully independent and can be toggled separately in the MemoryProxy configuration.

### Where are the cost guard and rate limiting defined in the source code?

Both configurations are defined in [`MemoryProxy/src/types.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryProxy/src/types.ts). The `CostGuardConfig` interface is declared first, containing the `enabled`, `markerOptIn`, `agentProfile`, and `anthropicUpstream` fields, while the `RateLimitConfig` interface follows with the `tpm` and `qpm` limit fields. The enforcement itself happens in [`MemoryProxy/src/handler.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryProxy/src/handler.ts) inside the main request pipeline.