# How to Configure Rate Limits for MemoryProxy Services: Global Defaults and Per-Model Overrides

> Learn how to configure rate limits for MemoryProxy services using global defaults and per-model overrides. Discover Redis-backed TPM and QPM caps for enhanced control.

- Repository: [Tencent Cloud/TencentDB-Agent-Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory)
- Tags: how-to-guide
- Published: 2026-09-02

---

**MemoryProxy enforces token-per-minute (TPM) and query-per-minute (QPM) caps through a Redis-backed limiter that supports both global YAML configuration and dynamic per-instance, per-model overrides via the administrator API.**

The TencentDB-Agent-Memory repository provides MemoryProxy, a high-performance AI model proxy that protects upstream providers and ensures fair resource allocation through configurable rate limiting. Understanding how to configure rate limits for MemoryProxy services allows operators to set baseline usage policies while retaining granular control over specific workloads.

## Global Rate Limit Configuration via YAML

### Setting Default TPM and QPM Values

MemoryProxy establishes baseline limits through the `rateLimit` section in [`config.example.yaml`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/config.example.yaml) (typically copied to [`config.yaml`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/config.yaml)). These values apply to every request unless explicitly overridden via the admin API.

```yaml
rateLimit:
  tpm: 1000000   # input tokens allowed in the last 60s

  qpm: 100       # requests allowed in the last 60s

```

The [`src/config.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/src/config.ts) module parses these values at startup, and the limiter instantiation occurs in [`src/rate-limit/guard.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/src/rate-limit/guard.ts). The `tpm` field controls the rolling window for input tokens, while `qpm` restricts the total number of API calls per minute.

### Redis Storage Configuration

The limiter requires Redis for atomic counter operations. Configure the connection parameters in the same YAML file under the `redis` key. The [`src/rate-limit/redis-store.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/src/rate-limit/redis-store.ts) implementation handles all counter persistence and expiration logic, ensuring consistent rate tracking even when multiple MemoryProxy instances share the same Redis cluster.

## Dynamic Overrides via Admin API

While global settings provide a safety net, production deployments often require different limits for specific customers or model tiers. The admin API defined in [`src/routes/rate-limits.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/src/routes/rate-limits.ts) enables runtime adjustments.

### Querying Effective Limits

To inspect the current limits for a specific combination of instance and model, issue a GET request to `/v3/admin/rate-limits`. The handler in [`src/routes/rate-limits.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/src/routes/rate-limits.ts) returns whether the effective limits derive from global defaults or an explicit override.

```bash
curl -G http://localhost:8096/v3/admin/rate-limits \
  -H "Authorization: Bearer <admin-api-key>" \
  --data-urlencode "instance_id=inst-abc123" \
  --data-urlencode "model_id=gpt-4o-mini"

```

### Creating Per-Instance, Per-Model Overrides

Administrators can establish custom caps using the PUT method. This creates a persistent override in Redis that takes precedence over the global configuration for the specified `instance_id` and `model_id` pair.

```bash
curl -X PUT http://localhost:8096/v3/admin/rate-limits \
  -H "Authorization: Bearer <admin-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
        "instance_id": "inst-abc123",
        "model_id": "gpt-4o-mini",
        "input_tpm": 250000,
        "qpm": 50
      }'

```

### Removing Custom Limits

To revert an instance-model pair to the global defaults, send a DELETE request. The route handler removes the override record from the underlying store.

```bash
curl -X DELETE http://localhost:8096/v3/admin/rate-limits \
  -H "Authorization: Bearer <admin-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
        "instance_id": "inst-abc123",
        "model_id": "gpt-4o-mini"
      }'

```

## Rate Limit Enforcement and Error Handling

### The enforceRateLimit Implementation

Every incoming request passes through the `enforceRateLimit` function in [`src/rate-limit/guard.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/src/rate-limit/guard.ts). This function queries the current counters from [`src/rate-limit/redis-store.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/src/rate-limit/redis-store.ts) and compares them against the effective limits. If the request would exceed either the TPM or QPM threshold, the function throws a `RateLimitExceededError`.

### Response Headers and Status Codes

When limits are exceeded, MemoryProxy returns HTTP 429 (Too Many Requests) with informative headers. As implemented in [`src/rate-limit/guard.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/src/rate-limit/guard.ts) (lines 66-70), the response includes:

- **`X-Ratelimit-Limit-Input-Tokens`** – The configured TPM cap
- **`X-Ratelimit-Remaining-Input-Tokens`** – Tokens remaining in the current window
- **`X-Ratelimit-Limit-Requests`** – The configured QPM cap
- **`X-Ratelimit-Remaining-Requests`** – Requests remaining in the current window
- **`Retry-After`** – Seconds until the window resets

The JSON error body follows OpenAI-compatible formatting:

```http
HTTP/1.1 429 Too Many Requests
Content-Type: application/json; charset=utf-8
Retry-After: 30
X-Ratelimit-Limit-Input-Tokens: 250000
X-Ratelimit-Remaining-Input-Tokens: 0

{
  "error": {
    "message": "该模型输入 Token 用量已达上限，请稍后重试",
    "type": "rate_limit_error",
    "code": "input_tpm_exceeded"
  }
}

```

## Fail-Open Behavior for High Availability

If Redis becomes unreachable, the limiter enters a degraded mode. According to the error handling logic in [`src/rate-limit/guard.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/src/rate-limit/guard.ts) (lines 36-50), the proxy logs a warning but allows the request to proceed. This fail-open design prevents Redis outages from causing total service disruption, though rate limits will not be enforced during the connectivity gap.

## Summary

- **Configure global defaults** in [`config.example.yaml`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/config.example.yaml) under the `rateLimit` key to establish baseline TPM and QPM values for all traffic.
- **Manage overrides dynamically** using the `/v3/admin/rate-limits` endpoints implemented in [`src/routes/rate-limits.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/src/routes/rate-limits.ts) to customize limits per `instance_id` and `model_id`.
- **Enforcement occurs** in [`src/rate-limit/guard.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/src/rate-limit/guard.ts) via the `enforceRateLimit` function, which validates counters against Redis and returns HTTP 429 with standard headers when exceeded.
- **Fail-open protection** ensures that Redis connectivity issues do not block legitimate traffic, though limit checking is bypassed during outages.
- **Atomic storage** relies on [`src/rate-limit/redis-store.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/src/rate-limit/redis-store.ts) for consistent counter management across distributed MemoryProxy instances.

## Frequently Asked Questions

### What happens when a request exceeds the rate limit in MemoryProxy?

The `enforceRateLimit` function in [`src/rate-limit/guard.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/src/rate-limit/guard.ts) detects the violation and throws a `RateLimitExceededError`. The proxy catches this exception and returns an HTTP 429 response with headers indicating the limit type (TPM or QPM) and a JSON error body containing the code `input_tpm_exceeded` or similar, depending on which threshold was breached.

### How does MemoryProxy handle Redis connectivity issues?

If Redis is unreachable, the guard logic in [`src/rate-limit/guard.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/src/rate-limit/guard.ts) (lines 36-50) logs a warning and permits the request to continue in a fail-open mode. This ensures temporary Redis outages do not cause service downtime, though rate limits will not be enforced until connectivity is restored.

### Can I set different rate limits for different AI models in the same instance?

Yes. The admin API supports per-model granularity by requiring both `instance_id` and `model_id` parameters. When you create an override via PUT `/v3/admin/rate-limits`, MemoryProxy applies those specific limits only to requests matching that exact pair, allowing other models on the same instance to operate under different constraints or global defaults.

### Where are the rate limit counters stored?

All counters persist in Redis using the implementation in [`src/rate-limit/redis-store.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/src/rate-limit/redis-store.ts). The system leverages atomic Redis operations to increment and expire counters within rolling 60-second windows, ensuring accurate TPM and QPM tracking even when multiple MemoryProxy instances share the same Redis backend.