How to Configure Rate Limits for MemoryProxy Services: Global Defaults and Per-Model Overrides

MemoryProxy enforces token-per-minute (TPM) and query-per-minute (QPM) caps through a Redis-backed limiter that supports both global YAML configuration and dynamic per-instance, per-model overrides via the administrator API.

The TencentDB-Agent-Memory repository provides MemoryProxy, a high-performance AI model proxy that protects upstream providers and ensures fair resource allocation through configurable rate limiting. Understanding how to configure rate limits for MemoryProxy services allows operators to set baseline usage policies while retaining granular control over specific workloads.

Global Rate Limit Configuration via YAML

Setting Default TPM and QPM Values

MemoryProxy establishes baseline limits through the rateLimit section in config.example.yaml (typically copied to config.yaml). These values apply to every request unless explicitly overridden via the admin API.

rateLimit:
  tpm: 1000000   # input tokens allowed in the last 60s

  qpm: 100       # requests allowed in the last 60s

The src/config.ts module parses these values at startup, and the limiter instantiation occurs in src/rate-limit/guard.ts. The tpm field controls the rolling window for input tokens, while qpm restricts the total number of API calls per minute.

Redis Storage Configuration

The limiter requires Redis for atomic counter operations. Configure the connection parameters in the same YAML file under the redis key. The src/rate-limit/redis-store.ts implementation handles all counter persistence and expiration logic, ensuring consistent rate tracking even when multiple MemoryProxy instances share the same Redis cluster.

Dynamic Overrides via Admin API

While global settings provide a safety net, production deployments often require different limits for specific customers or model tiers. The admin API defined in src/routes/rate-limits.ts enables runtime adjustments.

Querying Effective Limits

To inspect the current limits for a specific combination of instance and model, issue a GET request to /v3/admin/rate-limits. The handler in src/routes/rate-limits.ts returns whether the effective limits derive from global defaults or an explicit override.

curl -G http://localhost:8096/v3/admin/rate-limits \
  -H "Authorization: Bearer <admin-api-key>" \
  --data-urlencode "instance_id=inst-abc123" \
  --data-urlencode "model_id=gpt-4o-mini"

Creating Per-Instance, Per-Model Overrides

Administrators can establish custom caps using the PUT method. This creates a persistent override in Redis that takes precedence over the global configuration for the specified instance_id and model_id pair.

curl -X PUT http://localhost:8096/v3/admin/rate-limits \
  -H "Authorization: Bearer <admin-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
        "instance_id": "inst-abc123",
        "model_id": "gpt-4o-mini",
        "input_tpm": 250000,
        "qpm": 50
      }'

Removing Custom Limits

To revert an instance-model pair to the global defaults, send a DELETE request. The route handler removes the override record from the underlying store.

curl -X DELETE http://localhost:8096/v3/admin/rate-limits \
  -H "Authorization: Bearer <admin-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
        "instance_id": "inst-abc123",
        "model_id": "gpt-4o-mini"
      }'

Rate Limit Enforcement and Error Handling

The enforceRateLimit Implementation

Every incoming request passes through the enforceRateLimit function in src/rate-limit/guard.ts. This function queries the current counters from src/rate-limit/redis-store.ts and compares them against the effective limits. If the request would exceed either the TPM or QPM threshold, the function throws a RateLimitExceededError.

Response Headers and Status Codes

When limits are exceeded, MemoryProxy returns HTTP 429 (Too Many Requests) with informative headers. As implemented in src/rate-limit/guard.ts (lines 66-70), the response includes:

• X-Ratelimit-Limit-Input-Tokens – The configured TPM cap
• X-Ratelimit-Remaining-Input-Tokens – Tokens remaining in the current window
• X-Ratelimit-Limit-Requests – The configured QPM cap
• X-Ratelimit-Remaining-Requests – Requests remaining in the current window
• Retry-After – Seconds until the window resets

The JSON error body follows OpenAI-compatible formatting:

HTTP/1.1 429 Too Many Requests
Content-Type: application/json; charset=utf-8
Retry-After: 30
X-Ratelimit-Limit-Input-Tokens: 250000
X-Ratelimit-Remaining-Input-Tokens: 0

{
  "error": {
    "message": "该模型输入 Token 用量已达上限，请稍后重试",
    "type": "rate_limit_error",
    "code": "input_tpm_exceeded"
  }
}

Fail-Open Behavior for High Availability

If Redis becomes unreachable, the limiter enters a degraded mode. According to the error handling logic in src/rate-limit/guard.ts (lines 36-50), the proxy logs a warning but allows the request to proceed. This fail-open design prevents Redis outages from causing total service disruption, though rate limits will not be enforced during the connectivity gap.

Summary

• Configure global defaults in config.example.yaml under the rateLimit key to establish baseline TPM and QPM values for all traffic.
• Manage overrides dynamically using the /v3/admin/rate-limits endpoints implemented in src/routes/rate-limits.ts to customize limits per instance_id and model_id.
• Enforcement occurs in src/rate-limit/guard.ts via the enforceRateLimit function, which validates counters against Redis and returns HTTP 429 with standard headers when exceeded.
• Fail-open protection ensures that Redis connectivity issues do not block legitimate traffic, though limit checking is bypassed during outages.
• Atomic storage relies on src/rate-limit/redis-store.ts for consistent counter management across distributed MemoryProxy instances.

Frequently Asked Questions

What happens when a request exceeds the rate limit in MemoryProxy?

The enforceRateLimit function in src/rate-limit/guard.ts detects the violation and throws a RateLimitExceededError. The proxy catches this exception and returns an HTTP 429 response with headers indicating the limit type (TPM or QPM) and a JSON error body containing the code input_tpm_exceeded or similar, depending on which threshold was breached.

How does MemoryProxy handle Redis connectivity issues?

If Redis is unreachable, the guard logic in src/rate-limit/guard.ts (lines 36-50) logs a warning and permits the request to continue in a fail-open mode. This ensures temporary Redis outages do not cause service downtime, though rate limits will not be enforced until connectivity is restored.

Can I set different rate limits for different AI models in the same instance?

Yes. The admin API supports per-model granularity by requiring both instance_id and model_id parameters. When you create an override via PUT /v3/admin/rate-limits, MemoryProxy applies those specific limits only to requests matching that exact pair, allowing other models on the same instance to operate under different constraints or global defaults.

Where are the rate limit counters stored?

All counters persist in Redis using the implementation in src/rate-limit/redis-store.ts. The system leverages atomic Redis operations to increment and expire counters within rolling 60-second windows, ensuring accurate TPM and QPM tracking even when multiple MemoryProxy instances share the same Redis backend.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}