How FastProxy Implements Traffic Throttling: Token Bucket Rate Limiting in Go

FastProxy implements traffic throttling using a per-service token bucket rate limiter from golang.org/x/time/rate, storing limiters in a sync.Map and rejecting excess requests with HTTP 429 status codes.

FastProxy is an open-source Go proxy that protects upstream services through configurable traffic throttling. This article examines how the inproxy module implements per-service and per-endpoint rate limiting using token bucket algorithms, based on the actual source code in the kingson4wu/fast_proxy repository.

Token Bucket Architecture

FastProxy employs a token bucket rate limiter to control inbound traffic. This algorithm allows bursts up to a configured capacity while maintaining a steady long-term rate, making it ideal for API gateway scenarios.

Limiter Storage with sync.Map

The core throttling logic resides in inproxy/internal/limiter/limitManager.go. A global sync.Map named limitMap stores *rate.Limiter instances, keyed by a composite string serviceName_uri.

This design provides lock-free reads for the hot path, minimizing latency for requests that fall within rate limits.

Lazy Initialization and Per-Key Mutexes

When a request arrives, the IsLimit function checks limitMap for an existing limiter:

• If found, it immediately calls Allow() on the limiter. A false return triggers throttling.
• If not found, the function acquires a per-key mutex from a keyLocks map to prevent duplicate limiter creation.

The initialization sequence reads the configured QPS from inconfig.Get().ServiceQps(serviceName, uri), then creates a new limiter:

rate.NewLimiter(rate.Limit(qps), qps)

The bucket capacity equals the QPS value, allowing temporary bursts while enforcing the average rate.

Configuration and QPS Settings

Rate limits are defined in the in-proxy YAML configuration. Administrators specify QPS values per service and per URI pattern:

# examples/inproxy/config.yaml

services:
  example-service:
    "/api/v1/resource": 100   # 100 requests per second for this endpoint

    "/health": 500            # Higher limit for health checks

The ServiceQps method in inproxy/inconfig/config.go retrieves these values at runtime, enabling dynamic throttling without code changes.

Request Handling Integration

Throttling is enforced in inproxy/internal/proxy/httpclientProxy.go before forwarding requests upstream. The proxy checks the limiter status:

// inproxy/internal/proxy/httpclientProxy.go (excerpt)
if limiter.IsLimit(clientServiceName, requestPath) {
    // request exceeds the allowed rate → reject with 429 Too Many Requests
    w.WriteHeader(http.StatusTooManyRequests)
    return
}

Rejected requests receive an HTTP 429 Too Many Requests status code, signaling clients to implement backoff strategies.

Concurrency Safety Mechanisms

FastProxy's throttling implementation balances performance and correctness through two synchronization primitives:

• sync.Map provides lock-free reads for the common case where a limiter already exists, reducing contention on high-traffic endpoints.
• keyLocks map stores sync.Mutex instances for each unique service/URI key. This prevents race conditions during lazy initialization when multiple concurrent requests arrive for a new endpoint.

Practical Implementation Examples

Custom Handler Integration

You can leverage the limiter in custom handlers:

package myhandler

import (
    "net/http"
    "github.com/Kingson4Wu/fast_proxy/inproxy/internal/limiter"
)

func MyHandler(w http.ResponseWriter, r *http.Request) {
    svc := "example-service"
    uri := r.URL.Path

    if limiter.IsLimit(svc, uri) {
        http.Error(w, "Too Many Requests", http.StatusTooManyRequests)
        return
    }

    // normal processing …
    w.Write([]byte("OK"))
}

Manual Limiter Creation

For testing or specialized use cases, create limiters directly:

import (
    "golang.org/x/time/rate"
    "time"
)

func NewCustomLimiter(qps int) *rate.Limiter {
    // bucket size == burst capacity == qps
    return rate.NewLimiter(rate.Limit(qps), qps)
}

Summary

• FastProxy uses a token bucket algorithm via golang.org/x/time/rate to enforce per-service, per-endpoint rate limits.
• Limiters are stored in a sync.Map keyed by serviceName_uri and initialized lazily with per-key mutexes to prevent race conditions.
• Configuration occurs through YAML files specifying QPS values, read via inconfig.Get().ServiceQps().
• Exceeded requests receive HTTP 429 responses before reaching upstream services.
• The implementation prioritizes performance through lock-free reads and targeted synchronization only during limiter creation.

Frequently Asked Questions

What algorithm does FastProxy use for traffic throttling?

FastProxy implements the token bucket algorithm using the standard Go package golang.org/x/time/rate. This allows configurable QPS limits with burst capacity equal to the QPS value, accommodating temporary traffic spikes while maintaining steady-state rate control.

How does FastProxy handle concurrent requests to the same endpoint?

FastProxy uses a two-tier concurrency strategy. A global sync.Map provides lock-free reads for existing limiters, ensuring minimal overhead for high-traffic endpoints. For new endpoints, a per-key sync.Mutex (stored in a separate keyLocks map) serializes limiter creation, preventing duplicate limiters for the same serviceName_uri key.

What HTTP status code does FastProxy return when throttling requests?

When a request exceeds the configured rate limit, FastProxy returns HTTP 429 Too Many Requests. This occurs in inproxy/internal/proxy/httpclientProxy.go before the request reaches upstream services, allowing clients to detect throttling and implement appropriate backoff strategies.

Can I configure different rate limits for different API endpoints?

Yes. FastProxy supports granular rate limiting through YAML configuration. You define QPS values per service and per URI pattern in the configuration file, such as "/api/v1/resource": 100 for 100 requests per second on that specific endpoint, while allowing different limits for other paths like "/health": 500.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}