Rate Limiting Techniques for Java Microservices: 5 Algorithms and Frameworks Explained

Rate limiting techniques for Java microservices protect distributed systems from traffic spikes using algorithms like Token Bucket and framework integrations such as Spring Cloud Gateway with Redis.

The doocs/advanced-java repository provides production-ready implementations of classic throttling algorithms and enterprise framework configurations. Understanding these patterns is essential for maintaining system stability and preventing cascading failures in high-concurrency environments.

Core Rate Limiting Algorithms

The repository documents four fundamental algorithms in docs/high-concurrency/how-to-limit-current.md, each offering different trade-offs between simplicity and burst handling.

Fixed Window Counter

The simplest approach tracks request counts within a fixed time window. While easy to implement, it suffers from burst issues at window boundaries.

In the source, the Counter class uses an AtomicInteger to track requests against a configurable limit per timeout milliseconds:

public class Counter {
    private final int limit = 10;
    private final long timeout = 1000L;
    private long windowStart = System.currentTimeMillis();
    private final AtomicInteger reqCount = new AtomicInteger(0);

    public boolean limit() {
        long now = System.currentTimeMillis();
        if (now - windowStart < timeout) {
            return reqCount.incrementAndGet() <= limit;
        } else {
            windowStart = now;
            reqCount.set(1);
            return true;
        }
    }
}

This implementation resets the counter atomically when the window expires, as shown in lines 15-49 of the concurrency documentation.

Sliding Window Counter

To smooth traffic spikes, the sliding window divides time into smaller slots. The TimeWindow class uses a ConcurrentLinkedQueue to store timestamps and a background cleaner thread to remove expired entries.

public class TimeWindow {
    private final ConcurrentLinkedQueue<Long> queue = new ConcurrentLinkedQueue<>();
    private final int max;          // max requests per window
    private final int seconds;      // window length

    public TimeWindow(int max, int seconds) {
        this.max = max;
        this.seconds = seconds;
    }

    public boolean tryAcquire() {
        long now = System.currentTimeMillis();
        clean(now);
        if (queue.size() >= max) {
            return false;
        }
        queue.offer(now);
        return true;
    }

    private void clean(long now) {
        long limit = now - seconds * 1000L;
        while (true) {
            Long ts = queue.peek();
            if (ts == null || ts > limit) break;
            queue.poll();
        }
    }
}

This approach provides smoother rate distribution compared to the fixed window, implemented in lines 60-84.

Leaky Bucket Algorithm

The leaky bucket models traffic as water flowing into a bucket with a fixed outflow rate. When the bucket overflows, requests drop. The LeakBucket class maintains a nowSize field representing current water level and drains it at a constant rate:

public class LeakBucket {
    private final int capacity;     // bucket capacity
    private final int rate;         // outflow rate per second
    private int nowSize;            // current water level
    private long lastLeakTime;

    public synchronized boolean tryConsume() {
        leak();
        if (nowSize < capacity) {
            nowSize++;
            return true;
        }
        return false;
    }

    private void leak() {
        long now = System.currentTimeMillis();
        int leakAmount = (int) ((now - lastLeakTime) / 1000 * rate);
        nowSize = Math.max(0, nowSize - leakAmount);
        lastLeakTime = now;
    }
}

This algorithm enforces a strict output rate regardless of input burstiness, detailed in lines 90-124.

Token Bucket Algorithm

The most flexible algorithm for microservices, token bucket adds tokens at a steady rate while requests consume them. The TokenBucket implementation refills tokens based on elapsed time between requests:

public class TokenBucket {
    private final double capacity;   // max tokens
    private final double rate;       // tokens added per millisecond
    private double tokens;
    private long lastRefill;

    public TokenBucket(double capacity, double rate) {
        this.capacity = capacity;
        this.rate = rate;
        this.tokens = capacity;
        this.lastRefill = System.currentTimeMillis();
    }

    public synchronized boolean tryConsume() {
        refill();
        if (tokens < 1) return false;
        tokens -= 1;
        return true;
    }

    private void refill() {
        long now = System.currentTimeMillis();
        double added = (now - lastRefill) * rate;
        tokens = Math.min(capacity, tokens + added);
        lastRefill = now;
    }
}

This allows configurable burst capacity while maintaining long-term rate limits, as implemented in lines 130-168.

Framework-Based Rate Limiting Solutions

For production microservices, the repository recommends integrating with established frameworks rather than custom implementations.

Spring Cloud Gateway with Redis

Spring Cloud Gateway provides distributed rate limiting using Redis as a backing store for a token-bucket implementation. Configuration uses application.yml with a KeyResolver bean to extract client identifiers (typically IP addresses):

spring:
  cloud:
    gateway:
      routes:
        - id: requestratelimiter_route
          uri: lb://my-service
          predicates:
            - Path=/api/**
          filters:
            - name: RequestRateLimiter
              args:
                redis-rate-limiter.replenishRate: 5
                redis-rate-limiter.burstCapacity: 10
                key-resolver: '#{@remoteAddrKeyResolver}'

The corresponding KeyResolver bean:

@Bean
public KeyResolver remoteAddrKeyResolver() {
    return exchange -> Mono.just(
        Objects.requireNonNull(exchange.getRequest().getRemoteAddress())
               .getAddress().getHostAddress());
}

This configuration applies per-IP limiting across all gateway instances, documented in the repository's gateway section.

Alibaba Sentinel

Sentinel offers declarative flow control via rules stored in Nacos. It supports QPS limiting, thread-count constraints, and advanced strategies like warm-up and queueing:

[
  {
    "resource": "/order",
    "limitApp": "default",
    "grade": 1,
    "count": 20,
    "strategy": 0,
    "controlBehavior": 0,
    "clusterMode": false
  }
]

The grade: 1 indicates QPS-based limiting, while grade: 0 would indicate thread-count limiting. This approach centralizes rate limiting policies in Nacos for dynamic updates without deployment.

Hystrix Thread Pool Isolation

For concurrency-based limiting rather than rate-based, Hystrix isolates commands in dedicated thread pools. According to docs/high-availability/hystrix-thread-pool-current-limiting.md, this approach limits simultaneous calls per command key, preventing slow dependencies from exhausting shared thread pools.

Configure thread pool size and queue capacity:

HystrixCommand.Setter.withGroupKey(HystrixCommandGroupKey.Factory.asKey("ExampleGroup"))
    .andCommandPropertiesDefaults(
        HystrixCommandProperties.Setter()
            .withExecutionIsolationThreadTimeoutInMilliseconds(500))
    .andThreadPoolPropertiesDefaults(
        HystrixThreadPoolProperties.Setter()
            .withCoreSize(10)
            .withMaxQueueSize(100));

This protects downstream services from being overwhelmed by concurrent requests, complementing rate-based algorithms.

Architectural Best Practices for Microservices

When implementing rate limiting techniques for Java microservices, consider these strategies from the doocs/advanced-java repository:

• Choose appropriate granularity: Per-endpoint limits protect specific resources, while per-user limits prevent individual clients from monopolizing capacity. The Spring Cloud Gateway KeyResolver demonstrates per-IP limiting.
• Use distributed stores for multi-instance deployments: Local counters only protect a single JVM. Redis-backed solutions ensure consistent limiting across horizontally scaled services.
• Layer multiple algorithms: Combine an in-memory token bucket for fast-path rejection with a Redis fallback for distributed burst protection.
• Monitor and instrument: Expose QPS, rejection counts, and bucket fill levels via Micrometer or Prometheus. The repository emphasizes tuning limits based on observed metrics rather than theoretical calculations.

Summary

• Fixed window counters offer simple implementation but allow bursts at window boundaries, implemented via AtomicInteger in docs/high-concurrency/how-to-limit-current.md.
• Sliding window counters smooth traffic using ConcurrentLinkedQueue to track individual request timestamps.
• Leaky bucket algorithms enforce strict output rates using a drain mechanism with nowSize tracking.
• Token bucket implementations provide flexible burst handling by calculating token refill based on elapsed time between requests.
• Framework integrations like Spring Cloud Gateway (Redis), Sentinel (Nacos rules), and Hystrix (thread pools) provide production-ready distributed limiting for microservices architectures.

Frequently Asked Questions

What is the difference between leaky bucket and token bucket algorithms?

The leaky bucket enforces a strict, constant outflow rate regardless of input patterns, dropping excess traffic immediately. It is ideal for smoothing bursty traffic into steady streams. The token bucket allows bursts up to the bucket capacity while maintaining a long-term average rate, making it more flexible for microservices that need to handle occasional traffic spikes without dropping requests.

How does Spring Cloud Gateway implement distributed rate limiting?

Spring Cloud Gateway uses Redis as a shared state store to implement a distributed token bucket. The RequestRateLimiter filter calculates token availability across all gateway instances using Redis Lua scripts for atomic operations. The KeyResolver bean determines the limiting granularity (IP, user, or endpoint), ensuring consistent enforcement in clustered deployments.

When should I use Sentinel instead of Gateway rate limiting?

Use Alibaba Sentinel when you need centralized, dynamic rule management via Nacos or when requiring advanced control behaviors like warm-up (gradual ramp-up) or uniform rate pacing. Use Spring Cloud Gateway limiting for edge-level protection at the API gateway layer. Often, both are used together: Gateway for coarse-grained perimeter defense and Sentinel for fine-grained service-level protection.

Can thread pool limiting replace rate limiting in Hystrix?

No, they serve different purposes. Thread pool limiting restricts concurrent execution to prevent resource exhaustion from slow dependencies, but it does not limit requests per second. A service could receive 1000 QPS while the thread pool only allows 10 concurrent executions, queueing or rejecting the rest. For complete protection, combine Hystrix thread pools with a rate limiter (Token Bucket or Sentinel) to control both throughput and concurrency.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}