How to Configure Rate Limiting for API Calls Made by Ax

Ax provides a built-in token-based rate limiter that tracks API usage and automatically delays requests until your configured token budget replenishes.

The Ax library includes sophisticated rate limiting capabilities designed to prevent quota exhaustion when calling LLM providers like Groq, Mistral, and X-Grok. By configuring rate limiting for API calls made by Ax, you can enforce token-per-minute budgets or implement completely custom throttling strategies that match your specific infrastructure constraints.

Understanding Ax's Token-Based Rate Limiter

At the core of Ax's rate limiting system is the AxRateLimiterTokenUsage class, implemented in src/ax/util/rate-limit.ts. This class implements a token bucket algorithm that manages two critical parameters:

• Token budget: The maximum number of tokens allowed per time window (typically per minute)
• Refill rate: How quickly tokens regenerate (tokensPerMinute / 60)

When you configure an AI provider in Ax, the constructor automatically instantiates this limiter unless you provide a custom alternative.

Configuring Rate Limits on AI Providers

Ax exposes rate limiting configuration through the options parameter when constructing any AI provider. The system checks for a custom rateLimiter function first, then falls back to creating an AxRateLimiterTokenUsage instance using the tokensPerMinute value.

Setting a Custom Token Budget

The simplest way to configure rate limiting is to set the tokensPerMinute option when creating a provider. For example, when using Groq (which defaults to 4800 tokens per minute), you can override this budget:

import { AxAIGroq } from '@ax-llm/ax';

const groq = new AxAIGroq({
  apiKey: process.env.GROQ_APIKEY!,
  // Limit to 3,000 tokens per minute
  options: { tokensPerMinute: 3000 },
});

In src/ax/ai/groq/api.ts (lines 118-124), the constructor passes this value to new AxRateLimiterTokenUsage(tokensPerMin, tokensPerMin / 60), creating a limiter that refills at 50 tokens per second when configured for 3,000 tokens per minute.

Implementing a Custom Rate Limiter Function

For advanced use cases—such as request-count-based limiting, external quota systems, or multi-tenant throttling—you can provide a custom rateLimiter function via the provider options:

import { AxAIGroq, type AxRateLimiterFunction } from '@ax-llm/ax';

let callsInCurrentMinute = 0;
let minuteStart = Date.now();

const requestCountLimiter: AxRateLimiterFunction = async (fn, info) => {
  const now = Date.now();
  
  // Reset counter every minute
  if (now - minuteStart >= 60_000) {
    minuteStart = now;
    callsInCurrentMinute = 0;
  }
  
  // Enforce max 30 calls per minute
  if (callsInCurrentMinute >= 30) {
    const waitTime = minuteStart + 60_000 - now;
    await new Promise((r) => setTimeout(r, waitTime));
    minuteStart = Date.now();
    callsInCurrentMinute = 0;
  }
  
  callsInCurrentMinute++;
  return fn();
};

const groq = new AxAIGroq({
  apiKey: process.env.GROQ_APIKEY!,
  options: { rateLimiter: requestCountLimiter },
});

The provider invokes this function before every LLM request, passing the original request function and metadata about the call. Your custom logic can inspect info.modelUsage?.tokens?.totalTokens to make decisions based on anticipated token consumption.

Disabling Rate Limiting

While not recommended for production, you can disable rate limiting by providing a no-op function:

const noopLimiter: AxRateLimiterFunction = async (fn) => fn();

const groq = new AxAIGroq({
  apiKey: process.env.GROQ_APIKEY!,
  options: { rateLimiter: noopLimiter },
});

Alternatively, setting tokensPerMinute to an extremely high value effectively disables throttling while maintaining the token-tracking infrastructure.

Practical Code Examples

Basic Token Budget Configuration

This example configures a Groq provider with a conservative 2,000 tokens per minute limit, suitable for applications with strict quota constraints:

import { AxAIGroq } from '@ax-llm/ax';

const groq = new AxAIGroq({
  apiKey: process.env.GROQ_APIKEY!,
  options: { tokensPerMinute: 2000 },
});

await groq.chat({ 
  messages: [{ role: 'user', content: 'Explain rate limiting strategies' }] 
});

As implemented in src/ax/ai/groq/api.ts, this configuration creates an AxRateLimiterTokenUsage instance that refills approximately 33 tokens per second.

Stand-Alone Rate Limiter Usage

For scenarios where you need to protect non-Ax API calls or implement custom batching logic, you can use AxRateLimiterTokenUsage directly:

import { AxRateLimiterTokenUsage } from '@ax-llm/ax/util';

const limiter = new AxRateLimiterTokenUsage(
  5000,        // tokensPerMinute (bucket size)
  5000 / 60,   // refillRate (tokens per second)
  { debug: true }
);

async function protectedApiCall(fn: () => Promise<any>, tokenCount: number) {
  await limiter.acquire(tokenCount);
  return fn();
}

// Usage: protect a function that consumes 120 tokens
await protectedCall(() => fetchExpensiveData(), 120);

The acquire() method, defined in src/ax/util/rate-limit.ts, handles the token bucket logic, including waiting for sufficient tokens to become available before resolving.

Summary

• Ax implements token-based rate limiting through the AxRateLimiterTokenUsage class in src/ax/util/rate-limit.ts, using a token bucket algorithm that tracks consumption and refill rates.
• Configure limits via tokensPerMinute when constructing AI providers like AxAIGroq to automatically create a token budget that refills at tokensPerMinute / 60 tokens per second.
• Implement custom strategies by providing a rateLimiter function in the provider options, enabling request-count limits, external quota integration, or multi-tenant throttling.
• Reference implementations in src/ax/ai/groq/api.ts, src/ax/ai/mistral/api.ts, and src/ax/ai/x-grok/api.ts demonstrate how providers instantiate limiters and expose configuration options.

Frequently Asked Questions

How does Ax calculate the token refill rate when using tokensPerMinute?

Ax divides the tokensPerMinute value by 60 to determine the refill rate in tokens per second. For example, if you configure tokensPerMinute: 3000, the limiter refills at 50 tokens per second. This calculation occurs in the provider constructor, such as in src/ax/ai/groq/api.ts, where the value is passed to new AxRateLimiterTokenUsage(tokensPerMin, tokensPerMin / 60).

Can I use Ax's rate limiter for non-Ax API calls?

Yes, you can instantiate AxRateLimiterTokenUsage directly from src/ax/util/rate-limit.ts and use it to protect any asynchronous function. Create an instance with your desired token budget and refill rate, then call await limiter.acquire(tokenCount) before executing your function. This approach works for external APIs, database queries, or any resource you want to throttle based on token consumption.

What happens if I don't configure tokensPerMinute or a custom rateLimiter?

If you do not specify either option, Ax uses provider-specific defaults. For example, the Groq provider defaults to 4800 tokens per minute as defined in src/ax/ai/groq/api.ts. The constructor automatically creates an AxRateLimiterTokenUsage instance with this default value, ensuring that all API calls are throttled according to the token bucket algorithm even without explicit configuration.

How do I implement a request-count-based limit instead of token-based?

Pass a custom rateLimiter function to the provider options that tracks the number of requests rather than tokens. Your function should accept fn (the original request) and info (call metadata), implement your counting logic (such as incrementing a counter and checking against a maximum), and return fn() once conditions are met. This approach allows you to enforce limits like "30 requests per minute" regardless of token consumption per request.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}