How Maigret Handles Rate Limiting and Throttling in OSINT Searches

Maigret detects rate limiting through HTTP 429 status codes and keyword matching in responses, but leaves throttling implementation and retry logic to the user rather than handling it automatically.

Maigret is a powerful asynchronous username investigation tool that searches across hundreds of social platforms using aiohttp as its core HTTP client. When scraping sites at scale, understanding how Maigret handles rate limiting and throttling helps prevent account restrictions and ensures reliable OSINT data collection. The tool identifies rate limits in two primary locations but deliberately avoids automatic back-off mechanisms.

Rate Limit Detection in Site Checks

Maigret's primary rate limit detection occurs in utils/site_check.py, where it monitors both HTTP status codes and response body content for indicators of throttling.

HTTP 429 Status Code Monitoring

After fetching a URL, the code checks the response status explicitly for the 429 Too Many Requests code. When detected, it appends a specific issue identifier to the diagnosis dictionary:

# utils/site_check.py (excerpt)

if result_claimed["status"] == 429:
    diagnosis["issues"].append("429 Rate Limited")

This creates a clear audit trail in the output, flagging exactly which sites have enforced rate limiting during the search process.

Textual Pattern Matching

Beyond status codes, Maigret performs case-insensitive string matching against the response body to catch rate-limit messages that may return HTTP 200 with embedded error text. The code searches for three specific indicators:

# utils/site_check.py (excerpt)

text_lower = response_text.lower()
diagnosis["rate_limit"] = any(
    m in text_lower for m in ["rate limit", "too many requests", "429"]
)

When any of these phrases appear in the response, the diagnosis["rate_limit"] field is set to True, allowing downstream logic to identify throttled sites even when the status code differs from 429.

OpenAI API Rate Limit Handling

The OpenAI integration in maigret/ai.py implements stricter error handling for API-specific rate limits. When the --ai option triggers a request to OpenAI's API, the code explicitly raises a RuntimeError rather than silently logging the issue:

# maigret/ai.py (excerpt)

async with aiohttp.ClientSession() as session:
    async with session.post(url, json=payload, headers=headers) as resp:
        if resp.status == 429:
            raise RuntimeError("OpenAI API rate limit exceeded (HTTP 429)")

This hard failure ensures users immediately recognize when OpenAI throttling occurs, preventing silent data gaps in AI-assisted investigations. Unlike site checks, this implementation aborts execution rather than continuing with partial results.

Connection Pooling and Default Throttling

While Maigret does not implement custom retry logic or exponential back-off, it relies on aiohttp's default connection pooling for basic throttling. The underlying HTTP client caps concurrent connections per host at approximately 100 by default, providing a rudimentary form of flow control without explicit rate limiting code in maigret/checking.py.

This means Maigret naturally limits concurrent requests to the same domain, but users must implement additional throttling externally when scraping hundreds of profiles across the same platform to avoid IP bans.

Summary

• Maigret detects rate limiting in utils/site_check.py by checking for HTTP 429 status codes and scanning response text for "rate limit", "too many requests", and "429".
• OpenAI API rate limits in maigret/ai.py raise a RuntimeError with the message "OpenAI API rate limit exceeded (HTTP 429)", halting execution immediately.
• The tool does not implement automatic retry loops, exponential back-off, or adaptive throttling, leaving these decisions to the user.
• Basic throttling occurs through aiohttp's default connection pool limits (≈100 concurrent connections per host), but this alone is insufficient for aggressive scanning.

Frequently Asked Questions

Does Maigret automatically retry requests after hitting a rate limit?

No, Maigret does not implement automatic retry logic or exponential back-off. When the code in utils/site_check.py detects a 429 status or rate-limit keywords, it logs the issue and continues. Users must catch these conditions and implement their own retry mechanisms if needed.

How can I identify which sites rate-limited my Maigret scan?

Check the issues array in the output for the string "429 Rate Limited" and the rate_limit boolean field in the diagnosis results. These are set in utils/site_check.py when the response contains rate-limit indicators, providing clear visibility into which platforms throttled your requests.

Why does the OpenAI integration abort on rate limits while site checks continue?

The OpenAI handler in maigret/ai.py raises a RuntimeError immediately upon receiving a 429 response because AI-assisted analysis represents a critical dependency. Continuing without AI results could produce misleading investigation outputs, whereas traditional username checks can proceed with partial data from other sites.

What is the maximum number of concurrent requests Maigret sends to a single site?

Maigret relies on aiohttp's default configuration, which limits concurrent connections per host to approximately 100. This provides basic protection against overwhelming servers, but aggressive scanning of a single platform may still trigger rate limits well before hitting this connection cap.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}