How to Check AI Crawler Access and robots.txt with GEO-SEO Claude

Audit your site's visibility to AI search engines by analyzing robots.txt directives for GPTBot, ClaudeBot, and PerplexityBot using the geo crawlers command.

The zubair-trabzada/geo-seo-claude toolkit provides automated analysis of how AI crawlers interact with your site's robots.txt file. This guide explains how to check AI crawler access and generate actionable reports that maximize your Generative Engine Optimization (GEO) score.

Web Fetching and robots.txt Parsing

The foundation of the audit resides in scripts/fetch_page.py, which contains the fetch_robots_txt() function. This utility downloads a domain's robots.txt file, parses User-Agent rules, and records any fetch or parsing errors.

The parser specifically targets directives for 14+ recognized AI crawlers including GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and standard search engine bots. It evaluates each Allow or Disallow entry to determine whether critical AI systems can access your content.

The GEO-AI-Visibility Agent Orchestration

Located at agents/geo-ai-visibility.md, the agent drives the complete audit workflow. This component maps robots.txt directives to specific AI crawlers and produces the GEO-CRAWLER-ACCESS.md report containing your AI-visibility score and recommendations.

The agent also scans for the emerging Content-Signal directive (per IETF draft draft-romm-aipref-contentsignals). When present in robots.txt, it parses key-value pairs such as ai-train=yes to determine training permissions, reporting these as non-scored recommendations.

Running AI Crawler Checks from the Command Line

Execute the audit using the geo CLI suite documented in docs/commands-reference.md:

geo crawlers https://example.com

The command internally calls the crawler-access agent and outputs a formatted summary:

🤖 AI Crawler Access Report – example.com
-------------------------------------------------
GPTBot        → ALLOWED
ClaudeBot     → ALLOWED
PerplexityBot → BLOCKED  (recommendation: allow in robots.txt)
OAI-SearchBot → ALLOWED
GoogleBot     → ALLOWED

AI Visibility Score: 85 / 100
Recommended robots.txt snippet:
User-agent: PerplexityBot
Disallow:

Understanding the AI Visibility Score Methodology

According to docs/scoring-methodology.md, crawlability accounts for up to 15 points in the overall GEO assessment. The scoring deducts 15 points for each blocked critical AI crawler among the five priority agents: GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and GoogleBot.

A well-formed robots.txt that explicitly Allow these bots achieves the maximum crawlability score. Conversely, blocking any primary AI crawler significantly impacts your site's discoverability in AI-driven search engines.

Programmatic Access to robots.txt Analysis

Integrate the fetcher directly into Python workflows:

from scripts.fetch_page import fetch_robots_txt

result = fetch_robots_txt("https://example.com")
print(result["status"])          # e.g. "OK"

print(result["directives"])      # dict of User-Agent → Allow/Disallow paths

print(result["errors"])          # any fetch or parsing errors

This approach enables batch processing of multiple domains or custom integration into CI/CD pipelines for automated SEO monitoring.

Advanced: Crawler-Specific Configuration

Detailed specifications for each supported AI crawler reside in skills/geo-crawlers/SKILL.md. This file documents user-agent strings, crawling behaviors, and specific recommendations for each bot.

When the audit detects blocked crawlers, it generates copy-paste-ready robots.txt snippets tailored to the specific AI systems you need to accommodate, ensuring rapid implementation of GEO best practices.

Summary

• Primary audit function: fetch_robots_txt() in scripts/fetch_page.py downloads and parses robots.txt for AI-crawler directives.
• Command-line interface: Run geo crawlers <url> to execute the complete audit via the agent defined in agents/geo-ai-visibility.md.
• Scoring impact: AI crawlability contributes up to 15 points; blocking critical crawlers (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, GoogleBot) deducts 15 points each.
• Emerging standards: The system detects Content-Signal directives per IETF draft specifications for AI training permissions.
• Deliverables: Each audit generates GEO-CRAWLER-ACCESS.md with visibility scores and actionable robots.txt recommendations.

Frequently Asked Questions

How does GEO-SEO Claude identify which AI crawlers to check?

The system references skills/geo-crawlers/SKILL.md which maintains a registry of 14+ recognized AI crawlers including GPTBot, ClaudeBot, and PerplexityBot. The agent maps these user-agent strings against your robots.txt directives to determine access permissions.

What happens if my site blocks GPTBot or ClaudeBot?

According to the scoring methodology in docs/scoring-methodology.md, blocking any of the five critical AI crawlers (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, or GoogleBot) results in a 15-point deduction from your AI visibility score. The audit report will flag this as a critical issue with a specific remediation snippet.

Can I check multiple domains simultaneously?

While the geo crawlers command analyzes single URLs, you can programmatically batch process domains by importing fetch_robots_txt() from scripts/fetch_page.py into a Python script. This enables automated monitoring of multiple properties through custom workflows.

Does the tool support the new Content-Signal robots.txt directive?

Yes. The agent in agents/geo-ai-visibility.md scans for the Content-Signal directive per IETF draft draft-romm-aipref-contentsignals. It parses values like ai-train=yes and reports them as recommendations, though these currently do not affect the numerical GEO score.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}