How OpenSEO Enforces robots.txt and Meta Robots Tags Compliance

OpenSEO enforces robots.txt and meta robots tags compliance through a multi-phase pipeline that deterministically parses access rules, filters URLs before crawling, validates links against parsed policies, and extracts indexability directives from both HTML content and HTTP headers.

OpenSEO (every-app/open-seo) is an open-source SEO auditing tool that guarantees robots.txt and meta robots tags compliance through tightly-coupled validation gates. The system prevents unauthorized URL fetching by parsing robots.txt deterministically, checkpointing results for replay accuracy, and applying policy checks at both the frontier seeding and active crawl stages. This article examines the specific TypeScript implementations that ensure crawls honor webmaster directives across discovery, crawling, and reporting phases.

Phase 1: robots.txt Discovery and Deterministic Parsing

The compliance workflow initiates during the discovery phase by retrieving and parsing the site's robots.txt file with strict constraints.

Fetching with Size Limits

OpenSEO fetches raw robots.txt content via fetchRobotsTxtText, enforcing a strict 500 KiB size limit to prevent parser abuse. This constraint ensures the crawler processes only valid, reasonably-sized policy files while rejecting oversized or potentially malformed responses that could disrupt the audit.

Source: src/server/lib/audit/discovery.ts lines 40-68

Parsing with robots-parser

The fetched text is processed by parseRobotsTxt, which utilizes the robots-parser library to generate a deterministic RobotsResult object. This object exposes an isAllowed(url) method that evaluates URL paths against parsed Disallow and Allow directives.

The parsing operation is deterministic—identical file contents always produce identical allow/deny decisions. OpenSEO checkpoints these results to ensure audit replays apply the exact same rules originally used during the initial crawl.

// Parse robots.txt during discovery phase
import { parseRobotsTxt } from "@/server/lib/audit/discovery";

const robots = parseRobotsTxt(origin, robotsTxtText);
if (!robots.isAllowed(url)) {
  // Skip URLs disallowed by robots.txt
}

Source: src/server/lib/audit/discovery.ts lines 40-68

Phase 2: URL Seeding and Frontier Filtering

During runDiscoveryPhase, OpenSEO filters every candidate URL against the parsed robots object before adding it to the crawl frontier. This includes URLs discovered from XML sitemaps and the initial start URL.

The system normalizes each candidate and invokes robots.isAllowed(normalized) as a gatekeeping check. Any URL violating the robots.txt policy is excluded from the crawl queue before network requests are initiated, ensuring zero unauthorized fetches from seed sources.

Source: src/server/workflows/siteAuditWorkflowPhases.ts lines 49-53

Phase 3: Per-URL Validation During Crawling

After seeding, the crawler validates each dequeued link through shouldQueueCrawlLink in the crawl phase. This function performs three validation checks: same-origin verification, crawlability assessment, and crucially, validation against the previously parsed robots object.

If robots.isAllowed returns false for a given URL, the crawler abandons the link immediately, preventing any HTTP request that would violate the site's robots.txt directives.

Source: src/server/workflows/siteAuditWorkflowCrawl.ts lines 70-77

Phase 4: Meta Robots and X-Robots-Tag Processing

Once a page is fetched, OpenSEO extracts indexability directives from both HTML content and HTTP headers to ensure meta robots tags compliance.

Extracting Directives

The crawlPage function utilizes analyzeHtml to parse the <meta name="robots"> tag content from the document head. Simultaneously, it reads the X-Robots-Tag HTTP response header. These directives are merged into a unified robotsDirectives object.

Determining Indexability

The system scans the merged directives for the "noindex" instruction. If present, the page receives an isIndexable: false flag. This boolean determines whether downstream issue reporters flag the URL as non-indexable content.

// Crawl a page and respect meta-robots / X-Robots-Tag
import { crawlPage } from "@/server/workflows/site-audit-workflow-helpers";

const result = await crawlPage(url, depth, inSitemap);
if (!result.isIndexable) {
  // Page is flagged as non-indexable (noindex in meta or header)
}

Source: src/server/workflows/site-audit-workflow-helpers.ts lines 38-44

Phase 5: Persisting Compliance Data

OpenSEO stores robots-related metadata in the audit database schema. The system records robots_meta and x_robots_tag values for each crawled page, enabling the audit UI to surface "Indexability" issues for pages marked as non-indexable.

This persistence layer ensures that compliance determinations—whether originating from robots.txt denials or meta-tag exclusions—are documented and available for longitudinal reporting.

Source: src/db/audit.schema.ts lines 73-78

Summary

• Deterministic parsing: OpenSEO uses the robots-parser library in parseRobotsTxt to generate consistent isAllowed() decisions that are checkpointed for replay accuracy.
• Frontier filtering: The runDiscoveryPhase function validates sitemap and seed URLs against parsed robots.txt rules before adding them to the crawl queue.
• Link-level gates: The shouldQueueCrawlLink validator enforces robots.txt compliance during the active crawl phase, preventing forbidden fetches.
• Meta-tag extraction: The crawlPage helper extracts and merges directives from <meta name="robots"> tags and X-Robots-Tag headers to set the isIndexable flag.
• Database persistence: Schema columns robots_meta and x_robots_tag store compliance data for audit reporting and UI issue surfacing.

Frequently Asked Questions

How does OpenSEO handle oversized robots.txt files?

OpenSEO enforces a strict 500 KiB size limit when fetching robots.txt via fetchRobotsTxtText. Files exceeding this threshold are truncated or rejected, preventing parser denial-of-service attacks and ensuring processing of only valid policy files.

What happens if a URL is allowed by robots.txt but contains a noindex meta tag?

The crawler will fetch the page (since robots.txt permits it), but the crawlPage function will detect the "noindex" directive in the merged robotsDirectives. The page receives isIndexable: false, and the audit UI flags it as a non-indexable page without treating it as a crawl error.

Does OpenSEO support the X-Robots-Tag HTTP header?

Yes. OpenSEO explicitly reads the X-Robots-Tag header alongside HTML meta robots tags. Both sources are merged in robotsDirectives, ensuring HTTP-header-based directives receive equal weight in indexability determinations.

Where does OpenSEO validate URLs against robots.txt during the crawl?

Validation occurs at two critical points: during runDiscoveryPhase (filtering sitemap URLs before queueing) and in shouldQueueCrawlLink (checking individual links when dequeued from the crawl frontier). This dual-gate approach ensures compliance throughout the entire audit pipeline.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}