# How OpenSEO Enforces robots.txt and Meta Robots Tags Compliance

> OpenSEO ensures robots.txt and meta robots tags compliance by parsing rules, filtering URLs, validating links, and extracting directives from content and headers. Maintain SEO integrity effortlessly.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: how-to-guide
- Published: 2026-09-10

---

**OpenSEO enforces robots.txt and meta robots tags compliance through a multi-phase pipeline that deterministically parses access rules, filters URLs before crawling, validates links against parsed policies, and extracts indexability directives from both HTML content and HTTP headers.**

OpenSEO (every-app/open-seo) is an open-source SEO auditing tool that guarantees **robots.txt and meta robots tags compliance** through tightly-coupled validation gates. The system prevents unauthorized URL fetching by parsing [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) deterministically, checkpointing results for replay accuracy, and applying policy checks at both the frontier seeding and active crawl stages. This article examines the specific TypeScript implementations that ensure crawls honor webmaster directives across discovery, crawling, and reporting phases.

## Phase 1: robots.txt Discovery and Deterministic Parsing

The compliance workflow initiates during the discovery phase by retrieving and parsing the site's [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) file with strict constraints.

### Fetching with Size Limits

OpenSEO fetches raw [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) content via `fetchRobotsTxtText`, enforcing a strict **500 KiB size limit** to prevent parser abuse. This constraint ensures the crawler processes only valid, reasonably-sized policy files while rejecting oversized or potentially malformed responses that could disrupt the audit.

Source: [`src/server/lib/audit/discovery.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts) lines 40-68

### Parsing with robots-parser

The fetched text is processed by `parseRobotsTxt`, which utilizes the `robots-parser` library to generate a deterministic `RobotsResult` object. This object exposes an `isAllowed(url)` method that evaluates URL paths against parsed `Disallow` and `Allow` directives.

The parsing operation is **deterministic**—identical file contents always produce identical allow/deny decisions. OpenSEO checkpoints these results to ensure audit replays apply the exact same rules originally used during the initial crawl.

```typescript
// Parse robots.txt during discovery phase
import { parseRobotsTxt } from "@/server/lib/audit/discovery";

const robots = parseRobotsTxt(origin, robotsTxtText);
if (!robots.isAllowed(url)) {
  // Skip URLs disallowed by robots.txt
}

```

Source: [`src/server/lib/audit/discovery.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts) lines 40-68

## Phase 2: URL Seeding and Frontier Filtering

During `runDiscoveryPhase`, OpenSEO filters every candidate URL against the parsed `robots` object before adding it to the crawl frontier. This includes URLs discovered from XML sitemaps and the initial start URL.

The system normalizes each candidate and invokes `robots.isAllowed(normalized)` as a gatekeeping check. Any URL violating the [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) policy is excluded from the crawl queue before network requests are initiated, ensuring zero unauthorized fetches from seed sources.

Source: [`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts) lines 49-53

## Phase 3: Per-URL Validation During Crawling

After seeding, the crawler validates each dequeued link through `shouldQueueCrawlLink` in the crawl phase. This function performs three validation checks: same-origin verification, crawlability assessment, and crucially, validation against the previously parsed `robots` object.

If `robots.isAllowed` returns false for a given URL, the crawler abandons the link immediately, preventing any HTTP request that would violate the site's [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) directives.

Source: [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts) lines 70-77

## Phase 4: Meta Robots and X-Robots-Tag Processing

Once a page is fetched, OpenSEO extracts indexability directives from both HTML content and HTTP headers to ensure **meta robots tags compliance**.

### Extracting Directives

The `crawlPage` function utilizes `analyzeHtml` to parse the `<meta name="robots">` tag content from the document head. Simultaneously, it reads the **`X-Robots-Tag`** HTTP response header. These directives are merged into a unified `robotsDirectives` object.

### Determining Indexability

The system scans the merged directives for the `"noindex"` instruction. If present, the page receives an `isIndexable: false` flag. This boolean determines whether downstream issue reporters flag the URL as non-indexable content.

```typescript
// Crawl a page and respect meta-robots / X-Robots-Tag
import { crawlPage } from "@/server/workflows/site-audit-workflow-helpers";

const result = await crawlPage(url, depth, inSitemap);
if (!result.isIndexable) {
  // Page is flagged as non-indexable (noindex in meta or header)
}

```

Source: [`src/server/workflows/site-audit-workflow-helpers.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/site-audit-workflow-helpers.ts) lines 38-44

## Phase 5: Persisting Compliance Data

OpenSEO stores robots-related metadata in the audit database schema. The system records `robots_meta` and `x_robots_tag` values for each crawled page, enabling the audit UI to surface "Indexability" issues for pages marked as non-indexable.

This persistence layer ensures that compliance determinations—whether originating from [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) denials or meta-tag exclusions—are documented and available for longitudinal reporting.

Source: [`src/db/audit.schema.ts`](https://github.com/every-app/open-seo/blob/main/src/db/audit.schema.ts) lines 73-78

## Summary

- **Deterministic parsing**: OpenSEO uses the `robots-parser` library in `parseRobotsTxt` to generate consistent `isAllowed()` decisions that are checkpointed for replay accuracy.
- **Frontier filtering**: The `runDiscoveryPhase` function validates sitemap and seed URLs against parsed [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) rules before adding them to the crawl queue.
- **Link-level gates**: The `shouldQueueCrawlLink` validator enforces [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) compliance during the active crawl phase, preventing forbidden fetches.
- **Meta-tag extraction**: The `crawlPage` helper extracts and merges directives from `<meta name="robots">` tags and `X-Robots-Tag` headers to set the `isIndexable` flag.
- **Database persistence**: Schema columns `robots_meta` and `x_robots_tag` store compliance data for audit reporting and UI issue surfacing.

## Frequently Asked Questions

### How does OpenSEO handle oversized robots.txt files?

OpenSEO enforces a strict **500 KiB size limit** when fetching [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) via `fetchRobotsTxtText`. Files exceeding this threshold are truncated or rejected, preventing parser denial-of-service attacks and ensuring processing of only valid policy files.

### What happens if a URL is allowed by robots.txt but contains a noindex meta tag?

The crawler will fetch the page (since [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) permits it), but the `crawlPage` function will detect the `"noindex"` directive in the merged `robotsDirectives`. The page receives `isIndexable: false`, and the audit UI flags it as a non-indexable page without treating it as a crawl error.

### Does OpenSEO support the X-Robots-Tag HTTP header?

Yes. OpenSEO explicitly reads the **`X-Robots-Tag`** header alongside HTML meta robots tags. Both sources are merged in `robotsDirectives`, ensuring HTTP-header-based directives receive equal weight in indexability determinations.

### Where does OpenSEO validate URLs against robots.txt during the crawl?

Validation occurs at two critical points: during `runDiscoveryPhase` (filtering sitemap URLs before queueing) and in `shouldQueueCrawlLink` (checking individual links when dequeued from the crawl frontier). This dual-gate approach ensures compliance throughout the entire audit pipeline.