What Is the User Agent for the OpenSEO Scraper? A Deep Dive into the Audit Engine

The OpenSEO scraper identifies itself with the static User-Agent string OpenSEO-Audit/1.0 whenever it performs HTTP requests.

Every request made by the audit engine in the every-app/open-seo repository carries this specific identifier, allowing target servers to recognize legitimate crawler traffic from the OpenSEO platform. Understanding this user agent is essential for developers debugging crawl behavior, configuring firewall allowlists, or building integrations that need to mimic the scraper's requests.

The OpenSEO Scraper User-Agent String

The user agent for the OpenSEO scraper is hard-coded as a constant value across the entire audit workflow. The exact string is:

OpenSEO-Audit/1.0

This identifier appears in all outbound HTTP requests, including page fetches during site audits and HEAD requests for redirect resolution. Unlike configurable crawler settings found in some SEO tools, this value is static and versioned, making it predictable for server administrators and developers monitoring their logs.

Where the User Agent Is Defined in the Source Code

The codebase centralizes the user agent definition in specific workflow and policy modules, ensuring consistency across different crawling operations.

Audit Workflow Helpers (site-audit-workflow-helpers.ts)

In src/server/workflows/site-audit-workflow-helpers.ts, the user agent is defined as the constant CRAWL_USER_AGENT at lines 8–9. This constant is imported and used whenever the crawlPage function executes HTTP requests during site audits.

According to the source code, the constant is implemented as:

const CRAWL_USER_AGENT = "OpenSEO-Audit/1.0";

This value is then injected into the headers of every fetch request made by the crawling infrastructure, ensuring that pages being audited receive the proper identification headers.

URL Policy and Redirect Resolution (url-policy.ts)

The same user agent string appears in src/server/lib/audit/url-policy.ts at lines 66–67, where it handles start-URL redirect resolution. When the audit engine validates and follows redirects for the initial URL, it sends a HEAD request with the identical user agent header to maintain consistency across the audit chain.

This dual implementation ensures that both the primary crawling logic and the preliminary redirect detection present the same crawler identity to web servers.

How the User Agent Is Applied in HTTP Requests

The OpenSEO scraper applies this user agent in two primary scenarios: full page crawling and redirect resolution.

Crawling Pages in the Audit Workflow

When the crawlPage utility executes, it automatically includes the user agent in the request headers. Consider this example from the audit workflow:

import { crawlPage } from "@/server/workflows/site-audit-workflow-helpers";

await crawlPage("https://example.com", null, false);

Internally, the fetch call constructs headers that include:

headers: {
  "User-Agent": "OpenSEO-Audit/1.0",
  // additional headers...
}

Resolving Redirects Before Audit Start

Before the main crawl begins, the resolveStartUrlRedirects function in the URL policy module performs redirect detection:

import { resolveStartUrlRedirects } from "@/server/lib/audit/url-policy";

const finalUrl = await resolveStartUrlRedirects("https://example.com");

This function sends a HEAD request using the same user agent string, ensuring that any redirect chains encountered during the initial URL validation phase are processed with consistent identification.

Implementing the OpenSEO User Agent in Custom Scripts

If you need to replicate the OpenSEO scraper's behavior for testing or integration purposes, include the exact header in your fetch implementations:

await fetch(url, {
  method: "GET",
  headers: { "User-Agent": "OpenSEO-Audit/1.0" },
});

This approach is particularly useful when testing how your server responds to the OpenSEO crawler before deploying to production, or when building middleware that needs to specifically identify and handle OpenSEO audit traffic differently from general web traffic.

Summary

• The OpenSEO scraper uses the static user agent string OpenSEO-Audit/1.0 for all HTTP requests.
• The constant is defined as CRAWL_USER_AGENT in src/server/workflows/site-audit-workflow-helpers.ts (lines 8–9).
• The same string is hard-coded in src/server/lib/audit/url-policy.ts (lines 66–67) for redirect resolution.
• Both page crawling and HEAD requests for redirect detection use this identical identifier.
• The value is currently non-configurable and hard-coded as version 1.0 in the every-app/open-seo repository.

Frequently Asked Questions

What is the exact user agent string for the OpenSEO scraper?

The exact user agent string is OpenSEO-Audit/1.0. This static identifier appears in all HTTP requests made by the audit engine, including both GET requests for page content and HEAD requests for redirect resolution.

Where is the OpenSEO user agent defined in the codebase?

The primary definition occurs in src/server/workflows/site-audit-workflow-helpers.ts as the constant CRAWL_USER_AGENT. A duplicate hard-coded value also exists in src/server/lib/audit/url-policy.ts for the redirect resolution logic, ensuring consistency across different audit phases.

Can I change the user agent string in OpenSEO?

No. According to the source code analysis of the every-app/open-seo repository, the user agent is defined as a constant (CRAWL_USER_AGENT) and hard-coded in multiple locations. There are no configuration options or environment variables exposed to modify this value in the current implementation.

How can I identify OpenSEO crawler traffic in my server logs?

Look for requests containing the header User-Agent: OpenSEO-Audit/1.0. Since this string is static and version-specific, you can create precise firewall rules or analytics filters to distinguish OpenSEO audit traffic from other crawlers like Googlebot or generic HTTP clients.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}