# How OpenSEO's Page Analyzer Extracts Information for Site Audits: A Deep Dive into the Core Engine

> Discover how OpenSEO's page analyzer extracts crucial site audit information using a streaming HTML tokenizer. Learn about title tags, meta descriptions, headings, and more.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: deep-dive
- Published: 2026-09-01

---

**OpenSEO's page analyzer extracts SEO information through a streaming HTML tokenizer in [`src/server/lib/audit/page-analyzer.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/page-analyzer.ts) that parses title tags, meta descriptions, headings, links, images, and robots directives while enforcing memory limits on content and crawlable URLs.**

The `analyzeHtml` function powers every site audit in the open-source OpenSEO project. When the crawler discovers a URL, this lightweight parser transforms raw HTML into structured `PageAnalysis` objects without loading the entire DOM into memory. This article examines the extraction pipeline, configuration options, and how the results feed into multi-page issue detection and Lighthouse performance scoring.

## The Site Audit Architecture

OpenSEO orchestrates page analysis through a coordinated workflow. The `AuditService` in [`src/server/features/audit/services/AuditService.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/AuditService.ts) manages the full lifecycle: URL discovery, HTML fetching, page-level extraction, and final report generation.

The workflow proceeds through these phases defined in [`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts):

1. **Discovery** — `discoverUrls` in [`src/server/lib/audit/discovery.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts) finds crawlable pages
2. **Fetch** — HTTP retrieval with respect to [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) (parsed via `parseRobotsTxt`)
3. **Analyze** — `analyzeHtml` extracts structured SEO data
4. **Persist** — `AuditRepository.insertPage` stores results
5. **Aggregate** — Multi-page issue detection and Lighthouse scoring

Progress tracking flows through [`src/server/lib/audit/progress-kv.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/progress-kv.ts), enabling real-time UI updates during large-scale audits.

## The Core Extraction Engine: `analyzeHtml`

The `analyzeHtml` function in [`src/server/lib/audit/page-analyzer.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/page-analyzer.ts) implements a custom streaming tokenizer rather than a full browser engine. This design choice prioritizes speed and memory efficiency for server-side audit workflows.

### Function Signature and Limits

```typescript
// src/server/lib/audit/page-analyzer.ts
export function analyzeHtml(
  html: string,
  pageUrl: string,
  maxChars?: number,
  maxLinks?: number
): PageAnalysis;

```

The optional `maxChars` and `maxLinks` parameters prevent runaway memory usage on malformed or enormous pages. The type definitions in [`src/server/lib/audit/types.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/types.ts) specify the returned `PageAnalysis` structure.

### Extracted Data Fields

The tokenizer captures these SEO-critical elements:

- **Title** — Content of the `<title>` element
- **Meta description** — `content` attribute of `<meta name="description">`
- **Open Graph tags** — All `og:*` property attributes
- **Twitter Card tags** — All `twitter:*` name attributes
- **Canonical URL** — `href` from `<link rel="canonical">`
- **Robots directives** — `content` from `<meta name="robots">`
- **Heading structure** — Text content of all `<h1>` through `<h6>` tags
- **Internal and external links** — `href` attributes filtered by `isCrawlableUrl`
- **Image assets** — `src` and `alt` attributes from `<img>` tags
- **Script and style indicators** — Presence flags for resource-heavy pages
- **Visible text content** — Stripped text for keyword analysis and density calculations

## Running Single-Page Analysis

For testing, CLI tools, or ad-hoc inspections, import `analyzeHtml` directly:

```typescript
import { analyzeHtml } from '@/server/lib/audit/page-analyzer';

const html = await fetch('https://example.com').then(r => r.text());
const pageUrl = 'https://example.com';

// Enforce 2000 character text limit and 100 link maximum
const analysis = analyzeHtml(html, pageUrl, 2000, 100);

console.log(analysis.title);           // "Example Domain"
console.log(analysis.headings.h1);     // ["Example Domain"]
console.log(analysis.links.length);    // Number of crawlable URLs found

```

The returned object conforms to the `PageAnalysis` interface, providing typed access to all extracted fields.

## Triggering Full Site Audits

Server-side functions initiate comprehensive audits through the workflow system:

```typescript
// Server function entry point
import { startAudit } from '@/serverFunctions/audit';

export async function POST(req: Request) {
  const { url, maxPages = 500 } = await req.json();
  
  const auditId = await startAudit({ 
    startUrl: url, 
    maxPages 
  });
  
  return Response.json({ auditId });
}

```

The `startAudit` function returns immediately with an audit identifier. Background workers in [`siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/siteAuditWorkflowPhases.ts) then process pages concurrently, calling `analyzeHtml` for each discovered URL.

## Consuming Audit Results

Client applications query completed audits through the API layer:

```typescript
import { useQuery } from '@tanstack/react-query';

function useAuditResults(auditId: string) {
  return useQuery({
    queryKey: ['auditResults', auditId],
    queryFn: async () => {
      const res = await fetch(`/api/audit/${auditId}/results`);
      return res.json(); // PageAnalysis[] aggregated with issue metadata
    }
  });
}

```

The response includes accumulated page data enriched with multi-page issue classifications from [`src/server/lib/audit/issues/multipage.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/issues/multipage.ts) and performance metrics from [`src/server/lib/audit/lighthouse.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/lighthouse.ts).

## Downstream Processing Pipeline

### Multi-Page Issue Detection

After individual page extraction, [`src/server/lib/audit/issues/multipage.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/issues/multipage.ts) runs detectors that require cross-page context:

- Duplicate title detection across the crawl scope
- Duplicate meta description identification
- Canonical chain validation
- Orphan page detection

### Lighthouse Integration

[`src/server/lib/audit/lighthouse.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/lighthouse.ts) invokes Chrome's Lighthouse engine for performance, accessibility, and best-practice scores. These metrics merge with the structural SEO data from `analyzeHtml` to produce comprehensive audit reports.

### Progress Tracking

The [`progress-kv.ts`](https://github.com/every-app/open-seo/blob/main/progress-kv.ts) module maintains crawl state in a key-value store, enabling:

- Real-time progress bars in the UI
- Resume capability for interrupted audits
- Per-user concurrent audit limits

## Validation and Testing

The test suite in [`src/server/lib/audit/page-analyzer.test.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/page-analyzer.test.ts) validates:

- Correct extraction of all defined fields
- Respect for `maxChars` and `maxLinks` limits
- Handling of malformed HTML without crashes
- Unicode and encoding edge cases

Tests run against fixture HTML files representing common site patterns: single-page applications, server-rendered frameworks, and legacy table-based layouts.

## Summary

- **OpenSEO's page analyzer** uses a streaming HTML tokenizer in [`page-analyzer.ts`](https://github.com/every-app/open-seo/blob/main/page-analyzer.ts) rather than a full browser, optimizing for server-side throughput
- The `analyzeHtml` function extracts **10+ SEO-critical data categories** with configurable memory limits
- Results flow through **multi-phase workflows** coordinating discovery, analysis, persistence, and aggregation
- **Multi-page issue detection** and **Lighthouse scoring** consume extracted data to generate actionable audit reports
- The modular architecture separates concerns across `AuditService`, workflow phases, and specialized analyzers

## Frequently Asked Questions

### What limits does OpenSEO's page analyzer enforce during extraction?

The `analyzeHtml` function accepts optional `maxChars` and `maxLinks` parameters that cap visible text content and crawlable URL discovery. These defaults prevent memory exhaustion on pages with excessive content or link spam. The tokenizer streams through HTML without building a full DOM tree, keeping memory footprint constant regardless of page size.

### How does OpenSEO handle robots.txt and meta robots directives?

The discovery phase parses [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) through `parseRobotsTxt` to filter crawlable URLs before fetching. After retrieval, `analyzeHtml` extracts `<meta name="robots">` content into the `PageAnalysis` object. Both sources inform whether pages should be indexed, followed, or excluded from the audit report.

### Can I use OpenSEO's page analyzer independently of the full audit workflow?

Yes. Import `analyzeHtml` directly from [`src/server/lib/audit/page-analyzer.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/page-analyzer.ts) for single-page analysis. This supports testing, CLI utilities, and custom integrations. The function requires only an HTML string and URL, with no dependencies on the database or workflow infrastructure.

### What performance data does OpenSEO collect beyond structural SEO analysis?

After page extraction completes, [`src/server/lib/audit/lighthouse.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/lighthouse.ts) runs Google Lighthouse audits for Core Web Vitals, accessibility scores, and best-practice compliance. These performance metrics merge with the structural data from `analyzeHtml` to provide unified reports covering both technical SEO and user experience factors.