How OpenSEO's Page Analyzer Extracts Information for Site Audits: A Deep Dive into the Core Engine

OpenSEO's page analyzer extracts SEO information through a streaming HTML tokenizer in src/server/lib/audit/page-analyzer.ts that parses title tags, meta descriptions, headings, links, images, and robots directives while enforcing memory limits on content and crawlable URLs.

The analyzeHtml function powers every site audit in the open-source OpenSEO project. When the crawler discovers a URL, this lightweight parser transforms raw HTML into structured PageAnalysis objects without loading the entire DOM into memory. This article examines the extraction pipeline, configuration options, and how the results feed into multi-page issue detection and Lighthouse performance scoring.

The Site Audit Architecture

OpenSEO orchestrates page analysis through a coordinated workflow. The AuditService in src/server/features/audit/services/AuditService.ts manages the full lifecycle: URL discovery, HTML fetching, page-level extraction, and final report generation.

The workflow proceeds through these phases defined in src/server/workflows/siteAuditWorkflowPhases.ts:

1. Discovery — discoverUrls in src/server/lib/audit/discovery.ts finds crawlable pages
2. Fetch — HTTP retrieval with respect to robots.txt (parsed via parseRobotsTxt)
3. Analyze — analyzeHtml extracts structured SEO data
4. Persist — AuditRepository.insertPage stores results
5. Aggregate — Multi-page issue detection and Lighthouse scoring

Progress tracking flows through src/server/lib/audit/progress-kv.ts, enabling real-time UI updates during large-scale audits.

The Core Extraction Engine: analyzeHtml

The analyzeHtml function in src/server/lib/audit/page-analyzer.ts implements a custom streaming tokenizer rather than a full browser engine. This design choice prioritizes speed and memory efficiency for server-side audit workflows.

Function Signature and Limits

// src/server/lib/audit/page-analyzer.ts
export function analyzeHtml(
  html: string,
  pageUrl: string,
  maxChars?: number,
  maxLinks?: number
): PageAnalysis;

The optional maxChars and maxLinks parameters prevent runaway memory usage on malformed or enormous pages. The type definitions in src/server/lib/audit/types.ts specify the returned PageAnalysis structure.

Extracted Data Fields

The tokenizer captures these SEO-critical elements:

• Title — Content of the <title> element
• Meta description — content attribute of <meta name="description">
• Open Graph tags — All og:* property attributes
• Twitter Card tags — All twitter:* name attributes
• Canonical URL — href from <link rel="canonical">
• Robots directives — content from <meta name="robots">
• Heading structure — Text content of all <h1> through <h6> tags
• Internal and external links — href attributes filtered by isCrawlableUrl
• Image assets — src and alt attributes from <img> tags
• Script and style indicators — Presence flags for resource-heavy pages
• Visible text content — Stripped text for keyword analysis and density calculations

Running Single-Page Analysis

For testing, CLI tools, or ad-hoc inspections, import analyzeHtml directly:

import { analyzeHtml } from '@/server/lib/audit/page-analyzer';

const html = await fetch('https://example.com').then(r => r.text());
const pageUrl = 'https://example.com';

// Enforce 2000 character text limit and 100 link maximum
const analysis = analyzeHtml(html, pageUrl, 2000, 100);

console.log(analysis.title);           // "Example Domain"
console.log(analysis.headings.h1);     // ["Example Domain"]
console.log(analysis.links.length);    // Number of crawlable URLs found

The returned object conforms to the PageAnalysis interface, providing typed access to all extracted fields.

Triggering Full Site Audits

Server-side functions initiate comprehensive audits through the workflow system:

// Server function entry point
import { startAudit } from '@/serverFunctions/audit';

export async function POST(req: Request) {
  const { url, maxPages = 500 } = await req.json();
  
  const auditId = await startAudit({ 
    startUrl: url, 
    maxPages 
  });
  
  return Response.json({ auditId });
}

The startAudit function returns immediately with an audit identifier. Background workers in siteAuditWorkflowPhases.ts then process pages concurrently, calling analyzeHtml for each discovered URL.

Consuming Audit Results

Client applications query completed audits through the API layer:

import { useQuery } from '@tanstack/react-query';

function useAuditResults(auditId: string) {
  return useQuery({
    queryKey: ['auditResults', auditId],
    queryFn: async () => {
      const res = await fetch(`/api/audit/${auditId}/results`);
      return res.json(); // PageAnalysis[] aggregated with issue metadata
    }
  });
}

The response includes accumulated page data enriched with multi-page issue classifications from src/server/lib/audit/issues/multipage.ts and performance metrics from src/server/lib/audit/lighthouse.ts.

Downstream Processing Pipeline

Multi-Page Issue Detection

After individual page extraction, src/server/lib/audit/issues/multipage.ts runs detectors that require cross-page context:

• Duplicate title detection across the crawl scope
• Duplicate meta description identification
• Canonical chain validation
• Orphan page detection

Lighthouse Integration

src/server/lib/audit/lighthouse.ts invokes Chrome's Lighthouse engine for performance, accessibility, and best-practice scores. These metrics merge with the structural SEO data from analyzeHtml to produce comprehensive audit reports.

Progress Tracking

The progress-kv.ts module maintains crawl state in a key-value store, enabling:

• Real-time progress bars in the UI
• Resume capability for interrupted audits
• Per-user concurrent audit limits

Validation and Testing

The test suite in src/server/lib/audit/page-analyzer.test.ts validates:

• Correct extraction of all defined fields
• Respect for maxChars and maxLinks limits
• Handling of malformed HTML without crashes
• Unicode and encoding edge cases

Tests run against fixture HTML files representing common site patterns: single-page applications, server-rendered frameworks, and legacy table-based layouts.

Summary

• OpenSEO's page analyzer uses a streaming HTML tokenizer in page-analyzer.ts rather than a full browser, optimizing for server-side throughput
• The analyzeHtml function extracts 10+ SEO-critical data categories with configurable memory limits
• Results flow through multi-phase workflows coordinating discovery, analysis, persistence, and aggregation
• Multi-page issue detection and Lighthouse scoring consume extracted data to generate actionable audit reports
• The modular architecture separates concerns across AuditService, workflow phases, and specialized analyzers

Frequently Asked Questions

What limits does OpenSEO's page analyzer enforce during extraction?

The analyzeHtml function accepts optional maxChars and maxLinks parameters that cap visible text content and crawlable URL discovery. These defaults prevent memory exhaustion on pages with excessive content or link spam. The tokenizer streams through HTML without building a full DOM tree, keeping memory footprint constant regardless of page size.

How does OpenSEO handle robots.txt and meta robots directives?

The discovery phase parses robots.txt through parseRobotsTxt to filter crawlable URLs before fetching. After retrieval, analyzeHtml extracts <meta name="robots"> content into the PageAnalysis object. Both sources inform whether pages should be indexed, followed, or excluded from the audit report.

Can I use OpenSEO's page analyzer independently of the full audit workflow?

Yes. Import analyzeHtml directly from src/server/lib/audit/page-analyzer.ts for single-page analysis. This supports testing, CLI utilities, and custom integrations. The function requires only an HTML string and URL, with no dependencies on the database or workflow infrastructure.

What performance data does OpenSEO collect beyond structural SEO analysis?

After page extraction completes, src/server/lib/audit/lighthouse.ts runs Google Lighthouse audits for Core Web Vitals, accessibility scores, and best-practice compliance. These performance metrics merge with the structural data from analyzeHtml to provide unified reports covering both technical SEO and user experience factors.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}