What Kind of Data Can Open-SEO Extract? A Complete Technical Breakdown

Open-SEO extracts both high-level site snapshots (URLs, titles, and plain text content) and granular SEO audit data (meta tags, headings, links, images, structured data, and hreflang tags) through its modular TypeScript functions.

The every-app/open-seo repository provides a TypeScript-based toolkit for programmatic website analysis. Understanding exactly what kind of data Open-SEO can extract helps developers build comprehensive audit pipelines and content processing workflows.

Site-Wide Content Discovery and Reading

Automated URL Discovery via Sitemaps

The discoverSiteUrls function in src/server/lib/scrape.ts performs automated site reconnaissance. It extracts the homepage URL and parses XML sitemaps to return same-origin HTML pages, while src/server/lib/audit/url-policy.ts applies SSRF guards to validate and sanitize URLs before any network request occurs.

Page Content Extraction

The readPages function fetches up to five pages (configurable) and processes each through scrapePage. This pipeline uses fetchText, extractTitle, and htmlToText to return a ScrapedPage object containing:

• Page URL
• Page title (from the HTML <title> tag)
• Plain-text content (cleaned HTML stripped of tags)

Comprehensive SEO Analysis with analyzeHtml

The analyzeHtml function in src/server/lib/audit/page-analyzer.ts uses cheerio to parse HTML and extract detailed SEO signals.

Meta Tags and Social Graph Data

The analyzer captures critical meta information:

• Title tag content
• Meta description
• Canonical URL
• Robots meta tag directives
• Open Graph tags (title, description, image)

Document Structure and Media Assets

Content structure is evaluated through:

• All headings (h1 through h6) and their hierarchical order
• Word count of visible body text
• Images with their src URLs and alt text attributes

Link Profile and Navigation Analysis

The system catalogs every anchor tag with:

• Destination URL and anchor text
• Internal/external classification flags
• No-follow attribute detection

Technical SEO Signals

Advanced extraction includes:

• Structured data presence (specifically application/ld+json script tags)
• Hreflang tags for internationalization signals

Practical Implementation Examples

The following TypeScript examples demonstrate how to leverage Open-SEO's extraction capabilities:

// 1️⃣ Discover URLs for a domain (homepage + sitemap)
import { discoverSiteUrls } from "./src/server/lib/scrape";

const { urls, blocked } = await discoverSiteUrls("https://example.com", 10);
if (blocked) console.log("Site unreachable or blocked");

// 2️⃣ Read up to 5 pages (title + plain text)
import { readPages } from "./src/server/lib/scrape";

const { pages, blocked: readBlocked } = await readPages(urls);
pages.forEach(p => {
  console.log(`🗒️ ${p.title ?? "No title"} → ${p.text.slice(0, 200)}…`);
});

// 3️⃣ Run a full SEO analysis on a page’s HTML
import { analyzeHtml } from "./src/server/lib/audit/page-analyzer";

const html = await fetch(pages[0].url).then(r => r.text());
const analysis = analyzeHtml(html, pages[0].url, 200, 120);
console.log("SEO title:", analysis.title);
console.log("Meta description:", analysis.metaDescription);
console.log("Headings:", analysis.h1s);
console.log("Word count:", analysis.wordCount);
console.log("Links (first 5):", analysis.links.slice(0, 5));

Key Source Files and Architecture

Understanding the codebase structure clarifies how data extraction operates:

• src/server/lib/scrape.ts: Handles site discovery, bounded fetching, and title/plain-text extraction.
• src/server/lib/audit/page-analyzer.ts: Contains the cheerio-based SEO audit logic.
• src/server/lib/audit/types.ts: Defines TypeScript interfaces for PageAnalysis results.
• src/server/lib/audit/url-utils.ts: Provides URL normalization helpers used throughout the analyzer.

Summary

• Open-SEO extracts two primary data layers: high-level content snapshots and detailed SEO audit signals.
• Content reading functions in scrape.ts provide URLs, titles, and cleaned plain text from multiple pages.
• SEO analysis via analyzeHtml captures meta tags, headings (h1-h6), images with alt text, links with internal/external flags, word counts, structured data, and hreflang tags.
• Discovery features automatically parse sitemaps and validate URLs against SSRF policies before fetching.

Frequently Asked Questions

Does Open-SEO extract JavaScript-rendered content?

No, the current implementation in src/server/lib/scrape.ts uses standard HTTP fetching and cheerio-based HTML parsing, which processes the initial server-rendered markup only. It does not execute JavaScript or wait for client-side hydration.

How many pages can Open-SEO process in a single run?

The readPages function defaults to processing five pages, though this limit is configurable. The discoverSiteUrls function accepts a maximum URL parameter to bound the discovery phase, preventing unbounded crawling.

What format does the structured data extraction return?

The analyzeHtml function detects the presence of application/ld+json script tags but currently reports their presence as part of the technical SEO audit. The specific parsing of JSON-LD content into structured objects depends on the implementation details in src/server/lib/audit/page-analyzer.ts.

Can Open-SEO differentiate between internal and external links?

Yes, the link extraction logic in src/server/lib/audit/page-analyzer.ts classifies each link with an internal/external flag based on the URL origin, and also detects no-follow attributes, providing complete link profile data for audit reports.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}