How the github/docs Link Validation System Detects Broken Links

The link validation system in the github/docs repository employs two specialized checkers—an internal-link checker for cross-references and assets, and an external-link checker for HTTP(S) URLs—sharing a common pipeline that parses Markdown, renders Liquid templates, and validates links against a cached page map or live HTTP endpoints.

The github/docs repository maintains millions of cross-references across versioned documentation, requiring a robust system to catch broken links before they reach users. The link validation system combines static analysis of the content graph with live HTTP probes to verify both internal navigation paths and external resources. Understanding this architecture reveals how the Docs team maintains link integrity across thousands of Markdown files and multiple product versions.

Overview of the Link Validation Architecture

At its core, the validation pipeline splits concerns between two distinct checkers implemented in TypeScript. The internal-link checker, orchestrated by src/links/scripts/check-links-internal.ts, validates relative paths pointing to other Docs pages, assets, and anchors. The external-link checker, defined in src/links/scripts/check-links-external.ts, validates absolute HTTPS URLs against live endpoints.

Both tools rely on a shared extraction layer in src/links/lib/extract-links.ts that parses raw Markdown, executes Liquid conditionals ({% ifversion %}), and maps every link back to its exact source line number for precise error reporting.

How the Internal-Link Checker Validates Cross-References

The internal-link verification process reconciles the static content graph against the rendered output of versioned Markdown files.

Extracting and Rendering Links from Markdown

The pipeline begins with extractLinksFromMarkdown in src/links/lib/extract-links.ts, which scans raw Markdown with fenced code blocks stripped to build an O(log L) line-offset map. This function captures internal paths (]\(/…\)), Liquid-prefixed links (]({% … %})), image references, and anchor definitions.

To handle version-gated content, renderAndExtractLinks passes the Markdown through the Liquid engine, applying conditional logic so only links visible to the target version are validated. The getLinksFromMarkdown function then reconciles rendered links with their original line numbers using the rawLinesByHref map, ensuring error reports reference the correct source locations.

Normalizing Paths and Resolving Versions

Before validation, each href passes through normalizeLinkPath, which strips query strings, fragments, and trailing slashes while enforcing a leading /. The checkInternalLink function then resolves version placeholders like /enterprise-server@latest to concrete stable versions (e.g., /enterprise-server@<latestStable>) before looking up the target.

Validating Page Existence and Redirects

The checker queries three data structures loaded by warmServer from src/frame/lib/warm-server.ts:

• pageMap: A map of all known Docs pages for the target language/version
• redirects: A map of URL redirects generated from frontmatter definitions

For each normalized path, the system checks direct existence in pageMap, fallback matches in redirects, and language-prefixed variants (stripping /en prefixes when necessary). Links beginning with /assets/ bypass the page map and are verified directly against the filesystem via fs.existsSync.

Checking Asset and Anchor Links

Anchor validation uses github-slugger to regenerate heading IDs from rendered Markdown, comparing these against extracted #anchor fragments. If a referenced ID is missing from the computed set, the system records a broken anchor error with the precise line number from the source map.

How the External-Link Checker Validates HTTP(S) URLs

The external checker validates outbound links through a cached, throttled HTTP probe system that minimizes unnecessary network traffic.

URL Extraction and Deduplication

extractLinksFromMarkdown identifies external URLs using the EXTERNAL_LINK_PATTERN constant. The system deduplicates targets using normalizeUrl, which removes fragments and standardizes root URLs (treating https://example.com and https://example.com/ as identical).

Caching and Domain-Aware Concurrency

To avoid redundant requests, the system maintains a low-db JSON cache (external-link-cache.json) with a default CACHE_MAX_AGE_DAYS of 7 days. Fresh cache entries return immediately without network overhead.

For live checks, URLs are grouped by hostname and processed with domain-aware concurrency: up to DOMAIN_CONCURRENCY workers handle different hosts in parallel, but URLs within the same domain are checked sequentially to respect rate limits.

HTTP Validation and GitHub API Optimization

The checkUrl function first issues a HEAD request for efficiency; if the server returns a 4xx or 5xx status, a GET request follows as a fallback. All requests include a custom User-Agent header.

For GitHub repository links matching github.com/<owner>/<repo>, the system optimizes validation through the GitHub REST API via checkGithubRepoUrl. A GET request to https://api.github.com/repos/<owner>/<repo> confirms repository existence and public visibility; private repositories trigger a fallback to standard HTTP checks.

The system flags malformed URLs immediately and collects self-referential docs.github.com links separately—not as errors, but as candidates for conversion to internal links.

Code Examples

Extracting Internal Links from a Page

The following TypeScript demonstrates how the pipeline extracts links after Liquid rendering:

import { extractLinksFromMarkdown, renderAndExtractLinks } from '@/links/lib/extract-links'
import { getLinksFromMarkdown } from '@/src/links/scripts/check-links-internal'

// Assume `page` is a Page object from warmServer and `ctx` is a Liquid context
const rawResult = extractLinksFromMarkdown(page.markdown)

const { renderedMarkdown, result: renderedResult } = await renderAndExtractLinks(
  page.markdown,
  ctx,
)

// Get a list of `{href, text, line}` for the links that actually appear after Liquid rendering
const links = await getLinksFromMarkdown(page, ctx, rawResult, renderedResult)

for (const link of links) {
  console.log(`${link.href} (line ${link.line})`)
}

Source: src/links/scripts/check-links-internal.ts (lines 118‑202) and src/links/lib/extract-links.ts (lines 17‑96, 118‑140).

Validating a Single Internal Link

To programmatically verify a path against the Docs page map:

import { normalizeLinkPath, checkInternalLink } from '@/links/lib/extract-links'

// `pageMap` and `redirects` come from warmServer()
const href = '/en/actions/using-workflows'
const { exists, isRedirect, redirectTarget } = checkInternalLink(href, pageMap, redirects)

if (!exists) {
  console.error(`Broken internal link: ${href}`)
} else if (isRedirect) {
  console.warn(`Redirected link – target: ${redirectTarget}`)
}

Source: src/links/lib/extract-links.ts (lines 22‑79).

Checking an External URL with Cache

For validating outbound URLs with caching support:

import { checkUrl } from '@/src/links/scripts/check-links-external'

const url = 'https://example.com/api'
const cache = { urls: {} } // Normally loaded from lowdb JSON

const result = await checkUrl(url, cache as any)

if (!result.ok) {
  console.error(`Broken external link: ${url} – ${result.error}`)
}

Source: src/links/scripts/check-links-external.ts (checkUrl starts at line 17).

Using the GitHub Repository Shortcut

Optimized validation for GitHub repository links:

import { checkGithubRepoUrl } from '@/src/links/scripts/check-links-external'

const repoUrl = 'https://github.com/octocat/Hello-World'
const { ok, error } = await checkGithubRepoUrl(
  repoUrl,
  'octocat',
  'Hello-World',
  db.data, // lowdb cache object
)

if (!ok) console.error(`Repo link broken: ${error}`)

Source: src/links/scripts/check-links-external.ts (checkGithubRepoUrl starts at line 90).

Key Source Files in the Validation Pipeline

File	Role
src/links/lib/extract-links.ts	Core Markdown extraction, Liquid rendering, line-number mapping, path normalization, and checkInternalLink implementation.
src/links/scripts/check-links-internal.ts	Orchestrates per-page internal link validation, anchor checking with github-slugger, and report generation.
src/links/scripts/check-links-external.ts	Scans Markdown for external URLs, deduplicates targets, manages the lowdb cache, performs HTTP checks, and implements GitHub API optimization.
src/links/lib/link-report.ts	Formats broken-link data into Markdown tables, groups internal vs. external failures, and generates summary statistics.
src/frame/lib/warm-server.ts	Pre-loads all Docs pages to build the pageMap and redirects tables required for internal link resolution.

Summary

• The github/docs link validation system operates two distinct checkers: an internal-link checker for cross-references and assets, and an external-link checker for HTTP(S) endpoints.
• Internal validation parses Markdown, renders Liquid conditionals, normalizes paths, and verifies targets against a pre-built pageMap and redirects table using functions like checkInternalLink.
• External validation extracts URLs, deduplicates them via normalizeUrl, caches results for 7 days in external-link-cache.json, and probes endpoints using domain-aware concurrency with HEAD/GET fallback logic.
• Anchor validation regenerates heading IDs using the github-slugger algorithm to ensure fragment links point to valid sections.
• GitHub repository links receive optimized validation through the REST API (checkGithubRepoUrl), reducing false positives from private repositories.

Frequently Asked Questions

How does the system handle versioned content in links?

The validation pipeline renders Markdown through the Liquid engine before checking links, ensuring that version-gated conditionals ({% ifversion %}) are applied. The checkInternalLink function rewrites version placeholders like /enterprise-server@latest to concrete stable versions before querying the pageMap, so only links visible to the target version are validated.

What happens when an external URL returns a 404 status?

When checkUrl encounters a 4xx or 5xx response from an initial HEAD request, it automatically falls back to a GET request to confirm the failure. If both requests fail, the URL is recorded as broken in the lowdb cache to prevent repeated checks for 7 days (configurable via CACHE_MAX_AGE_DAYS). The result appears in the Markdown report generated by generateExternalLinkReport.

How does the system distinguish between internal and external links?

During extraction in extractLinksFromMarkdown, the system pattern-matches hrefs using EXTERNAL_LINK_PATTERN to identify absolute HTTPS URLs for the external checker. Relative paths starting with /, including those prefixed with Liquid tags, are routed to the internal checker. Self-referential docs.github.com URLs are flagged separately as candidates for conversion to internal relative paths.

Why does the internal checker use github-slugger for anchor validation?

The checker uses github-slugger to compute heading IDs because it matches the exact algorithm used by the live GitHub Docs site. After rendering a page's Markdown, the system regenerates all heading slugs and compares them against extracted #anchor fragments. This ensures that anchor links like #example-heading actually correspond to generated IDs in the final HTML output, catching broken deep-links that would otherwise pass basic URL checks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}