# How the github/docs Link Validation System Detects Broken Links

> Discover how the github/docs link validation system finds broken links. Learn about its internal and external checkers, parsing, rendering, and validation processes.

- Repository: [GitHub/docs](https://github.com/github/docs)
- Tags: internals
- Published: 2026-06-01

---

**The link validation system in the github/docs repository employs two specialized checkers—an internal-link checker for cross-references and assets, and an external-link checker for HTTP(S) URLs—sharing a common pipeline that parses Markdown, renders Liquid templates, and validates links against a cached page map or live HTTP endpoints.**

The github/docs repository maintains millions of cross-references across versioned documentation, requiring a robust system to catch broken links before they reach users. The link validation system combines static analysis of the content graph with live HTTP probes to verify both internal navigation paths and external resources. Understanding this architecture reveals how the Docs team maintains link integrity across thousands of Markdown files and multiple product versions.

## Overview of the Link Validation Architecture

At its core, the validation pipeline splits concerns between two distinct checkers implemented in TypeScript. The **internal-link checker**, orchestrated by [`src/links/scripts/check-links-internal.ts`](https://github.com/github/docs/blob/main/src/links/scripts/check-links-internal.ts), validates relative paths pointing to other Docs pages, assets, and anchors. The **external-link checker**, defined in [`src/links/scripts/check-links-external.ts`](https://github.com/github/docs/blob/main/src/links/scripts/check-links-external.ts), validates absolute HTTPS URLs against live endpoints.

Both tools rely on a shared extraction layer in [`src/links/lib/extract-links.ts`](https://github.com/github/docs/blob/main/src/links/lib/extract-links.ts) that parses raw Markdown, executes Liquid conditionals (`{% ifversion %}`), and maps every link back to its exact source line number for precise error reporting.

## How the Internal-Link Checker Validates Cross-References

The internal-link verification process reconciles the static content graph against the rendered output of versioned Markdown files.

### Extracting and Rendering Links from Markdown

The pipeline begins with `extractLinksFromMarkdown` in [`src/links/lib/extract-links.ts`](https://github.com/github/docs/blob/main/src/links/lib/extract-links.ts), which scans raw Markdown with fenced code blocks stripped to build an O(log L) line-offset map. This function captures internal paths (`]\(/…\)`), Liquid-prefixed links (`]({% … %})`), image references, and anchor definitions.

To handle version-gated content, `renderAndExtractLinks` passes the Markdown through the Liquid engine, applying conditional logic so only links visible to the target version are validated. The `getLinksFromMarkdown` function then reconciles rendered links with their original line numbers using the `rawLinesByHref` map, ensuring error reports reference the correct source locations.

### Normalizing Paths and Resolving Versions

Before validation, each href passes through `normalizeLinkPath`, which strips query strings, fragments, and trailing slashes while enforcing a leading `/`. The `checkInternalLink` function then resolves version placeholders like `/enterprise-server@latest` to concrete stable versions (e.g., `/enterprise-server@<latestStable>`) before looking up the target.

### Validating Page Existence and Redirects

The checker queries three data structures loaded by `warmServer` from [`src/frame/lib/warm-server.ts`](https://github.com/github/docs/blob/main/src/frame/lib/warm-server.ts):

- **pageMap**: A map of all known Docs pages for the target language/version
- **redirects**: A map of URL redirects generated from frontmatter definitions

For each normalized path, the system checks direct existence in `pageMap`, fallback matches in `redirects`, and language-prefixed variants (stripping `/en` prefixes when necessary). Links beginning with `/assets/` bypass the page map and are verified directly against the filesystem via `fs.existsSync`.

### Checking Asset and Anchor Links

Anchor validation uses **github-slugger** to regenerate heading IDs from rendered Markdown, comparing these against extracted `#anchor` fragments. If a referenced ID is missing from the computed set, the system records a broken anchor error with the precise line number from the source map.

## How the External-Link Checker Validates HTTP(S) URLs

The external checker validates outbound links through a cached, throttled HTTP probe system that minimizes unnecessary network traffic.

### URL Extraction and Deduplication

`extractLinksFromMarkdown` identifies external URLs using the `EXTERNAL_LINK_PATTERN` constant. The system deduplicates targets using `normalizeUrl`, which removes fragments and standardizes root URLs (treating `https://example.com` and `https://example.com/` as identical).

### Caching and Domain-Aware Concurrency

To avoid redundant requests, the system maintains a **low-db JSON cache** ([`external-link-cache.json`](https://github.com/github/docs/blob/main/external-link-cache.json)) with a default `CACHE_MAX_AGE_DAYS` of 7 days. Fresh cache entries return immediately without network overhead.

For live checks, URLs are grouped by hostname and processed with domain-aware concurrency: up to `DOMAIN_CONCURRENCY` workers handle different hosts in parallel, but URLs within the same domain are checked sequentially to respect rate limits.

### HTTP Validation and GitHub API Optimization

The `checkUrl` function first issues a **HEAD** request for efficiency; if the server returns a 4xx or 5xx status, a **GET** request follows as a fallback. All requests include a custom `User-Agent` header.

For GitHub repository links matching `github.com/<owner>/<repo>`, the system optimizes validation through the GitHub REST API via `checkGithubRepoUrl`. A `GET` request to `https://api.github.com/repos/<owner>/<repo>` confirms repository existence and public visibility; private repositories trigger a fallback to standard HTTP checks.

The system flags malformed URLs immediately and collects self-referential `docs.github.com` links separately—not as errors, but as candidates for conversion to internal links.

## Code Examples

### Extracting Internal Links from a Page

The following TypeScript demonstrates how the pipeline extracts links after Liquid rendering:

```typescript
import { extractLinksFromMarkdown, renderAndExtractLinks } from '@/links/lib/extract-links'
import { getLinksFromMarkdown } from '@/src/links/scripts/check-links-internal'

// Assume `page` is a Page object from warmServer and `ctx` is a Liquid context
const rawResult = extractLinksFromMarkdown(page.markdown)

const { renderedMarkdown, result: renderedResult } = await renderAndExtractLinks(
  page.markdown,
  ctx,
)

// Get a list of `{href, text, line}` for the links that actually appear after Liquid rendering
const links = await getLinksFromMarkdown(page, ctx, rawResult, renderedResult)

for (const link of links) {
  console.log(`${link.href} (line ${link.line})`)
}

```

*Source*: [`src/links/scripts/check-links-internal.ts`](https://github.com/github/docs/blob/main/src/links/scripts/check-links-internal.ts) (lines 118‑202) and [`src/links/lib/extract-links.ts`](https://github.com/github/docs/blob/main/src/links/lib/extract-links.ts) (lines 17‑96, 118‑140).

### Validating a Single Internal Link

To programmatically verify a path against the Docs page map:

```typescript
import { normalizeLinkPath, checkInternalLink } from '@/links/lib/extract-links'

// `pageMap` and `redirects` come from warmServer()
const href = '/en/actions/using-workflows'
const { exists, isRedirect, redirectTarget } = checkInternalLink(href, pageMap, redirects)

if (!exists) {
  console.error(`Broken internal link: ${href}`)
} else if (isRedirect) {
  console.warn(`Redirected link – target: ${redirectTarget}`)
}

```

*Source*: [`src/links/lib/extract-links.ts`](https://github.com/github/docs/blob/main/src/links/lib/extract-links.ts) (lines 22‑79).

### Checking an External URL with Cache

For validating outbound URLs with caching support:

```typescript
import { checkUrl } from '@/src/links/scripts/check-links-external'

const url = 'https://example.com/api'
const cache = { urls: {} } // Normally loaded from lowdb JSON

const result = await checkUrl(url, cache as any)

if (!result.ok) {
  console.error(`Broken external link: ${url} – ${result.error}`)
}

```

*Source*: [`src/links/scripts/check-links-external.ts`](https://github.com/github/docs/blob/main/src/links/scripts/check-links-external.ts) (`checkUrl` starts at line 17).

### Using the GitHub Repository Shortcut

Optimized validation for GitHub repository links:

```typescript
import { checkGithubRepoUrl } from '@/src/links/scripts/check-links-external'

const repoUrl = 'https://github.com/octocat/Hello-World'
const { ok, error } = await checkGithubRepoUrl(
  repoUrl,
  'octocat',
  'Hello-World',
  db.data, // lowdb cache object
)

if (!ok) console.error(`Repo link broken: ${error}`)

```

*Source*: [`src/links/scripts/check-links-external.ts`](https://github.com/github/docs/blob/main/src/links/scripts/check-links-external.ts) (`checkGithubRepoUrl` starts at line 90).

## Key Source Files in the Validation Pipeline

| File | Role |
| --- | --- |
| [`src/links/lib/extract-links.ts`](https://github.com/github/docs/blob/main/src/links/lib/extract-links.ts) | Core Markdown extraction, Liquid rendering, line-number mapping, path normalization, and `checkInternalLink` implementation. |
| [`src/links/scripts/check-links-internal.ts`](https://github.com/github/docs/blob/main/src/links/scripts/check-links-internal.ts) | Orchestrates per-page internal link validation, anchor checking with github-slugger, and report generation. |
| [`src/links/scripts/check-links-external.ts`](https://github.com/github/docs/blob/main/src/links/scripts/check-links-external.ts) | Scans Markdown for external URLs, deduplicates targets, manages the lowdb cache, performs HTTP checks, and implements GitHub API optimization. |
| [`src/links/lib/link-report.ts`](https://github.com/github/docs/blob/main/src/links/lib/link-report.ts) | Formats broken-link data into Markdown tables, groups internal vs. external failures, and generates summary statistics. |
| [`src/frame/lib/warm-server.ts`](https://github.com/github/docs/blob/main/src/frame/lib/warm-server.ts) | Pre-loads all Docs pages to build the `pageMap` and redirects tables required for internal link resolution. |

## Summary

- The github/docs link validation system operates two distinct checkers: an **internal-link checker** for cross-references and assets, and an **external-link checker** for HTTP(S) endpoints.
- **Internal validation** parses Markdown, renders Liquid conditionals, normalizes paths, and verifies targets against a pre-built `pageMap` and redirects table using functions like `checkInternalLink`.
- **External validation** extracts URLs, deduplicates them via `normalizeUrl`, caches results for 7 days in [`external-link-cache.json`](https://github.com/github/docs/blob/main/external-link-cache.json), and probes endpoints using domain-aware concurrency with HEAD/GET fallback logic.
- **Anchor validation** regenerates heading IDs using the **github-slugger** algorithm to ensure fragment links point to valid sections.
- **GitHub repository links** receive optimized validation through the REST API (`checkGithubRepoUrl`), reducing false positives from private repositories.

## Frequently Asked Questions

### How does the system handle versioned content in links?

The validation pipeline renders Markdown through the Liquid engine before checking links, ensuring that version-gated conditionals (`{% ifversion %}`) are applied. The `checkInternalLink` function rewrites version placeholders like `/enterprise-server@latest` to concrete stable versions before querying the `pageMap`, so only links visible to the target version are validated.

### What happens when an external URL returns a 404 status?

When `checkUrl` encounters a 4xx or 5xx response from an initial HEAD request, it automatically falls back to a GET request to confirm the failure. If both requests fail, the URL is recorded as broken in the lowdb cache to prevent repeated checks for 7 days (configurable via `CACHE_MAX_AGE_DAYS`). The result appears in the Markdown report generated by `generateExternalLinkReport`.

### How does the system distinguish between internal and external links?

During extraction in `extractLinksFromMarkdown`, the system pattern-matches hrefs using `EXTERNAL_LINK_PATTERN` to identify absolute HTTPS URLs for the external checker. Relative paths starting with `/`, including those prefixed with Liquid tags, are routed to the internal checker. Self-referential `docs.github.com` URLs are flagged separately as candidates for conversion to internal relative paths.

### Why does the internal checker use github-slugger for anchor validation?

The checker uses **github-slugger** to compute heading IDs because it matches the exact algorithm used by the live GitHub Docs site. After rendering a page's Markdown, the system regenerates all heading slugs and compares them against extracted `#anchor` fragments. This ensures that anchor links like `#example-heading` actually correspond to generated IDs in the final HTML output, catching broken deep-links that would otherwise pass basic URL checks.