Content Extraction System in 30 Seconds of Code: How Markdown Becomes Structured Data

The content extraction system is a Node.js pipeline that transforms raw Markdown snippets, language definitions, and collection metadata into a structured JSON object for static site generation.

The 30-seconds-of-code repository powers a popular coding knowledge base through a sophisticated content extraction system. This system bridges the gap between author-friendly Markdown files and the structured data required by the Next.js frontend. By parsing, highlighting, and serializing content at build time, the pipeline ensures fast page loads and searchable snippet indexes.

How the Content Extraction System Works

Entry Point: The extractData Orchestrator

Located in src/lib/contentUtils/extractor.js, the extractData(highlighter) function serves as the central orchestrator. It coordinates four specialized model workers to build a complete content graph.

Model Workers: Domain-Specific Parsers

Each worker in src/lib/contentUtils/modelWorkers/ handles a specific content type:

Language Worker (language.js): Parses content/grammars.yaml to build a map of language metadata, including file extensions and highlighter IDs for Shiki and Prism.

Snippet Worker (snippet.js): Uses fast-glob to discover Markdown files in content/snippets/**/*.md. It reads each file, invokes the markdown parser to extract front-matter and code blocks, and generates syntax-highlighted HTML using the configured highlighter.

Collection Worker (collection.js): Loads content/collection-template.yaml to construct the hierarchical organization of snippet groups (e.g., "Array", "String").

Collection Snippet Worker (collectionSnippet.js): Associates individual snippets with their parent collections by matching front-matter tags to collection definitions.

Markdown Parsing and AST Transformation

The src/lib/contentUtils/markdownParser/markdownParser.js module converts raw Markdown into an abstract syntax tree (AST). It runs plugins such as highlightCode.js to locate fenced code blocks and replace them with highlighted HTML strings. This transformation occurs during the extraction phase, ensuring the frontend receives pre-rendered code blocks rather than raw Markdown.

Syntax Highlighting: Shiki and Prism

The extraction system supports two highlighters: Shiki (default, works offline, fast) and Prism (fallback). The highlighter parameter passed to extractData determines which engine highlights code blocks during the AST transformation phase.

Output Generation and Serialization

After extraction, the system writes the structured data to .content/content.json via the bin/prepare script. This JSON object contains four top-level keys: languages, snippets, collections, and collectionSnippets. Serializers in src/serializers/*.js then consume this data to generate static pages, search indexes, and API endpoints for the Next.js application.

Practical Examples

Running the Extraction Pipeline

import { extractData } from '#src/lib/contentUtils/extractor.js';

(async () => {
  // Use default Shiki highlighter
  const data = await extractData();
  
  // Returns: { languages, snippets, collections, collectionSnippets }
  console.log(Object.keys(data));
  
  // Write to content.json (as done by bin/prepare)
  await import('fs/promises')
    .then(fs => fs.writeFile('.content/content.json', JSON.stringify(data, null, 2)));
})();

Extracting Individual Snippets

import { readFile } from 'fs/promises';
import { parse } from '#src/lib/contentUtils/markdownParser/markdownParser.js';

async function extractOneSnippet(filePath) {
  const raw = await readFile(filePath, 'utf-8');
  const { metadata, highlightedCode } = await parse(raw);
  return { ...metadata, highlightedCode };
}

// Example usage
extractOneSnippet('content/snippets/js/s/flatten-array.md')
  .then(console.log);

Building the Language Map

import { extractLanguageData } from '#src/lib/contentUtils/modelWorkers/language.js';

(async () => {
  const languages = await extractLanguageData('content/grammars.yaml');
  console.log(languages['javascript']);
  // → { name: 'JavaScript', extensions: ['js', 'mjs'], shikiId: 'javascript', prismId: 'javascript' }
})();

Linking Snippets to Collections

import { extractCollectionData } from '#src/lib/contentUtils/modelWorkers/collection.js';
import { extractCollectionSnippetData } from '#src/lib/contentUtils/modelWorkers/collectionSnippet.js';
import { extractSnippetData } from '#src/lib/contentUtils/modelWorkers/snippet.js';

(async () => {
  const collections = await extractCollectionData('content/collection-template.yaml');
  const snippets = await extractSnippetData('content/snippets/**/*.md');
  const collectionSnippets = extractCollectionSnippetData(collections, snippets);
  console.log(collectionSnippets['Array']); // array of snippet IDs belonging to the "Array" collection
})();

Summary

• The content extraction system transforms raw Markdown into structured JSON via a pipeline orchestrated by extractData in src/lib/contentUtils/extractor.js.
• Model workers handle specific domains: languages (language.js), snippets (snippet.js), collections (collection.js), and snippet-to-collection mapping (collectionSnippet.js).
• The markdown parser converts Markdown to AST and applies syntax highlighting via plugins, supporting both Shiki and Prism engines.
• Output is persisted to .content/content.json and consumed by serializers to generate static pages and search indexes for the Next.js frontend.

Frequently Asked Questions

What is the content extraction system in 30 seconds of code?

The content extraction system is a Node.js build pipeline that converts author-written Markdown files into a structured JSON object containing highlighted code snippets, language metadata, and collection hierarchies. Located primarily in src/lib/contentUtils/, this system enables the static generation of the 30-seconds-of-code website by pre-processing all content at build time rather than runtime.

How does the extraction system handle syntax highlighting?

The system supports dual highlighters: Shiki (default) and Prism (fallback). During the extraction phase in src/lib/contentUtils/markdownParser/plugins/ast/highlightCode.js, fenced code blocks are transformed into highlighted HTML strings using the selected highlighter. This ensures the frontend receives pre-rendered code blocks, eliminating client-side highlighting overhead and improving page load performance.

Where is the extracted data stored and how is it used?

After extraction, the pipeline writes the structured data to .content/content.json via the bin/prepare script. This JSON file contains four top-level keys: languages, snippets, collections, and collectionSnippets. Serializers in src/serializers/*.js consume this data to generate static HTML pages, search indexes, and API endpoints for the Next.js application.

Can I run the content extraction system independently?

Yes, you can import and run the extraction pipeline independently by importing extractData from src/lib/contentUtils/extractor.js. The function accepts an optional highlighter argument ('shiki' or 'prism') and returns a Promise resolving to the complete data object. This is useful for debugging, testing new snippets locally, or building custom tooling around the 30-seconds-of-code content structure.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}