How the Porter Stemming Algorithm Powers Search in 30 Seconds of Code

The Porter stemming algorithm normalizes English words to their root stems during both indexing and search time, ensuring that queries for "running" match snippets containing "run" or "ran."

The 30 seconds of code repository relies on a custom client-side search engine to help developers find JavaScript snippets instantly. At the heart of this system lies a JavaScript implementation of the Porter stemming algorithm (specifically the Porter‑2 variant), which strips suffixes from words to collapse grammatical variations into a common base form.

How Stemming Works in the Search Pipeline

The search system applies stemming symmetrically—once when building the searchable index and again when processing user queries. This guarantees that both sides of the search use the same normalized vocabulary.

During indexing, the src/lib/search/documentIndex.js module processes every snippet by tokenizing its text, converting tokens to lowercase, and passing each through the stemmer. This creates a compact inverted index where "running" and "runs" both map to the slot for "run."

During query time, src/lib/search/utils.js handles user input by splitting the search string into tokens, applying the same stemming logic, and matching the resulting stems against the pre-computed index. Because both paths invoke the same stem() function from src/lib/search/porterStemmer.js, a query for "running" reliably finds snippets containing "run" or "ran."

Core Implementation Details

The Porter stemmer lives in src/lib/search/porterStemmer.js, which exports a single function implementing the classic Porter‑2 algorithm through a series of suffix-stripping rules:

export function stem(word) { … }

This module integrates into the broader search architecture through two primary consumers:

• src/lib/search/documentIndex.js – Builds the inverted index by calling stem() on every token extracted from snippet bodies during the build process.
• src/lib/search/utils.js – Prepares user queries by tokenizing input and applying the stemming logic before index lookup.

Supporting files include src/lib/search/settings.js (which manages stop-word lists that work alongside stemming) and src/astro/scripts/omnisearch.js (the frontend omnibox script that transmits stemmed queries to the search backend).

Because the index stores only stems rather than full word forms, the total index size remains small and token lookups operate effectively in O(1) time, keeping the omnibox responsive even across the entire snippet collection.

Practical Code Examples

To normalize a user search query before execution:

import { stem } from '@/lib/search/porterStemmer.js';

function normaliseQuery(query) {
  return query
    .toLowerCase()
    .split(/\s+/)          // simple tokenisation
    .map(stem)              // apply Porter stemming
    .join(' ');
}

// Example usage
const userInput = 'Running examples';
const normalised = normaliseQuery(userInput); // "run exampl"
searchEngine.search(normalised); // finds snippets containing "run" or "running"

Inside the index builder, snippets undergo identical processing:

import { stem } from '@/lib/search/porterStemmer.js';
import { tokenize } from '@/lib/search/utils.js';

function indexSnippet(snippet) {
  const tokens = tokenize(snippet.body);   // split text into words
  const stems  = tokens.map(stem);         // convert each token to its stem
  // …store `stems` in the inverted index
}

Summary

• The Porter stemming algorithm maps word variants like "run," "running," and "ran" to a single stem using suffix-stripping rules defined in the Porter‑2 specification.
• The implementation resides in src/lib/search/porterStemmer.js, exporting the stem(word) function used throughout the search system.
• Symmetric processing in src/lib/search/documentIndex.js (indexing) and src/lib/search/utils.js (queries) ensures consistent normalization and prevents missed matches.
• Stemming reduces the search index size significantly and enables fast O(1) token lookups, maintaining performance across the full snippet collection.

Frequently Asked Questions

What is the difference between Porter and Porter‑2 stemming?

Porter‑2 is an enhanced version of the original Porter algorithm that corrects certain exceptions and adds more suffix rules for better accuracy. The 30 seconds of code implementation follows the Porter‑2 specification to ensure reliable normalization of English words before indexing.

Why does the search index store stems instead of full words?

Storing stems collapses multiple grammatical variations into a single index entry, which reduces memory footprint and guarantees that searching for "running" retrieves results containing "run" or "ran" without requiring complex pattern matching or OR-logic in the query engine.

How does the stemmer handle code-specific terms or non-English words?

The Porter algorithm is designed for English morphology. When processing JavaScript identifiers or technical terms, it applies the same suffix-stripping rules, which may truncate certain strings (e.g., "const" remains "const," but "running" becomes "run"). Because the algorithm applies identically during both indexing and querying, the truncation remains consistent and does not affect match accuracy.

Where exactly is the stemmer invoked when I type in the search box?

The omnibox interface in src/astro/scripts/omnisearch.js captures your input and routes it through src/lib/search/utils.js, which tokenizes the string and calls stem() from src/lib/search/porterStemmer.js on each token before performing the lookup against the inverted index.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}