How the Search Functionality in 30 Seconds of Code Works: Build-Time Indexing and Client-Side Ranking

The search functionality in 30 Seconds of Code implements a static-site search architecture that generates a JSON index at build time and performs TF-IDF and fuzzy n-gram matching entirely in the browser.

The 30 Seconds of Code website delivers sub-second search results across hundreds of JavaScript snippets without requiring a backend database or search server. This article examines the complete implementation in the Chalarangelo/30-seconds-of-code repository, tracing the pipeline from the static index generation in src/lib/searchIndex.js to the relevance scoring algorithms in src/lib/search/documentSearch.js.

Build-Time Index Generation

The search pipeline begins during the static site generation process. The SearchIndex.generate() method in src/lib/searchIndex.js creates a serialized JSON file at public/search-data.json that contains all searchable content.

This method queries the snippet database using Snippet.scope('published', 'listed', 'byId') and Collection.scope('listed', 'byId'), then serializes each item using SearchResultSerializer. The serializer produces a compact searchTokens string that represents the term frequency map for each document.

// src/lib/searchIndex.js
export default class SearchIndex {
  static generate() {
    const snippets = Snippet.scope('published', 'listed', 'byId');
    const collections = Collection.scope('listed', 'byId');

    const searchIndex = {
      searchIndex: SearchResultSerializer.serializeArray(
        snippets.concat(collections)
      ),
    };

    fs.writeJson('public/search-data.json', searchIndex, { spaces: 0 }, () => {});
  }
}

Each entry in the resulting JSON array contains the document URL, title, tags, and a serialized token string such as "debounce:3 function:2 delay:1", where numbers indicate term frequency.

Client-Side Index Loading and Initialization

When a user opens the search modal, src/astro/scripts/omnisearch.js fetches the static JSON and deserializes it into an in-memory DocumentIndex instance. The browser loads the entire search index in a single HTTP request, enabling offline-capable search functionality.

// src/astro/scripts/omnisearch.js
fetch('/search-data.json')
  .then(data => data.json())
  .then(json => {
    const documents = json.searchIndex.map(
      ({ url, searchTokens, ...data }) => ({
        id: url,
        content: deserializeTokens(searchTokens),
        rawContent: searchTokens,
        url,
        ...data,
      })
    );
    this.searchIndex = new DocumentIndex(documents);
    this.searchDocuments = search(this.searchIndex);
    this.searchIndexInitialized = true;
  });

The deserializeTokens function in src/lib/search/utils.js converts the compact string representation back into a Map<string, number> for fast term lookups.

// src/lib/search/utils.js
export const deserializeTokens = str =>
  str.split(' ').reduce((acc, words) => {
    const [word, count = 1] = words.split(':');
    acc.set(word, Number.parseInt(count));
    return acc;
  }, new Map());

The DocumentIndex Data Structure

The DocumentIndex class in src/lib/search/documentIndex.js maintains two inverted indexes to support both exact and fuzzy matching. The addDocument() method populates these structures when the index is initialized.

Inverted Term Index: Maps stemmed terms to document IDs with frequency counts for TF-IDF calculation.

N-gram Index: Maps 3-character n-grams to document ID sets for fuzzy matching when exact terms fail.

// src/lib/search/documentIndex.js
addDocument(docId, terms, data) {
  const ngrams = new Set(
    [...terms.keys()].reduce((ngrams, term) => {
      ngrams.push(...generateNgrams(term));
      return ngrams;
    }, [])
  );

  this.documents.set(docId, {
    terms,
    ngrams,
    length: [...terms.values()].reduce((a, b) => a + b),
    ...data,
  });

  // Term inverted index for TF-IDF
  terms.forEach((freq, term) => {
    if (!this.invertedIndex.has(term))
      this.invertedIndex.set(term, new Map());
    this.invertedIndex.get(term).set(docId, freq);
  });

  // N-gram inverted index for fuzzy matching
  ngrams.forEach(ngram => {
    if (!this.ngramsInvertedIndex.has(ngram))
      this.ngramsInvertedIndex.set(ngram, new Set());
    this.ngramsInvertedIndex.get(ngram).add(docId);
  });
}

The length property stores the total term count per document, which serves as the denominator for term frequency calculations.

Tokenization and Text Processing

The search engine processes queries using two distinct tokenization strategies defined in src/lib/search/documentSearch.js. Both paths use utilities from src/lib/search/utils.js and the Porter stemmer implementation in src/lib/search/porterStemmer.js.

Plain-Text Tokenization: Applies lowercasing, punctuation stripping, and Porter stemming for exact TF-IDF matching.

Token-Only Tokenization: Applies splitting and cleaning but preserves original word forms for n-gram fuzzy matching.

// src/lib/search/documentSearch.js
const tokenizePlainText = str =>
  splitTokens(str).map(tkn => cleanTokenPunctuation(stem(tkn)));

const tokenizeTokens = str => splitTokens(str).map(cleanTokenPunctuation);

The splitTokens function lowercases input and splits on non-alphanumeric characters, while cleanTokenPunctuation removes surrounding quotes and hyphens that could interfere with matching.

Ranking Algorithm: TF-IDF and Fuzzy Matching

The core search function in src/lib/search/documentSearch.js implements a hybrid scoring system that combines exact TF-IDF relevance with fuzzy n-gram similarity. The algorithm executes in three phases:

1. Exact TF-IDF Scoring: Calculates relevance using tf = termFrequency / documentLength and idf = ln(totalDocs / docsContainingTerm).
2. Partial Match Boosting: Applies additional scoring for prefix matches when the user is actively typing.
3. Fuzzy Blend: Combines TF-IDF scores with n-gram similarity using a configurable weight factor (default fuzzy = 0.7).

// src/lib/search/documentSearch.js
if (fuzzy > 0.0) {
  const ngramSimilarities = new Map(
    searchForNgrams(documentIndex, tokenizeTokens(query))
  );

  results = results.length
    ? results.map(([docId, score]) => {
        const ngramSimilarity = ngramSimilarities.get(docId) || 0;
        const finalScore = score * (1 - fuzzy) + ngramSimilarity * fuzzy;
        return [docId, finalScore];
      })
    : Array.from(ngramSimilarities.entries());
}

The final score formula blends exact and fuzzy matches: finalScore = tfidfScore * (1 - fuzzy) + ngramSimilarity * fuzzy. This ensures that exact term matches rank highest while typographical errors still return relevant results.

UI Integration and Real-Time Search

The omnisearch object in src/astro/scripts/omnisearch.js wires the search engine to the user interface. It manages index loading, query execution on keystroke events, and result rendering.

// src/astro/scripts/omnisearch.js
search(query) {
  if (!this.searchIndexInitialized || !this.isOpen) return;
  this.playSearchIconAnimation();
  const results = this.searchByKeyphrase(query);
  if (results.length > 0) this.displayResults(results);
  else if (query.length <= 1) this.displayEmptyState();
  else this.displayNotFoundState(query);
  this.focusedResult = -1;
}

The searchByKeyphrase() method delegates to the pre-bound searchDocuments function, which executes the TF-IDF and fuzzy ranking pipeline. Results return instantly because all data resides in memory, eliminating network latency during query execution.

Working with the Search API Programmatically

You can access the underlying search functionality directly in the browser console or in custom scripts once the page has initialized the omnisearch modal.

// Access the loaded index and search function
const idx = omnisearch.searchIndex;
const searchFn = omnisearch.searchDocuments;

// Search for 'debounce' and return top 10 results
const results = searchFn('debounce', 10);
// Returns: [{ id: '/js/functions/debounce', title: 'debounce', score: 0.85, ... }]

To simulate the complete search flow independently:

import DocumentIndex from '#src/lib/search/documentIndex.js';
import search from '#src/lib/search/documentSearch.js';
import { deserializeTokens } from '#src/lib/search/utils.js';

// Load the generated index
const { searchIndex } = await fetch('/search-data.json').then(r => r.json());

// Initialize the DocumentIndex
const docs = searchIndex.map(({ url, searchTokens, ...rest }) => ({
  id: url,
  content: deserializeTokens(searchTokens),
  url,
  ...rest,
}));
const idx = new DocumentIndex(docs);
const searchFn = search(idx);

// Execute query
const matches = searchFn('array flatten', 5);

Summary

• Static Generation: The build process creates public/search-data.json using SearchIndex.generate() in src/lib/searchIndex.js, serializing term frequencies for all snippets and collections.
• In-Memory Index: DocumentIndex in src/lib/search/documentIndex.js maintains dual inverted indexes for terms (TF-IDF) and n-grams (fuzzy matching).
• Hybrid Ranking: The search algorithm in src/lib/search/documentSearch.js combines exact TF-IDF scoring with n-gram similarity using a configurable blend factor.
• Client-Side Execution: All search operations occur in the browser after loading the JSON index, enabling instant results without server round-trips.
• Stemming and Tokenization: Queries undergo Porter stemming for exact matches and raw tokenization for fuzzy n-gram comparison.

Frequently Asked Questions

How does 30 Seconds of Code search work without a backend server?

The search functionality operates entirely client-side by loading a pre-generated JSON index file (search-data.json) into browser memory. When the user opens the search modal, src/astro/scripts/omnisearch.js fetches this file, deserializes the token data using deserializeTokens(), and builds a DocumentIndex instance. All query processing, TF-IDF calculations, and fuzzy matching execute in JavaScript within the browser, eliminating the need for database queries or search API calls.

What algorithm does the search use to rank results?

The ranking algorithm implements a hybrid TF-IDF and n-gram similarity approach. For exact matches, it calculates term frequency-inverse document frequency using the formula tf = termFrequency / documentLength and idf = ln(totalDocs / docsContainingTerm). For fuzzy matching, it compares 3-character n-grams between the query and indexed documents. The final score blends these values: finalScore = tfidfScore * 0.3 + ngramSimilarity * 0.7 (using the default fuzzy factor of 0.7).

Where is the search index data stored in the repository?

The static search index is generated at build time and written to public/search-data.json. The generation logic resides in src/lib/searchIndex.js, which uses SearchResultSerializer to convert database records into searchable token strings. Each entry contains the document URL, title, tags, and a serialized frequency map like "function:5 array:3".

How does the search handle typos and partial word matches?

The search engine uses n-gram fuzzy matching to handle typographical errors. When exact TF-IDF scoring returns insufficient results, the algorithm tokenizes the query into 3-character n-grams and compares these against the ngramsInvertedIndex in the DocumentIndex. This allows the system to match "debnce" to "debounce" through shared character sequences, with the similarity score blended into the final ranking at a default weight of 70%.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}