# Search System Architecture in 30-seconds-of-code: Key Files and Implementation

> Explore the key files behind the 30-seconds-of-code client-side search system. Learn how pure JavaScript powers full-text search with tokenization, stemming, and TF-IDF.

- Repository: [Angelos Chalaris/30-seconds-of-code](https://github.com/Chalarangelo/30-seconds-of-code)
- Tags: deep-dive
- Published: 2026-02-25

---

**The search system in 30-seconds-of-code is a client-side full-text engine built with pure JavaScript, utilizing tokenization, Porter stemming, TF-IDF scoring, and fuzzy n-gram matching across eight core files in `src/lib/search/` and `src/serializers/`.**

The 30-seconds-of-code repository is an Astro-based static site that powers a popular collection of code snippets. Its **search system** operates entirely without external services, running either in the browser or during server-side prerendering to deliver fast, typo-tolerant results.

## Core Architecture of the Search System

The engine follows a three-layer architecture that separates text processing, index management, and query execution.

### Tokenization and Normalization Layer

This layer handles text cleaning and linguistic processing. Located in [`src/lib/search/utils.js`](https://github.com/Chalarangelo/30-seconds-of-code/blob/main/src/lib/search/utils.js), [`src/lib/search/porterStemmer.js`](https://github.com/Chalarangelo/30-seconds-of-code/blob/main/src/lib/search/porterStemmer.js), and [`src/lib/search/settings.js`](https://github.com/Chalarangelo/30-seconds-of-code/blob/main/src/lib/search/settings.js), it splits content into words, removes punctuation, applies **Porter stemming** to normalize word forms, and filters out stop words defined in the settings configuration.

### Index Construction Layer

The `DocumentIndex` class in [`src/lib/search/documentIndex.js`](https://github.com/Chalarangelo/30-seconds-of-code/blob/main/src/lib/search/documentIndex.js) transforms each snippet into a searchable document. It creates:
- A term frequency map (`Map<term → frequency>`)
- A 3-gram character set for fuzzy matching
- Two inverted indexes mapping terms and n-grams to document IDs

### Search and Scoring Layer

Query execution happens in [`src/lib/search/documentSearch.js`](https://github.com/Chalarangelo/30-seconds-of-code/blob/main/src/lib/search/documentSearch.js) and [`src/lib/search/server.js`](https://github.com/Chalarangelo/30-seconds-of-code/blob/main/src/lib/search/server.js). The system computes **TF-IDF scores** for exact term matches, handles partial-term matching for incomplete queries, and optionally blends **fuzzy n-gram similarity** when the `fuzzy` parameter is enabled.

## Key Files in the Search System

The implementation spans seven utility modules and one serializer:

- **[`src/lib/search/utils.js`](https://github.com/Chalarangelo/30-seconds-of-code/blob/main/src/lib/search/utils.js)** – Token splitting, punctuation cleaning, and n-gram generation utilities.
- **[`src/lib/search/porterStemmer.js`](https://github.com/Chalarangelo/30-seconds-of-code/blob/main/src/lib/search/porterStemmer.js)** – The Porter stemming algorithm implementation for normalizing English words.
- **[`src/lib/search/settings.js`](https://github.com/Chalarangelo/30-seconds-of-code/blob/main/src/lib/search/settings.js)** – Stop-word lists and token-filter configuration rules.
- **[`src/lib/search/documentIndex.js`](https://github.com/Chalarangelo/30-seconds-of-code/blob/main/src/lib/search/documentIndex.js)** – Core class that builds and stores term and n-gram inverted indexes via `addDocument()`.
- **[`src/lib/search/documentSearch.js`](https://github.com/Chalarangelo/30-seconds-of-code/blob/main/src/lib/search/documentSearch.js)** – Ranking logic including `searchForTerms()` for TF-IDF scoring and `searchForNgrams()` for fuzzy matching.
- **[`src/lib/search/server.js`](https://github.com/Chalarangelo/30-seconds-of-code/blob/main/src/lib/search/server.js)** – Public API entry point exporting `search(query, limit?, fuzzy?)` that orchestrates tokenization and scoring.
- **[`src/serializers/searchResultSerializer.js`](https://github.com/Chalarangelo/30-seconds-of-code/blob/main/src/serializers/searchResultSerializer.js)** – Transforms raw document objects into UI-ready JSON containing `title`, `url`, `tag`, and `searchTokens`.

## Data Flow: From Build to Query

Understanding the lifecycle of a search query requires following the data through three distinct phases:

1. **Index Generation** – During `npm run build`, Astro parses each snippet's markdown into plain text. The `DocumentIndex` stores term frequencies, document lengths, and 3-gram sets for every snippet.

2. **Query Processing** – When a user types in the search box, [`src/lib/search/server.js`](https://github.com/Chalarangelo/30-seconds-of-code/blob/main/src/lib/search/server.js) tokenizes the query using the same pipeline as the index (stemming, stop-word removal). It produces two token streams: stemmed terms for exact matching and raw tokens for partial matching.

3. **Ranking and Serialization** – The engine calculates TF-IDF scores via `searchForTerms()`, optionally computes n-gram similarity through `searchForNgrams()` when fuzzy search is enabled, blends the scores using the `fuzzy` weight parameter, and finally passes the results through `SearchResultSerializer.serialize()` to strip internal metadata before sending JSON to the UI.

## Code Examples

### Building the Index at Build Time

```javascript
import DocumentIndex from '#src/lib/search/documentIndex.js';
import { tokenize } from '#src/lib/search/server.js';
import { readFileSync } from 'fs';
import path from 'path';

function loadSnippet(id) {
  const file = path.join('src/content', `${id}.md`);
  const markdown = readFileSync(file, 'utf8');
  const plain = markdown.replace(/<\/?[^>]+(>|$)/g, '');
  return plain;
}

const index = new DocumentIndex();

['array-flatten', 'clone-deep', 'debounce'].forEach(id => {
  const text = loadSnippet(id);
  const termMap = new Map();
  tokenize(text).split(' ').forEach(t => {
    termMap.set(t, (termMap.get(t) ?? 0) + 1);
  });
  index.addDocument(id, termMap, { title: id });
});

```

### Performing a Search Query

```javascript
import createSearch from '#src/lib/search/server.js';

const search = createSearch(index);

const results = search('deep clone object', 5, 0.6);
console.log(results);
/*
[
  { id: 'clone-deep', score: 0.92345, title: 'clone-deep', ... },
  { id: 'clone',       score: 0.41712, title: 'clone',     ... }
]
*/

```

### Serializing Results for the UI

```javascript
import SearchResultSerializer from '#src/serializers/searchResultSerializer.js';

const serialized = results.map(r => SearchResultSerializer.serialize(r));
/*
[
  {
    title: 'clone-deep',
    url: '/clone-deep',
    tag: '<span class="tag">JS</span>',
    searchTokens: 'clone deep object ...'
  },
  …
]
*/

```

## Summary

- The **search system** in 30-seconds-of-code is a pure JavaScript full-text engine requiring no external services.
- Eight core files in `src/lib/search/` and `src/serializers/` handle tokenization, Porter stemming, index construction, TF-IDF scoring, and result serialization.
- The `DocumentIndex` class builds inverted indexes for terms and 3-grams at build time, enabling both exact and fuzzy matching.
- The `search(query, limit?, fuzzy?)` API in [`server.js`](https://github.com/Chalarangelo/30-seconds-of-code/blob/main/server.js) orchestrates query tokenization, partial-term handling, and weighted scoring between TF-IDF and n-gram similarity.

## Frequently Asked Questions

### How does the search system handle typos and fuzzy matching?

The engine generates 3-gram character sets for every document during indexing. When the `fuzzy` parameter is enabled in the `search()` call, the system calculates n-gram similarity between the query and candidate documents via `searchForNgrams()`, then blends this score with the TF-IDF results using the fuzzy weight value to tolerate minor typos.

### Can the search system work without JavaScript on the client?

Yes. Because the site is built with Astro, the search index can be prerendered server-side during the build process. The [`server.js`](https://github.com/Chalarangelo/30-seconds-of-code/blob/main/server.js) module can execute in a serverless function or during static generation, allowing search to function before hydration or for users with JavaScript disabled, though the interactive client-side experience requires JavaScript.

### What algorithm does the search system use for ranking documents?

The primary ranking algorithm is **TF-IDF** (Term Frequency-Inverse Document Frequency) implemented in [`documentSearch.js`](https://github.com/Chalarangelo/30-seconds-of-code/blob/main/documentSearch.js). The system calculates term frequencies against document length and inverse document frequency across the corpus. When fuzzy matching is requested, it augments TF-IDF with **n-gram similarity** scores to handle approximate matches.

### How is the search index built during the Astro build process?

During `npm run build`, each snippet's markdown content is parsed into plain text and tokenized using the same pipeline as runtime queries (stemming, stop-word removal). The `DocumentIndex` class in [`documentIndex.js`](https://github.com/Chalarangelo/30-seconds-of-code/blob/main/documentIndex.js) stores term frequency maps, document lengths, and 3-gram sets for every snippet, creating compact inverted indexes that are serialized and embedded into the static build or API routes.