# How the Context Hub Registry Is Built and Maintained: Architecture and CLI Workflow

> Learn how the Context Hub registry is built and maintained. Explore its architecture and the chub CLI workflow for managing this JSON catalogue.

- Repository: [Andrew Ng/context-hub](https://github.com/andrewyng/context-hub)
- Tags: architecture
- Published: 2026-03-20

---

**The Context Hub registry is a JSON catalogue generated by scanning content directories for markdown files, validating front-matter metadata, and merging author-supplied configurations, then maintained through cache-aware remote fetches, bundle downloads, and in-memory merging via the `chub` CLI.**

The Context Hub registry serves as the central index for all documentation and skills within the `andrewyng/context-hub` ecosystem. This JSON-based catalogue tracks every [`DOC.md`](https://github.com/andrewyng/context-hub/blob/main/DOC.md) and [`SKILL.md`](https://github.com/andrewyng/context-hub/blob/main/SKILL.md) entry across multiple sources, enabling fast local search and offline access. Understanding how this registry is built from raw content and kept synchronized reveals the architecture powering the CLI's content discovery and retrieval capabilities.

## Build Phase: Generating the registry.json Catalogue

The build process transforms raw markdown content into a structured JSON catalogue. Located in [`cli/src/commands/build.js`](https://github.com/andrewyng/context-hub/blob/main/cli/src/commands/build.js), the build command orchestrates file discovery, metadata extraction, and index generation.

### Discovery and Front-Matter Parsing

The `discoverAuthor()` function recursively walks each author’s folder via `findEntryFiles()`, identifying [`DOC.md`](https://github.com/andrewyng/context-hub/blob/main/DOC.md) (documentation) and [`SKILL.md`](https://github.com/andrewyng/context-hub/blob/main/SKILL.md) (skill) entries. Front-matter is parsed using `parseFrontmatter()` from [`cli/src/lib/frontmatter.js`](https://github.com/andrewyng/context-hub/blob/main/cli/src/lib/frontmatter.js), with strict validation for required fields including `name`, `metadata.languages`, and `metadata.versions`. Docs are grouped by language and version, while skills are stored in a flat structure.

### Handling Author-Provided Registries

When an author supplies a pre-built [`registry.json`](https://github.com/andrewyng/context-hub/blob/main/registry.json), the builder loads it directly via `JSON.parse(readFileSync(...))`. Relative paths are automatically prefixed with the author name to ensure globally unique identifiers across the merged registry.

### Validation and Index Generation

All entries accumulate in `allDocs` and `allSkills` arrays. Duplicate IDs trigger build-time errors, while missing descriptions or metadata generate warnings. The final output includes [`dist/registry.json`](https://github.com/andrewyng/context-hub/blob/main/dist/registry.json) containing version, timestamp, docs, skills, and optional `base_url`. Simultaneously, `buildIndex()` from [`cli/src/lib/bm25.js`](https://github.com/andrewyng/context-hub/blob/main/cli/src/lib/bm25.js) generates a BM25 search index saved as [`dist/search-index.json`](https://github.com/andrewyng/context-hub/blob/main/dist/search-index.json).

### CLI Build Command

Use the `chub build` command to generate a fresh registry from a local content tree:

```bash
chub build ./content --output ./dist --base-url https://cdn.example.com/chub

```

- `--validate-only` prints a summary without writing files.
- `--output` overrides the default `content/dist` destination.

## Maintenance Phase: Synchronizing Remote Sources

The maintenance layer ensures the local registry stays synchronized with remote sources. Implemented across [`cli/src/commands/update.js`](https://github.com/andrewyng/context-hub/blob/main/cli/src/commands/update.js), [`cli/src/lib/cache.js`](https://github.com/andrewyng/context-hub/blob/main/cli/src/lib/cache.js), and [`cli/src/lib/registry.js`](https://github.com/andrewyng/context-hub/blob/main/cli/src/lib/registry.js), this phase handles cache management, remote fetching, and runtime merging.

### Configuring Remote and Local Sources

The `loadConfig()` function in [`cli/src/lib/config.js`](https://github.com/andrewyng/context-hub/blob/main/cli/src/lib/config.js) reads [`config.yaml`](https://github.com/andrewyng/context-hub/blob/main/config.yaml), which defines sources as either remote URLs or local filesystem paths:

```yaml
sources:
  - name: python
    url: https://github.com/andrewyng/context-hub-python
  - name: local-docs
    path: /home/user/python-content

```

### Cache Initialization and Freshness Checks

On CLI startup, `ensureRegistry()` in [`cli/src/lib/cache.js`](https://github.com/andrewyng/context-hub/blob/main/cli/src/lib/cache.js) verifies cache freshness via `isSourceCacheFresh`. If no registry exists, the system first attempts to use the bundled [`dist/registry.json`](https://github.com/andrewyng/context-hub/blob/main/dist/registry.json) shipped with the npm package. If unavailable, it downloads every remote source’s [`registry.json`](https://github.com/andrewyng/context-hub/blob/main/registry.json) via `fetchAllRegistries(true)`.

### Remote Updates and Full Bundle Downloads

The `chub update` command triggers `registerUpdateCommand`, which calls `fetchAllRegistries(force)` to conditionally refresh remote registries based on cache timestamps and refresh intervals.

For offline usage, the `--full` flag invokes `fetchFullBundle(sourceName)`, downloading a pre-packed `bundle.tar.gz` containing both [`registry.json`](https://github.com/andrewyng/context-hub/blob/main/registry.json) and [`search-index.json`](https://github.com/andrewyng/context-hub/blob/main/search-index.json).

```bash

# Force refresh ignoring cache timestamps

chub update --force

# Download full bundles for offline work

chub update --full

```

### Runtime Merging and Search

[`cli/src/lib/registry.js`](https://github.com/andrewyng/context-hub/blob/main/cli/src/lib/registry.js) provides `getMerged()`, which lazily loads and combines all source registries into a unified view. The function caches the result in `_merged` and combines BM25 indexes into `_searchIndex`. Operations like `searchEntries()` and `getEntry()` operate on this merged dataset, respecting source filters defined in [`config.yaml`](https://github.com/andrewyng/context-hub/blob/main/config.yaml).

## Working with the Context Hub Registry Programmatically

Developers can interact with the registry directly from Node.js applications. After ensuring the cache exists via `ensureRegistry()`, the merged registry provides searchable access to all content.

```javascript
import { loadConfig } from './cli/src/lib/config.js';
import { getMerged, searchEntries, getEntry } from './cli/src/lib/registry.js';
import { resolveDocPath, resolveEntryFile } from './cli/src/lib/registry.js';

// Ensure cache exists before accessing content
await import('./cli/src/lib/cache.js').then(m => m.ensureRegistry());

// Access all docs from merged registries
const { docs } = getMerged();
console.log(`Loaded ${docs.length} docs`);

// Search with BM25 ranking and filters
const results = searchEntries('redis cache', { tags: 'database' });
console.log('Top result:', results[0]);

// Resolve specific entry paths for a language/version
const { entry } = getEntry('python/redis', 'doc');
const resolved = resolveDocPath(entry, 'python', null);
const fileInfo = resolveEntryFile(resolved, 'doc');
console.log('File on disk:', fileInfo.filePath);

```

To add a new remote source, update [`config.yaml`](https://github.com/andrewyng/context-hub/blob/main/config.yaml) and run `chub update`:

```yaml
sources:
  - name: ruby
    url: https://github.com/andrewyng/context-hub-ruby

```

## Summary

- The **Context Hub registry** is built via `chub build`, which scans for [`DOC.md`](https://github.com/andrewyng/context-hub/blob/main/DOC.md) and [`SKILL.md`](https://github.com/andrewyng/context-hub/blob/main/SKILL.md) files in [`cli/src/commands/build.js`](https://github.com/andrewyng/context-hub/blob/main/cli/src/commands/build.js), validates front-matter using [`cli/src/lib/frontmatter.js`](https://github.com/andrewyng/context-hub/blob/main/cli/src/lib/frontmatter.js), and outputs [`registry.json`](https://github.com/andrewyng/context-hub/blob/main/registry.json) plus a BM25 [`search-index.json`](https://github.com/andrewyng/context-hub/blob/main/search-index.json) via [`cli/src/lib/bm25.js`](https://github.com/andrewyng/context-hub/blob/main/cli/src/lib/bm25.js).
- Author-provided [`registry.json`](https://github.com/andrewyng/context-hub/blob/main/registry.json) files are merged during the build process with prefixed paths to ensure unique IDs.
- Maintenance relies on `chub update` to fetch remote registries using `fetchAllRegistries()` and optional full bundles via `fetchFullBundle()` in [`cli/src/lib/cache.js`](https://github.com/andrewyng/context-hub/blob/main/cli/src/lib/cache.js).
- Runtime access uses `getMerged()` from [`cli/src/lib/registry.js`](https://github.com/andrewyng/context-hub/blob/main/cli/src/lib/registry.js) to provide a unified, searchable view across all configured sources, caching the result in `_merged` and the BM25 index in `_searchIndex`.
- The system supports both online (fetch-on-demand) and offline (full bundle) workflows through configurable cache freshness checks in `ensureRegistry()`.

## Frequently Asked Questions

### What file format does the Context Hub registry use?

The Context Hub registry uses a JSON catalogue format containing a version string, ISO timestamp, arrays of docs and skills, and an optional `base_url` for CDN prefixes. This structure is defined in [`cli/src/commands/build.js`](https://github.com/andrewyng/context-hub/blob/main/cli/src/commands/build.js) and consumed by [`cli/src/lib/registry.js`](https://github.com/andrewyng/context-hub/blob/main/cli/src/lib/registry.js). A separate BM25 search index is stored as [`search-index.json`](https://github.com/andrewyng/context-hub/blob/main/search-index.json) for full-text queries.

### How does the CLI handle duplicate entry IDs during the build process?

During the build phase in [`cli/src/commands/build.js`](https://github.com/andrewyng/context-hub/blob/main/cli/src/commands/build.js), the system accumulates all entries into `allDocs` and `allSkills` arrays. Duplicate IDs trigger a build-time error that aborts the process. Missing metadata fields generate warnings but allow the build to complete.

### Can I use the Context Hub registry offline?

Yes. The `chub update --full` command downloads a pre-packed `bundle.tar.gz` via `fetchFullBundle()` in [`cli/src/lib/cache.js`](https://github.com/andrewyng/context-hub/blob/main/cli/src/lib/cache.js), extracting both [`registry.json`](https://github.com/andrewyng/context-hub/blob/main/registry.json) and [`search-index.json`](https://github.com/andrewyng/context-hub/blob/main/search-index.json) for local offline use. The `ensureRegistry()` function prioritizes these local copies when network access is unavailable or when the cache is fresh.

### What search algorithm powers the Context Hub registry?

The registry uses **BM25** (Best Match 25) ranking for full-text search. The `buildIndex()` function in [`cli/src/lib/bm25.js`](https://github.com/andrewyng/context-hub/blob/main/cli/src/lib/bm25.js) generates the tokenized search index during the build phase, and `searchEntries()` in [`cli/src/lib/registry.js`](https://github.com/andrewyng/context-hub/blob/main/cli/src/lib/registry.js) queries this index at runtime against the merged view.