# How Wigolo Implements Keyword and Vector Hybrid Indexing for Local Cache Search

> Discover how Wigolo delivers lightning-fast local cache search with hybrid keyword and vector indexing. Learn about its RRF fusion and graceful degradation for optimal performance.

- Repository: [Towhid Khan/wigolo](https://github.com/KnockOutEZ/wigolo)
- Tags: internals
- Published: 2026-07-29

---

**Wigolo combines SQLite FTS5 for keyword search and SQLite-vec for semantic similarity, fusing both result sets using Reciprocal Rank Fusion (RRF) while gracefully degrading to keyword-only retrieval when vector services are unavailable.**

Wigolo, an open-source web caching tool from KnockOutEZ/wigolo, manages fetched web pages in a local SQLite database with a sophisticated dual-index strategy. The system stores page content alongside pre-computed embeddings, enabling queries that match both exact keywords and semantic meaning. According to the source code, this hybrid approach is orchestrated through parallel index lookups and statistical rank fusion to deliver more relevant cached results than either technique alone.

## The Dual-Index Storage Architecture

Wigolo’s local cache maintains two distinct indexes over the same corpus of cached web pages. Each entry in the primary cache table stores the URL, title, markdown body, and a **content hash** used for change detection, while a separate vector table holds dense embeddings for semantic comparison.

### Keyword Index with FTS5

The keyword layer uses SQLite’s **FTS5** extension to index the text content of cached pages. Located in [`src/cache/store.js`](https://github.com/KnockOutEZ/wigolo/blob/main/src/cache/store.js), the `ftsSearchRanked` function executes full-text queries against this index, returning lexical matches ranked by FTS5’s internal scoring algorithm. This provides fast, exact matching for terms present in the page title or markdown body.

### Vector Index with SQLite-vec

For semantic search, Wigolo leverages **SQLite-vec** via [`src/providers/vector-store.js`](https://github.com/KnockOutEZ/wigolo/blob/main/src/providers/vector-store.js). When pages are cached, the system generates embeddings through `getEmbedProvider` (defined in [`src/providers/embed-provider.js`](https://github.com/KnockOutEZ/wigolo/blob/main/src/providers/embed-provider.js)) and stores them in the vector store. During query time, the user’s query text is converted to a dense vector using the same embedding provider, then searched against the pre-computed page embeddings to find semantically similar content regardless of exact keyword overlap.

## Hybrid Search Orchestration

The cache tool entry point in [`src/tools/cache.ts`](https://github.com/KnockOutEZ/wigolo/blob/main/src/tools/cache.ts) handles mode routing through the `handleCache` function. When invoked with `mode: "hybrid"`, the system delegates to `runHybridSearch` (lines 71-84), which coordinates the dual retrieval process.

### Parallel Candidate Retrieval

`runHybridSearch` executes both index lookups simultaneously using `Promise.all`:

1. **FTS branch:** Calls `ftsSearchRanked` with the text query against the FTS5 index.
2. **Vector branch:** Embeds the query via `embedProvider.embed([query])` (lines 95-99) and searches the vector store via `store.search()`.

Both result sets are fetched in parallel to minimize latency. According to the implementation in [`src/tools/cache.ts`](https://github.com/KnockOutEZ/wigolo/blob/main/src/tools/cache.ts) (lines 105-108), the system retrieves candidates up to a calculated limit before fusion occurs.

## Reciprocal Rank Fusion (RRF) Pipeline

Once the dual indexes return their respective rankings, Wigolo merges them using a statistically robust fusion algorithm.

### Candidate Limit Calculation

Before fusion, results are trimmed using constants defined in [`src/tools/cache.ts`](https://github.com/KnockOutEZ/wigolo/blob/main/src/tools/cache.ts): `DEFAULT_HYBRID_LIMIT`, `HYBRID_CANDIDATE_FLOOR`, and `HYBRID_CANDIDATE_FACTOR`. The candidate pool defaults to `max(50, limit × 5)`, ensuring sufficient overlap for meaningful fusion while maintaining performance.

### Rank Fusion Algorithm

The `buildRankMap` and `reciprocalRankFusion` functions (located in [`src/search/rrf.js`](https://github.com/KnockOutEZ/wigolo/blob/main/src/search/rrf.js)) implement the standard RRF formula:

```

RRF score = Σ 1/(k + rank)

```

Where `k` is a constant (typically 60) and rank is the position in each individual result list. URLs appearing in both keyword and vector results receive higher scores, surfacing pages that are both lexically relevant and semantically similar.

### Result Hydration

After fusion, the top `limit` URLs are hydrated back into full cache rows. The system iterates through the fused ranking and calls `getCachedContentByNormalizedUrl` for each entry (lines 118-127 in [`src/tools/cache.ts`](https://github.com/KnockOutEZ/wigolo/blob/main/src/tools/cache.ts)), retrieving the complete title, markdown body, and fetched-at timestamp for the final result set.

## Fallback Mechanisms and Error Handling

The hybrid system includes defensive logic for when vector infrastructure is unavailable. If `getEmbedProvider()` or `getVectorStore()` fails to initialize, or if the query embedding throws an exception, `runHybridSearch` logs "hybrid search unavailable" and returns the standard FTS-only results (see the early-return at line 84 in [`src/tools/cache.ts`](https://github.com/KnockOutEZ/wigolo/blob/main/src/tools/cache.ts)). This ensures the cache tool remains functional even without embedding providers or when the SQLite-vec extension is not installed.

## Usage Examples

### Programmatic API

```typescript
// src/tools/cache.ts usage with hybrid mode
import { handleCache } from './src/tools/cache.js';

const result = await handleCache({
  mode: 'hybrid',          // enable keyword + vector search
  query: 'lightweight JavaScript frameworks',
  limit: 5,                // return top 5 results
  max_tokens_out: 500,     // optional output token budget
});

console.log(result.results);
// Returns hydrated cache items with title, markdown, and metadata

```

### Command Line Interface

```bash

# CLI invocation via wigolo cache sub-command

wigolo cache --mode hybrid "javascript frameworks" --limit 10

```

Both interfaces route through `handleCache`, automatically triggering `runHybridSearch` when the hybrid mode flag is detected.

## Summary

- **Dual indexes:** Wigolo maintains separate FTS5 (keyword) and SQLite-vec (semantic) indexes over cached web pages in [`src/tools/cache.ts`](https://github.com/KnockOutEZ/wigolo/blob/main/src/tools/cache.ts).
- **Parallel retrieval:** The `runHybridSearch` function queries both indexes simultaneously via `Promise.all` for optimal performance.
- **RRF fusion:** Results are merged using `reciprocalRankFusion` and `buildRankMap` from [`src/search/rrf.js`](https://github.com/KnockOutEZ/wigolo/blob/main/src/search/rrf.js), with candidate limits controlled by `HYBRID_CANDIDATE_FACTOR` and `HYBRID_CANDIDATE_FLOOR`.
- **Graceful degradation:** If vector services fail, the system automatically falls back to pure FTS5 search via `ftsSearchRanked`.
- **Content validation:** Each cache row includes a content hash for change detection, ensuring index consistency with stored data.

## Frequently Asked Questions

### What is Reciprocal Rank Fusion (RRF) and why does Wigolo use it?

Reciprocal Rank Fusion is a statistical method for combining multiple ranked result lists without requiring score normalization between different search algorithms. Wigolo uses RRF (implemented in [`src/search/rrf.js`](https://github.com/KnockOutEZ/wigolo/blob/main/src/search/rrf.js)) because it effectively balances keyword exactness from FTS5 with semantic similarity from vector embeddings, giving higher scores to documents that rank well in both systems while remaining robust to the different scoring scales of full-text and cosine similarity metrics.

### How does Wigolo handle cache invalidation with content hashes?

Each row in the cache table stores a content hash alongside the URL and markdown body. When re-fetching a URL, Wigolo recalculates the hash and compares it against the stored value; if they differ, the system updates both the FTS5 index and the SQLite-vec vector store with the new content, ensuring that the hybrid indexes remain synchronized with the latest page versions.

### What happens if the embedding provider is unavailable?

If the embedding provider fails to initialize or the query embedding throws an error, `runHybridSearch` in [`src/tools/cache.ts`](https://github.com/KnockOutEZ/wigolo/blob/main/src/tools/cache.ts) catches the exception and returns the results from `ftsSearchRanked` alone. The system logs a warning indicating that hybrid search is unavailable, ensuring the cache tool remains operational with keyword-only search capabilities.

### Can I adjust the number of hybrid search candidates?

Yes. The candidate pool size is calculated using `HYBRID_CANDIDATE_FACTOR` (default multiplier) and `HYBRID_CANDIDATE_FLOOR` (minimum threshold), defined in [`src/tools/cache.ts`](https://github.com/KnockOutEZ/wigolo/blob/main/src/tools/cache.ts). By default, the system retrieves `max(50, limit × 5)` candidates from each index before fusion, but these constants can be modified in the source to tune the trade-off between recall and performance.