How Wigolo Implements Keyword and Vector Hybrid Indexing for Local Cache Search

Wigolo combines SQLite FTS5 for keyword search and SQLite-vec for semantic similarity, fusing both result sets using Reciprocal Rank Fusion (RRF) while gracefully degrading to keyword-only retrieval when vector services are unavailable.

Wigolo, an open-source web caching tool from KnockOutEZ/wigolo, manages fetched web pages in a local SQLite database with a sophisticated dual-index strategy. The system stores page content alongside pre-computed embeddings, enabling queries that match both exact keywords and semantic meaning. According to the source code, this hybrid approach is orchestrated through parallel index lookups and statistical rank fusion to deliver more relevant cached results than either technique alone.

The Dual-Index Storage Architecture

Wigolo’s local cache maintains two distinct indexes over the same corpus of cached web pages. Each entry in the primary cache table stores the URL, title, markdown body, and a content hash used for change detection, while a separate vector table holds dense embeddings for semantic comparison.

Keyword Index with FTS5

The keyword layer uses SQLite’s FTS5 extension to index the text content of cached pages. Located in src/cache/store.js, the ftsSearchRanked function executes full-text queries against this index, returning lexical matches ranked by FTS5’s internal scoring algorithm. This provides fast, exact matching for terms present in the page title or markdown body.

Vector Index with SQLite-vec

For semantic search, Wigolo leverages SQLite-vec via src/providers/vector-store.js. When pages are cached, the system generates embeddings through getEmbedProvider (defined in src/providers/embed-provider.js) and stores them in the vector store. During query time, the user’s query text is converted to a dense vector using the same embedding provider, then searched against the pre-computed page embeddings to find semantically similar content regardless of exact keyword overlap.

Hybrid Search Orchestration

The cache tool entry point in src/tools/cache.ts handles mode routing through the handleCache function. When invoked with mode: "hybrid", the system delegates to runHybridSearch (lines 71-84), which coordinates the dual retrieval process.

Parallel Candidate Retrieval

runHybridSearch executes both index lookups simultaneously using Promise.all:

1. FTS branch: Calls ftsSearchRanked with the text query against the FTS5 index.
2. Vector branch: Embeds the query via embedProvider.embed([query]) (lines 95-99) and searches the vector store via store.search().

Both result sets are fetched in parallel to minimize latency. According to the implementation in src/tools/cache.ts (lines 105-108), the system retrieves candidates up to a calculated limit before fusion occurs.

Reciprocal Rank Fusion (RRF) Pipeline

Once the dual indexes return their respective rankings, Wigolo merges them using a statistically robust fusion algorithm.

Candidate Limit Calculation

Before fusion, results are trimmed using constants defined in src/tools/cache.ts: DEFAULT_HYBRID_LIMIT, HYBRID_CANDIDATE_FLOOR, and HYBRID_CANDIDATE_FACTOR. The candidate pool defaults to max(50, limit × 5), ensuring sufficient overlap for meaningful fusion while maintaining performance.

Rank Fusion Algorithm

The buildRankMap and reciprocalRankFusion functions (located in src/search/rrf.js) implement the standard RRF formula:

RRF score = Σ 1/(k + rank)

Where k is a constant (typically 60) and rank is the position in each individual result list. URLs appearing in both keyword and vector results receive higher scores, surfacing pages that are both lexically relevant and semantically similar.

Result Hydration

After fusion, the top limit URLs are hydrated back into full cache rows. The system iterates through the fused ranking and calls getCachedContentByNormalizedUrl for each entry (lines 118-127 in src/tools/cache.ts), retrieving the complete title, markdown body, and fetched-at timestamp for the final result set.

Fallback Mechanisms and Error Handling

The hybrid system includes defensive logic for when vector infrastructure is unavailable. If getEmbedProvider() or getVectorStore() fails to initialize, or if the query embedding throws an exception, runHybridSearch logs "hybrid search unavailable" and returns the standard FTS-only results (see the early-return at line 84 in src/tools/cache.ts). This ensures the cache tool remains functional even without embedding providers or when the SQLite-vec extension is not installed.

Usage Examples

Programmatic API

// src/tools/cache.ts usage with hybrid mode
import { handleCache } from './src/tools/cache.js';

const result = await handleCache({
  mode: 'hybrid',          // enable keyword + vector search
  query: 'lightweight JavaScript frameworks',
  limit: 5,                // return top 5 results
  max_tokens_out: 500,     // optional output token budget
});

console.log(result.results);
// Returns hydrated cache items with title, markdown, and metadata

Command Line Interface

# CLI invocation via wigolo cache sub-command

wigolo cache --mode hybrid "javascript frameworks" --limit 10

Both interfaces route through handleCache, automatically triggering runHybridSearch when the hybrid mode flag is detected.

Summary

• Dual indexes: Wigolo maintains separate FTS5 (keyword) and SQLite-vec (semantic) indexes over cached web pages in src/tools/cache.ts.
• Parallel retrieval: The runHybridSearch function queries both indexes simultaneously via Promise.all for optimal performance.
• RRF fusion: Results are merged using reciprocalRankFusion and buildRankMap from src/search/rrf.js, with candidate limits controlled by HYBRID_CANDIDATE_FACTOR and HYBRID_CANDIDATE_FLOOR.
• Graceful degradation: If vector services fail, the system automatically falls back to pure FTS5 search via ftsSearchRanked.
• Content validation: Each cache row includes a content hash for change detection, ensuring index consistency with stored data.

Frequently Asked Questions

What is Reciprocal Rank Fusion (RRF) and why does Wigolo use it?

Reciprocal Rank Fusion is a statistical method for combining multiple ranked result lists without requiring score normalization between different search algorithms. Wigolo uses RRF (implemented in src/search/rrf.js) because it effectively balances keyword exactness from FTS5 with semantic similarity from vector embeddings, giving higher scores to documents that rank well in both systems while remaining robust to the different scoring scales of full-text and cosine similarity metrics.

How does Wigolo handle cache invalidation with content hashes?

Each row in the cache table stores a content hash alongside the URL and markdown body. When re-fetching a URL, Wigolo recalculates the hash and compares it against the stored value; if they differ, the system updates both the FTS5 index and the SQLite-vec vector store with the new content, ensuring that the hybrid indexes remain synchronized with the latest page versions.

What happens if the embedding provider is unavailable?

If the embedding provider fails to initialize or the query embedding throws an error, runHybridSearch in src/tools/cache.ts catches the exception and returns the results from ftsSearchRanked alone. The system logs a warning indicating that hybrid search is unavailable, ensuring the cache tool remains operational with keyword-only search capabilities.

Can I adjust the number of hybrid search candidates?

Yes. The candidate pool size is calculated using HYBRID_CANDIDATE_FACTOR (default multiplier) and HYBRID_CANDIDATE_FLOOR (minimum threshold), defined in src/tools/cache.ts. By default, the system retrieves max(50, limit × 5) candidates from each index before fusion, but these constants can be modified in the source to tune the trade-off between recall and performance.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}