How wigolo's find_similar Tool Implements 3-Way Fusion for Intelligent Document Retrieval

The wigolo find_similar tool merges keyword-based FTS5 search, semantic embedding similarity, and live web search results using Reciprocal Rank Fusion (RRF) to deliver unified, relevance-scaled document rankings.

The find_similar tool in the KnockOutEZ/wigolo repository provides a hybrid search mechanism that combines three distinct ranking signals into a single coherent result set. By leveraging local SQLite FTS5 indexes, vector embeddings, and optional web fallback queries, the system ensures high recall even when working with sparse local caches.

Understanding the 3-Way Fusion Architecture

The architecture rests on three complementary signal sources that execute in a coordinated pipeline:

1. FTS5 Keyword Search: Fast, inverted-index retrieval from the local SQLite cache using exact and proximity matching
2. Semantic Embedding Search: Cosine similarity computation against cached vector embeddings when an embedding provider is available
3. Live Web Search: Real-time query expansion and retrieval when local signals prove insufficient for the requested result count

These modalities feed into the Reciprocal Rank Fusion (RRF) algorithm implemented in src/search/rrf.ts, which normalizes rankings across incompatible scoring systems without requiring calibration or score translation.

Data Flow from Request to Ranked Results

Entry Point and Validation (handleFindSimilar)

The execution flow begins in src/tools/find-similar.ts, where the handleFindSimilar function orchestrates request handling. It first validates that either a url or concept parameter is present (lines 37-44), then applies SSRF protection via guardFetchUrl (lines 49-58).

For URL-based queries, the system performs a cache-seed warm-up if the domain contains fewer than five cached pages. This preemptive step triggers searchTool.handleSearch to populate the local cache (lines 78-101) before delegating processing to the core search engine.

Signal Preparation and Parallel Local Search

The core algorithm resides in src/search/find-similar.ts. Here, prepareSignal extracts search terms, titles, and embedding query text from either cached page content or the raw concept string (lines 52-66, 96-115).

Two local searches execute in parallel using Promise.all:

• FTS5 search populates fts5RankMap with keyword-based rankings
• Embedding search populates embeddingRankMap with similarity scores

This parallel execution (lines 20-47) minimizes latency while maximizing the initial recall set from local caches.

Web Fallback Strategy

When the combined local hit-set is smaller than max_results and include_web is enabled, the system triggers runWebSearchFallback (lines 49-68). This mechanism generates up to three search queries using generateSearchQueries, executes them against the configured search provider, and stores results in searchRankMap.

If embedding generation is enabled and new pages are discovered during the fallback, a second embedding pass automatically rescores the newly cached content (lines 71-88) without adding perceptible delay.

The RRF Fusion Mechanism

The three rank maps converge in fuseResults (lines 91-104, 883-925). The reciprocalRankFusion function in src/search/rrf.ts implements the standard RRF formula with a smoothing constant:

// src/search/rrf.ts
export function reciprocalRankFusion(lists: Map<string, number>[]) {
  const k = 60;
  const scores = new Map<string, number>();
  for (const list of lists) {
    for (const [url, rank] of list.entries()) {
      const cur = scores.get(url) ?? 0;
      scores.set(url, cur + 1 / (k + rank));
    }
  }
  return scores;
}

With k=60, the fusion score for each URL becomes the sum of reciprocal ranked positions across all three sources. The sortByRRFScore helper orders results by descending fused score, while fuseResults normalizes the top score to 1.0 and stores the raw fused_score in each result's match_signals (lines 334-350).

When include_ranking_debug is enabled, the output exposes individual source ranks and the computed RRF score for transparency (lines 155-176).

Optional Evidence Generation

For applications requiring provenance, attachEvidence in src/tools/find-similar.ts (lines 40-73) constructs markdown evidence from the top result's content using buildEvidenceFromMarkdown. The system respects token budgets via applyTokenBudget, and unless include_full_markdown is true, strips the raw markdown field to minimize API payload sizes.

Using find_similar in Your Projects

Command Line Interface

# Find similar pages to a specific URL with domain filtering

wigolo find-similar https://example.com --limit=10 --domains=example.com,another.com

# Semantic concept search with live web fallback

wigolo find-similar "react hooks best practices" --include-web=true --max-results=15

Programmatic Integration

import { handleFindSimilar } from './src/tools/find-similar.js';
import { getEngines, getRouter } from './src/config.js';

const input = { 
  concept: 'react hooks', 
  max_results: 5, 
  include_ranking_debug: true 
};

const result = await handleFindSimilar(
  input, 
  await getEngines(), 
  await getRouter()
);

if (result.ok) {
  console.log('Method used:', result.data.method);
  console.log('Fusion results:', result.data.results);
} else {
  console.error('Search failed:', result.error_reason);
}

The function returns a StageResult<FindSimilarOutput> containing the fused results, the active retrieval method (hybrid, fts5, embedding, or search), per-source hit counts, and timing telemetry.

Summary

• Three-way signal integration: The tool merges FTS5 keyword rankings, semantic embedding similarities, and live web search results using RRF with k=60
• Cache-first performance: Local searches execute in parallel; web fallback triggers only when combined local results fall below the requested limit
• Normalized scoring: The fusion algorithm normalizes top scores to 1.0, making relevance metrics comparable across heterogeneous search modalities
• Security controls: All URL inputs pass through guardFetchUrl to prevent SSRF attacks before caching or embedding
• Debug transparency: Optional ranking debug output exposes per-source ranks and raw RRF scores for auditability

Frequently Asked Questions

What is the significance of k=60 in the RRF formula?

The constant k=60 in reciprocalRankFusion serves as a smoothing parameter that dampens the influence of low-ranked documents. According to the implementation in src/search/rrf.ts, this value prevents items ranked poorly in one source from being completely dominated by items ranked highly in another, ensuring balanced contribution from all three signal sources regardless of their native score distributions.

How does wigolo decide when to trigger web search fallback?

The findSimilar function evaluates fallback conditions at lines 49-68 in src/search/find-similar.ts. When the combined size of fts5RankMap and embeddingRankMap is smaller than max_results and include_web is true, the system invokes runWebSearchFallback to generate and execute up to three search queries via generateSearchQueries, appending live results to the local hit-set.

Can specific search modalities be excluded from the 3-way fusion?

While the default hybrid flow automatically utilizes all available modalities, the mode parameter influences execution. Setting mode to "crawl-rank" (lines 58-64 in src/search/find-similar.ts) bypasses the fusion entirely for specialized crawling operations. In standard auto mode, embedding search is automatically skipped if no embedding provider is configured, effectively reducing to two-way or one-way fusion when services are unavailable.

What debugging information does the ranking debug option expose?

When include_ranking_debug is enabled, each result object contains its raw fused_score and individual ranks from the FTS5, embedding, and web search components (lines 155-176 in src/search/find-similar.ts). This allows developers to inspect exactly how the RRF algorithm weighted each source and verify that the final ordering reflects the intended balance between keyword precision and semantic similarity.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}