# How wigolo's find_similar Tool Implements 3-Way Fusion for Intelligent Document Retrieval

> Discover how wigolo's find_similar tool uses 3-way fusion including FTS5, semantic search, and web results for intelligent document retrieval. Get unified, scaled rankings.

- Repository: [Towhid Khan/wigolo](https://github.com/KnockOutEZ/wigolo)
- Tags: deep-dive
- Published: 2026-07-29

---

**The wigolo `find_similar` tool merges keyword-based FTS5 search, semantic embedding similarity, and live web search results using Reciprocal Rank Fusion (RRF) to deliver unified, relevance-scaled document rankings.**

The `find_similar` tool in the KnockOutEZ/wigolo repository provides a hybrid search mechanism that combines three distinct ranking signals into a single coherent result set. By leveraging local SQLite FTS5 indexes, vector embeddings, and optional web fallback queries, the system ensures high recall even when working with sparse local caches.

## Understanding the 3-Way Fusion Architecture

The architecture rests on **three complementary signal sources** that execute in a coordinated pipeline:

1. **FTS5 Keyword Search**: Fast, inverted-index retrieval from the local SQLite cache using exact and proximity matching
2. **Semantic Embedding Search**: Cosine similarity computation against cached vector embeddings when an embedding provider is available
3. **Live Web Search**: Real-time query expansion and retrieval when local signals prove insufficient for the requested result count

These modalities feed into the **Reciprocal Rank Fusion (RRF)** algorithm implemented in [`src/search/rrf.ts`](https://github.com/KnockOutEZ/wigolo/blob/main/src/search/rrf.ts), which normalizes rankings across incompatible scoring systems without requiring calibration or score translation.

## Data Flow from Request to Ranked Results

### Entry Point and Validation (`handleFindSimilar`)

The execution flow begins in [`src/tools/find-similar.ts`](https://github.com/KnockOutEZ/wigolo/blob/main/src/tools/find-similar.ts), where the `handleFindSimilar` function orchestrates request handling. It first validates that either a `url` or `concept` parameter is present (lines 37-44), then applies SSRF protection via `guardFetchUrl` (lines 49-58).

For URL-based queries, the system performs a **cache-seed warm-up** if the domain contains fewer than five cached pages. This preemptive step triggers `searchTool.handleSearch` to populate the local cache (lines 78-101) before delegating processing to the core search engine.

### Signal Preparation and Parallel Local Search

The core algorithm resides in [`src/search/find-similar.ts`](https://github.com/KnockOutEZ/wigolo/blob/main/src/search/find-similar.ts). Here, `prepareSignal` extracts search terms, titles, and embedding query text from either cached page content or the raw concept string (lines 52-66, 96-115).

Two local searches execute in parallel using `Promise.all`:

- **FTS5 search** populates `fts5RankMap` with keyword-based rankings
- **Embedding search** populates `embeddingRankMap` with similarity scores

This parallel execution (lines 20-47) minimizes latency while maximizing the initial recall set from local caches.

### Web Fallback Strategy

When the combined local hit-set is smaller than `max_results` and `include_web` is enabled, the system triggers `runWebSearchFallback` (lines 49-68). This mechanism generates up to three search queries using `generateSearchQueries`, executes them against the configured search provider, and stores results in `searchRankMap`.

If embedding generation is enabled and new pages are discovered during the fallback, a second embedding pass automatically rescores the newly cached content (lines 71-88) without adding perceptible delay.

## The RRF Fusion Mechanism

The three rank maps converge in `fuseResults` (lines 91-104, 883-925). The `reciprocalRankFusion` function in [`src/search/rrf.ts`](https://github.com/KnockOutEZ/wigolo/blob/main/src/search/rrf.ts) implements the standard RRF formula with a smoothing constant:

```typescript
// src/search/rrf.ts
export function reciprocalRankFusion(lists: Map<string, number>[]) {
  const k = 60;
  const scores = new Map<string, number>();
  for (const list of lists) {
    for (const [url, rank] of list.entries()) {
      const cur = scores.get(url) ?? 0;
      scores.set(url, cur + 1 / (k + rank));
    }
  }
  return scores;
}

```

With `k=60`, the fusion score for each URL becomes the sum of reciprocal ranked positions across all three sources. The `sortByRRFScore` helper orders results by descending fused score, while `fuseResults` normalizes the top score to 1.0 and stores the raw `fused_score` in each result's `match_signals` (lines 334-350).

When `include_ranking_debug` is enabled, the output exposes individual source ranks and the computed RRF score for transparency (lines 155-176).

## Optional Evidence Generation

For applications requiring provenance, `attachEvidence` in [`src/tools/find-similar.ts`](https://github.com/KnockOutEZ/wigolo/blob/main/src/tools/find-similar.ts) (lines 40-73) constructs markdown evidence from the top result's content using `buildEvidenceFromMarkdown`. The system respects token budgets via `applyTokenBudget`, and unless `include_full_markdown` is true, strips the raw `markdown` field to minimize API payload sizes.

## Using find_similar in Your Projects

### Command Line Interface

```bash

# Find similar pages to a specific URL with domain filtering

wigolo find-similar https://example.com --limit=10 --domains=example.com,another.com

# Semantic concept search with live web fallback

wigolo find-similar "react hooks best practices" --include-web=true --max-results=15

```

### Programmatic Integration

```typescript
import { handleFindSimilar } from './src/tools/find-similar.js';
import { getEngines, getRouter } from './src/config.js';

const input = { 
  concept: 'react hooks', 
  max_results: 5, 
  include_ranking_debug: true 
};

const result = await handleFindSimilar(
  input, 
  await getEngines(), 
  await getRouter()
);

if (result.ok) {
  console.log('Method used:', result.data.method);
  console.log('Fusion results:', result.data.results);
} else {
  console.error('Search failed:', result.error_reason);
}

```

The function returns a `StageResult<FindSimilarOutput>` containing the fused results, the active retrieval method (`hybrid`, `fts5`, `embedding`, or `search`), per-source hit counts, and timing telemetry.

## Summary

- **Three-way signal integration**: The tool merges FTS5 keyword rankings, semantic embedding similarities, and live web search results using RRF with `k=60`
- **Cache-first performance**: Local searches execute in parallel; web fallback triggers only when combined local results fall below the requested limit
- **Normalized scoring**: The fusion algorithm normalizes top scores to 1.0, making relevance metrics comparable across heterogeneous search modalities
- **Security controls**: All URL inputs pass through `guardFetchUrl` to prevent SSRF attacks before caching or embedding
- **Debug transparency**: Optional ranking debug output exposes per-source ranks and raw RRF scores for auditability

## Frequently Asked Questions

### What is the significance of k=60 in the RRF formula?

The constant `k=60` in `reciprocalRankFusion` serves as a smoothing parameter that dampens the influence of low-ranked documents. According to the implementation in [`src/search/rrf.ts`](https://github.com/KnockOutEZ/wigolo/blob/main/src/search/rrf.ts), this value prevents items ranked poorly in one source from being completely dominated by items ranked highly in another, ensuring balanced contribution from all three signal sources regardless of their native score distributions.

### How does wigolo decide when to trigger web search fallback?

The `findSimilar` function evaluates fallback conditions at lines 49-68 in [`src/search/find-similar.ts`](https://github.com/KnockOutEZ/wigolo/blob/main/src/search/find-similar.ts). When the combined size of `fts5RankMap` and `embeddingRankMap` is smaller than `max_results` and `include_web` is true, the system invokes `runWebSearchFallback` to generate and execute up to three search queries via `generateSearchQueries`, appending live results to the local hit-set.

### Can specific search modalities be excluded from the 3-way fusion?

While the default hybrid flow automatically utilizes all available modalities, the `mode` parameter influences execution. Setting `mode` to `"crawl-rank"` (lines 58-64 in [`src/search/find-similar.ts`](https://github.com/KnockOutEZ/wigolo/blob/main/src/search/find-similar.ts)) bypasses the fusion entirely for specialized crawling operations. In standard `auto` mode, embedding search is automatically skipped if no embedding provider is configured, effectively reducing to two-way or one-way fusion when services are unavailable.

### What debugging information does the ranking debug option expose?

When `include_ranking_debug` is enabled, each result object contains its raw `fused_score` and individual ranks from the FTS5, embedding, and web search components (lines 155-176 in [`src/search/find-similar.ts`](https://github.com/KnockOutEZ/wigolo/blob/main/src/search/find-similar.ts)). This allows developers to inspect exactly how the RRF algorithm weighted each source and verify that the final ordering reflects the intended balance between keyword precision and semantic similarity.