How GitNexus Implements Hybrid Search with BM25, Semantic Vectors, and RRF

GitNexus combines BM25 keyword search with embedding-based semantic retrieval using Reciprocal Rank Fusion (RRF) to merge results without score normalization, implemented primarily in gitnexus/src/core/search/hybrid-search.ts.

GitNexus delivers high-precision code search by merging traditional full-text retrieval with modern vector similarity. The hybrid search architecture leverages BM25 for lexical matching, dense embeddings for semantic understanding, and Reciprocal Rank Fusion to combine disparate scoring systems into a unified ranking.

The Three Pillars of GitNexus Hybrid Search

BM25 Keyword Retrieval

The lexical foundation relies on BM25 scoring through the searchFTSFromKuzu function. This performs fast full-text search over indexed source code, returning BM25SearchResult objects containing filePath and raw BM25 relevance scores.

Semantic Vector Search

Complementing keyword search, the semanticSearch function queries vector embeddings stored in the vector database. Each SemanticSearchResult includes filePath, vector distance, and rich node metadata such as name, label, and line ranges.

Reciprocal Rank Fusion (RRF)

Rather than normalizing incompatible BM25 and cosine similarity scores, GitNexus uses RRF. The formula rrfScore = 1 / (K + r) is applied with K = 60, where r is the rank position in the original list. This rank-based approach ensures neither retrieval method dominates due to score scale differences.

Core Implementation in hybrid-search.ts

The mergeWithRRF Function

Located in gitnexus/src/core/search/hybrid-search.ts, the mergeWithRRF function orchestrates fusion. It processes BM25 results (lines 56-57) and semantic results (lines 70-71) separately, calculating RRF scores for each rank position.

Result Aggregation and Scoring

A Map<string, HybridSearchResult> aggregates hits keyed by filePath. For BM25-only entries, the map stores sources: ['bm25'] and the original bm25Score. For semantic-only hits, it records sources: ['semantic'] and converts distance to similarity via semanticScore = 1 - distance. When both sources agree on a file, their RRF scores sum and sources becomes ['bm25', 'semantic'].

Final Ranking and Output

After aggregation, the map converts to an array and sorts by the combined RRF score in descending order. The implementation returns the top limit results with assigned rank positions (lines 102-108), providing a unified, relevance-ordered list regardless of the original retrieval method.

End-to-End Integration

Web Worker Orchestration

The ingestion pipeline in gitnexus-web/src/workers/ingestion.worker.ts demonstrates practical deployment. At line 471, it fetches BM25 results via searchBM25, retrieves semantic matches through searchSemantic, and invokes mergeWithRRF to produce the final ranked list for indexing.

LLM Tool Interface

GitNexus exposes hybrid search as a tool for LLM agents in gitnexus-web/src/core/llm/tools.ts. The search command internally executes the same BM25 + semantic + RRF pipeline, returning results grouped by process or cluster for downstream reasoning tasks.

Practical Usage Examples

Direct implementation from a Node script:

import { searchBM25 } from './bm25-index.js';
import { semanticSearch } from '../embeddings/search.js';
import { mergeWithRRF } from '../core/search/hybrid-search.js';

async function hybrid(query: string, limit = 10) {
  const bm25 = await searchBM25(query, limit * 3);      // overshoot for better RRF
  const semantic = await semanticSearch(query, limit * 3);
  return mergeWithRRF(bm25, semantic, limit);
}

Using the built-in LLM tool:

await agent.run(`
  #search
  query: "how does the repo handle user authentication?"
`);

Result structure:

[
  {
    "filePath": "src/auth/login.ts",
    "score": 0.0162,
    "rank": 1,
    "sources": ["bm25","semantic"],
    "bm25Score": 2.3,
    "semanticScore": 0.84,
    "nodeId": "n123",
    "name": "loginHandler",
    "label": "function",
    "startLine": 12,
    "endLine": 28
  }
]

Summary

• GitNexus implements hybrid search in gitnexus/src/core/search/hybrid-search.ts by combining BM25 lexical retrieval with vector-based semantic search.
• The system uses Reciprocal Rank Fusion (RRF) with K=60 to merge results without normalizing incompatible score scales, applying the formula 1/(K + r) to each rank position.
• Results are aggregated in a Map<string, HybridSearchResult> keyed by filePath, with RRF scores summed when both retrieval methods return the same file.
• The pipeline is orchestrated by gitnexus-web/src/workers/ingestion.worker.ts and exposed to LLM agents via gitnexus-web/src/core/llm/tools.ts.

Frequently Asked Questions

What is the constant K value used in GitNexus RRF and why?

GitNexus uses K = 60 for the Reciprocal Rank Fusion formula. This value is the standard constant recommended in RRF literature, providing a balance that prevents top-ranked items from dominating while still giving significant weight to high positions in either retrieval list.

How does GitNexus handle cases where BM25 and semantic search return the same file?

When both retrieval methods return the same filePath, GitNexus sums their individual RRF scores to produce a combined relevance score. The resulting HybridSearchResult includes both bm25Score and semanticScore in its metadata, with the sources array set to ['bm25', 'semantic'] to indicate the consensus.

Why does GitNexus use rank-based fusion instead of score normalization?

GitNexus avoids score normalization because BM25 relevance scores and vector cosine distances operate on incompatible scales and distributions. Reciprocal Rank Fusion eliminates the need for calibration by relying solely on relative rank positions, ensuring that neither lexical nor semantic retrieval dominates the final ranking due to arbitrary score magnitudes.

Where can I find the unit tests for the hybrid search implementation?

The RRF merging behavior and hybrid search logic are validated in gitnexus/test/unit/hybrid-search.test.ts. These tests verify that the mergeWithRRF function correctly aggregates results from disparate sources, handles overlapping file paths, and produces stable rankings across different query scenarios.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}