Difference Between FTS5 Keyword Search and Semantic Vector Search in code-review-graph

FTS5 keyword search in code-review-graph uses SQLite FTS5 BM25 token matching for exact, deterministic lookups, while semantic vector search applies cosine similarity to learned embeddings to match natural-language intent; the hybrid_search engine merges both result sets via Reciprocal Rank Fusion when available.

The code-review-graph repository indexes code-knowledge graphs and exposes two distinct query paths for locating nodes. Understanding the difference between FTS5 keyword search and semantic vector search ensures you target the correct engine for exact identifier lookups or natural-language queries.

Underlying Technology and Architecture

FTS5 Keyword Search Implementation

According to the code-review-graph source code in code_review_graph/search.py, the _fts_search method queries the SQLite FTS5 virtual table nodes_fts. This table stores tokenised fields—name, qualified_name, file_path, and signature—using SQLite’s porter unicode61 tokenizer. Relevance scoring is handled by BM25 ranking, and index maintenance is performed by rebuild_fts_index in the same module.

Semantic Vector Search Implementation

Semantic search relies on the optional EmbeddingStore class defined in code_review_graph/embeddings.py, which is installed via the pip install .[embeddings] extra. As implemented in code_review_graph/search.py, the _embedding_search method loads embeddings lazily and computes cosine similarity between the query and stored vectors representing each node’s textual description. Node identities are resolved by mapping vectors back to the node’s qualified_name.

Query Matching Behavior

Exact Token Matching with FTS5

FTS5 keyword search performs exact token matching after SQLite tokenisation. It supports Boolean operators, phrase queries, and identifier boosting through detect_query_kind_boost. This behavior makes it ideal for precise lookups when you know the exact symbol, file path, or signature. If the FTS5 table is unreachable—such as on an older schema—the engine falls back to a simple LIKE keyword search through _keyword_search.

Semantic Similarity with Embeddings

Semantic vector search matches on the overall meaning of the query rather than specific tokens. It surfaces relevant nodes for synonyms, paraphrases, and natural-language descriptions like “fetch the user-profile API” or “code that validates a JWT” even when exact tokens are absent. Embeddings are generated using embed_graph_tool, and the index must be populated before queries can resolve.

Performance, Setup, and Dependencies

• FTS5: Executes as a very fast pure SQL lookup on the indexed nodes_fts table. It requires no additional dependencies and works out-of-the-box.
• Semantic: Incurs slightly higher latency due to embedding model loading and vector similarity computation. It requires the optional [embeddings] extra, the embed_graph_tool to populate the vector index, and sufficient memory for the model.

Fallbacks and Hybrid Result Merging

Automatic Fallback Chain

If embeddings are unavailable or the vector index is empty, _embedding_search automatically falls back to the FTS5 path. If FTS5 is also unavailable, the system falls back to _keyword_search. The final mode reported in _out_mode reflects the actual path taken.

Reciprocal Rank Fusion

When both FTS5 and embeddings are available, hybrid_search merges the two ranked result lists using rrf_merge (Reciprocal Rank Fusion). The _out_mode values reported are:

• "hybrid" – both FTS5 and embeddings contributed
• "semantic" – only embeddings produced results
• "fts" – only FTS5 produced results
• "keyword" – neither succeeded; the LIKE fallback was used

CLI and Python API Examples

Command-Line Usage

# FTSS (default) – exact token match

code-review-graph search --mode fts "UserService"

# Semantic vector search – natural-language query

code-review-graph search --mode semantic "fetch user profile data"

Python API Usage

from code_review_graph.graph import GraphStore
from code_review_graph.search import hybrid_search

store = GraphStore("path/to/graph.db")
out_modes = []

# Example 1 – FTSS only (no embeddings installed)

results = hybrid_search(store, "UserService", _out_mode=out_modes)
print(out_modes)          # → ['fts']

# Example 2 – Semantic vector search (embeddings installed)

results = hybrid_search(store, "fetch user profile data", _out_mode=out_modes)
print(out_modes)          # → ['semantic']

# Example 3 – Hybrid (both FTS5 and embeddings available)

results = hybrid_search(store, "create order", _out_mode=out_modes)
print(out_modes)          # → ['hybrid']

The public tool semantic_search_nodes in code_review_graph/tools/query.py wraps this same hybrid_search logic, as documented in docs/USAGE.md and docs/COMMANDS.md.

Summary

• FTS5 keyword search is a deterministic, token-based lookup backed by SQLite FTS5 and BM25 ranking; it is always available and extremely fast.
• Semantic vector search uses cosine similarity over learned embeddings to interpret natural-language intent, but requires the [embeddings] extra and a populated index.
• The engine automatically falls back from semantic to FTS5 to LIKE keyword search depending on availability.
• When both systems are present, hybrid_search merges ranked results via Reciprocal Rank Fusion and reports the active mode in _out_mode.
• Core implementations live in code_review_graph/search.py, with embeddings managed by code_review_graph/embeddings.py and tooling exposed through code_review_graph/tools/query.py.

Frequently Asked Questions

When should I use FTS5 keyword search over semantic vector search?

Use FTS5 keyword search when you know the exact identifier, class name, function signature, or file path. It is the best choice for deterministic, high-speed lookups because it matches precise tokens after SQLite porter unicode61 tokenisation.

What happens if I query without installing the embeddings extra?

If the [embeddings] extra is not installed or the vector index is empty, hybrid_search automatically falls back to _fts_search. If the FTS5 table is also unavailable, it falls back further to _keyword_search, which uses a simple LIKE clause.

How does the hybrid search mode combine results?

When both FTS5 and embeddings are available, hybrid_search invokes rrf_merge to combine the two ranked lists using Reciprocal Rank Fusion. The function then sets _out_mode to "hybrid" to indicate that both engines contributed to the final ranking.

Where is the FTS5 index maintained in the codebase?

The FTS5 virtual table nodes_fts is queried by _fts_search and rebuilt by rebuild_fts_index, both defined in code_review_graph/search.py. The table stores tokenised representations of name, qualified_name, file_path, and signature for every node in the graph.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}