# Difference Between FTS5 Keyword Search and Semantic Vector Search in code-review-graph

> Explore FTS5 keyword search vs semantic vector search in code-review-graph. Learn how BM25 token matching and cosine similarity deliver precise results, with hybrid search merging both.

- Repository: [Tirth Kanani/code-review-graph](https://github.com/tirth8205/code-review-graph)
- Tags: deep-dive
- Published: 2026-08-07

---

**FTS5 keyword search in `code-review-graph` uses SQLite FTS5 BM25 token matching for exact, deterministic lookups, while semantic vector search applies cosine similarity to learned embeddings to match natural-language intent; the `hybrid_search` engine merges both result sets via Reciprocal Rank Fusion when available.**

The `code-review-graph` repository indexes code-knowledge graphs and exposes two distinct query paths for locating nodes. Understanding the difference between FTS5 keyword search and semantic vector search ensures you target the correct engine for exact identifier lookups or natural-language queries.

## Underlying Technology and Architecture

### FTS5 Keyword Search Implementation

According to the `code-review-graph` source code in [`code_review_graph/search.py`](https://github.com/tirth8205/code-review-graph/blob/main/code_review_graph/search.py), the `_fts_search` method queries the SQLite FTS5 virtual table `nodes_fts`. This table stores tokenised fields—`name`, `qualified_name`, `file_path`, and `signature`—using SQLite’s `porter unicode61` tokenizer. Relevance scoring is handled by **BM25** ranking, and index maintenance is performed by `rebuild_fts_index` in the same module.

### Semantic Vector Search Implementation

Semantic search relies on the optional **`EmbeddingStore`** class defined in [`code_review_graph/embeddings.py`](https://github.com/tirth8205/code-review-graph/blob/main/code_review_graph/embeddings.py), which is installed via the `pip install .[embeddings]` extra. As implemented in [`code_review_graph/search.py`](https://github.com/tirth8205/code-review-graph/blob/main/code_review_graph/search.py), the `_embedding_search` method loads embeddings lazily and computes **cosine similarity** between the query and stored vectors representing each node’s textual description. Node identities are resolved by mapping vectors back to the node’s `qualified_name`.

## Query Matching Behavior

### Exact Token Matching with FTS5

**FTS5 keyword search** performs exact token matching after SQLite tokenisation. It supports Boolean operators, phrase queries, and identifier boosting through `detect_query_kind_boost`. This behavior makes it ideal for precise lookups when you know the exact symbol, file path, or signature. If the FTS5 table is unreachable—such as on an older schema—the engine falls back to a simple `LIKE` keyword search through `_keyword_search`.

### Semantic Similarity with Embeddings

**Semantic vector search** matches on the overall meaning of the query rather than specific tokens. It surfaces relevant nodes for synonyms, paraphrases, and natural-language descriptions like “fetch the user-profile API” or “code that validates a JWT” even when exact tokens are absent. Embeddings are generated using `embed_graph_tool`, and the index must be populated before queries can resolve.

## Performance, Setup, and Dependencies

- **FTS5**: Executes as a very fast pure SQL lookup on the indexed `nodes_fts` table. It requires no additional dependencies and works out-of-the-box.
- **Semantic**: Incurs slightly higher latency due to embedding model loading and vector similarity computation. It requires the optional `[embeddings]` extra, the `embed_graph_tool` to populate the vector index, and sufficient memory for the model.

## Fallbacks and Hybrid Result Merging

### Automatic Fallback Chain

If embeddings are unavailable or the vector index is empty, `_embedding_search` automatically falls back to the FTS5 path. If FTS5 is also unavailable, the system falls back to `_keyword_search`. The final mode reported in `_out_mode` reflects the actual path taken.

### Reciprocal Rank Fusion

When both FTS5 and embeddings are available, `hybrid_search` merges the two ranked result lists using **`rrf_merge`** (Reciprocal Rank Fusion). The `_out_mode` values reported are:

- `"hybrid"` – both FTS5 and embeddings contributed
- `"semantic"` – only embeddings produced results
- `"fts"` – only FTS5 produced results
- `"keyword"` – neither succeeded; the `LIKE` fallback was used

## CLI and Python API Examples

### Command-Line Usage

```bash

# FTSS (default) – exact token match

code-review-graph search --mode fts "UserService"

# Semantic vector search – natural-language query

code-review-graph search --mode semantic "fetch user profile data"

```

### Python API Usage

```python
from code_review_graph.graph import GraphStore
from code_review_graph.search import hybrid_search

store = GraphStore("path/to/graph.db")
out_modes = []

# Example 1 – FTSS only (no embeddings installed)

results = hybrid_search(store, "UserService", _out_mode=out_modes)
print(out_modes)          # → ['fts']

# Example 2 – Semantic vector search (embeddings installed)

results = hybrid_search(store, "fetch user profile data", _out_mode=out_modes)
print(out_modes)          # → ['semantic']

# Example 3 – Hybrid (both FTS5 and embeddings available)

results = hybrid_search(store, "create order", _out_mode=out_modes)
print(out_modes)          # → ['hybrid']

```

The public tool `semantic_search_nodes` in [`code_review_graph/tools/query.py`](https://github.com/tirth8205/code-review-graph/blob/main/code_review_graph/tools/query.py) wraps this same `hybrid_search` logic, as documented in [`docs/USAGE.md`](https://github.com/tirth8205/code-review-graph/blob/main/docs/USAGE.md) and [`docs/COMMANDS.md`](https://github.com/tirth8205/code-review-graph/blob/main/docs/COMMANDS.md).

## Summary

- **FTS5 keyword search** is a deterministic, token-based lookup backed by SQLite FTS5 and BM25 ranking; it is always available and extremely fast.
- **Semantic vector search** uses cosine similarity over learned embeddings to interpret natural-language intent, but requires the `[embeddings]` extra and a populated index.
- The engine automatically falls back from semantic to FTS5 to `LIKE` keyword search depending on availability.
- When both systems are present, `hybrid_search` merges ranked results via Reciprocal Rank Fusion and reports the active mode in `_out_mode`.
- Core implementations live in [`code_review_graph/search.py`](https://github.com/tirth8205/code-review-graph/blob/main/code_review_graph/search.py), with embeddings managed by [`code_review_graph/embeddings.py`](https://github.com/tirth8205/code-review-graph/blob/main/code_review_graph/embeddings.py) and tooling exposed through [`code_review_graph/tools/query.py`](https://github.com/tirth8205/code-review-graph/blob/main/code_review_graph/tools/query.py).

## Frequently Asked Questions

### When should I use FTS5 keyword search over semantic vector search?

Use **FTS5 keyword search** when you know the exact identifier, class name, function signature, or file path. It is the best choice for deterministic, high-speed lookups because it matches precise tokens after SQLite `porter unicode61` tokenisation.

### What happens if I query without installing the embeddings extra?

If the `[embeddings]` extra is not installed or the vector index is empty, `hybrid_search` automatically falls back to `_fts_search`. If the FTS5 table is also unavailable, it falls back further to `_keyword_search`, which uses a simple `LIKE` clause.

### How does the hybrid search mode combine results?

When both FTS5 and embeddings are available, `hybrid_search` invokes `rrf_merge` to combine the two ranked lists using Reciprocal Rank Fusion. The function then sets `_out_mode` to `"hybrid"` to indicate that both engines contributed to the final ranking.

### Where is the FTS5 index maintained in the codebase?

The FTS5 virtual table `nodes_fts` is queried by `_fts_search` and rebuilt by `rebuild_fts_index`, both defined in [`code_review_graph/search.py`](https://github.com/tirth8205/code-review-graph/blob/main/code_review_graph/search.py). The table stores tokenised representations of `name`, `qualified_name`, `file_path`, and `signature` for every node in the graph.