# How to Debug Memory Retrieval Issues with BM25 and Vector Hybrid Search

> Debug BM25 and vector hybrid search issues in TencentDB Agent Memory. Verify FTS5 support, enable nativeHybridSearch, and trace query execution with logs to resolve retrieval failures.

- Repository: [Tencent Cloud/TencentDB-Agent-Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory)
- Tags: how-to-guide
- Published: 2026-08-26

---

**To debug BM25 and vector hybrid search failures in TencentDB Agent Memory, verify that the vector store initialized correctly with FTS5 support, confirm `nativeHybridSearch` capability is enabled, and trace the query through embedding generation, isolation filtering, and Reciprocal Rank Fusion (RRF) merging using debug logs in [`memory-search.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/memory-search.ts).**

The TencentDB Agent Memory service retrieves relevant conversational memories using a hybrid pipeline that combines **BM25 full-text search (FTS5)** with **dense vector similarity**. This multi-layered architecture (L3 → L2 → L1 → L0) falls back from fast keyword matches to computationally expensive embedding searches when necessary. When retrieval fails or returns irrelevant results, developers must trace the execution path from store initialization through query construction to result merging.

## Verify Vector Store Initialization and Capabilities

First, ensure the underlying storage layer is healthy. The pipeline factory initializes the SQLite/FAISS store in [`MemoryCore/src/utils/pipeline-factory.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryCore/src/utils/pipeline-factory.ts) around lines 311-354.

### Check Store Initialization and Degraded Mode

If `vectorStore.init` fails, the system logs `Store init failed…` and operates in a degraded state where hybrid search is unavailable. Call `vectorStore.isDegraded()` to verify operational status before attempting retrieval.

### Confirm FTS5 Index Availability

The hybrid path requires the FTS5 keyword index. In [`MemoryCore/src/core/tools/memory-search.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryCore/src/core/tools/memory-search.ts) at line 127, the code checks `vectorStore.isFtsAvailable()`. If this returns `false`, the system falls back to pure embedding search, bypassing BM25 entirely.

### Validate Native Hybrid Search Support

For optimized retrieval, the store must advertise `nativeHybridSearch` and provide `searchL1Hybrid` / `searchL0Hybrid` methods. As implemented in [`memory-search.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/memory-search.ts) lines 149-151, absence of this capability forces the system to execute separate FTS5 and vector calls, merging results manually with RRF.

## Debug Query Construction and Embedding Generation

### Verify Embedding Service Connectivity

The `embeddingService.embed` call in [`MemoryCore/src/core/tdai-core.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryCore/src/core/tdai-core.ts) must successfully convert query text into embeddings. A missing or failed embedding aborts the vector branch entirely, causing the system to rely solely on BM25 if available.

### Inspect Query Payload Structure

In [`MemoryCore/src/core/tools/conversation-search.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryCore/src/core/tools/conversation-search.ts) lines 146-148, the hybrid query requires both `ftsQuery` (for BM25) and `queryEmbedding` fields. Ensure the request payload includes both the `query` text and optional `ftQuery` parameters to populate both search branches.

### Review Isolation Filters

Hybrid search can be scoped by `isolationFilter` parameters such as session-id or agent-id. In [`memory-search.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/memory-search.ts) lines 151-154, overly strict filters prune valid results before the RRF merge. Verify filter values passed from the caller match expected memory boundaries and are not excessively restrictive.

## Analyze Result Merging and Search Paths

### Monitor Path Selection and Logging

The pipeline logs the chosen retrieval path (`hybrid`, `embedding`, or `fts`) at line 111 in [`memory-search.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/memory-search.ts). Increase log verbosity to `logger.debug` to see exact branches taken during query execution and identify unexpected fallbacks.

### Debug Reciprocal Rank Fusion Logic

After BM25 and vector branches return results, the system applies **Reciprocal Rank Fusion (RRF)**. The `mergeRrf` implementation (lines 197-233 in [`memory-search.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/memory-search.ts)) handles scoring normalization and duplicate removal. Incorrect RRF weights or deduplication bugs can suppress valid hits or alter ranking order.

## Common Failure Scenarios and Solutions

**No results when BM25 alone works**: This indicates `nativeHybridSearch` is disabled or `searchL1Hybrid` is missing. Upgrade to a store version supporting native hybrid (SQLite FTS5 + vector extensions) or disable hybrid mode by setting `hybrid: false` in `retrievalConfig`.

**Too few results returned**: An overly restrictive `isolationFilter` or insufficient `candidateK` value limits the candidate pool. Increase `candidateK` or relax filter constraints in the retrieval request to allow more candidates into the merge phase.

**Wrong relevance ordering**: RRF merging bugs or missing score normalization skew final rankings. Add debug prints of individual branch scores before the `mergeRrf` call in [`memory-search.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/memory-search.ts) to verify BM25 and vector scores are normalized correctly.

**Vector store undefined errors**: The store failed to initialize and entered degraded mode. Check startup logs in [`pipeline-factory.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/pipeline-factory.ts) line 311 and verify the SQLite file (`vectors.db`) is writable and accessible at the configured path.

**High latency on retrieval**: Falling back to separate FTS and vector calls instead of native hybrid search doubles query time. Enable native hybrid support or pre-index vectors with FTS5 using `vectorStore.enableHybrid()` to use single-pass `searchL1Hybrid`.

## Hybrid Search Implementation Details

The core retrieval function `searchMemory` in [`MemoryCore/src/core/tools/memory-search.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryCore/src/core/tools/memory-search.ts) selects the strategy based on capability detection:

```typescript
// MemoryCore/src/core/tools/memory-search.ts
if (!vectorStore) {
  // No store → only fallback to static memory
}
const hasFts = vectorStore.isFtsAvailable();
if (vectorStore.getCapabilities().nativeHybridSearch && vectorStore.searchL1Hybrid) {
  // ✅ Native hybrid path
  results = await vectorStore.searchL1Hybrid(ftsQuery, queryEmbedding, candidateK, isolationFilter);
} else {
  // ❎ Separate searches with manual merge
  const ftsResults = await vectorStore.searchL1Fts(ftsQuery, candidateK);
  const vecResults = await vectorStore.searchL1Vector(queryEmbedding, candidateK, query);
  results = mergeRrf(ftsResults, vecResults);
}

```

This logic determines whether the system executes a single optimized query or branches into two separate searches requiring RRF reconciliation.

## Practical Debugging Examples

Use the Python SDK to test hybrid versus non-hybrid retrieval:

```python
from tencentdb_agent_memory import MemoryClient

client = MemoryClient(
    endpoint="https://memory.example.com",
    api_key="YOUR_API_KEY",
)

# Test hybrid search (BM25 + vector)

hybrid_response = client.retrieve_memory(
    query="How to reset my password?",
    retrieval_config={"hybrid": True, "candidateK": 20},
)

# Test BM25-only for comparison

bm25_response = client.retrieve_memory(
    query="How to reset my password?",
    retrieval_config={"hybrid": False, "candidateK": 20},
)

print(f"Hybrid hits: {len(hybrid_response['hits'])}")
print(f"BM25-only hits: {len(bm25_response['hits'])}")

```

Force pure BM25 in Node.js to isolate vector search issues:

```typescript
import { MemoryClient } from "tencentdb-agent-memory";

const client = new MemoryClient({ endpoint: "https://memory.example.com" });

const result = await client.retrieveMemory({
  query: "billing cycle explanation",
  retrievalConfig: { hybrid: false, candidateK: 10 }, // force BM25 only
});

console.log("BM25 results:", result.hits);

```

Inspect store capabilities directly in debug mode:

```typescript
// Inside a server extension or test harness
const store = core.getVectorStore();   // IMemoryStore instance
console.log("Hybrid supported:", store.getCapabilities().nativeHybridSearch);
console.log("FTS available:", store.isFtsAvailable());
console.log("Store degraded:", store.isDegraded());

```

## Summary

- Verify `vectorStore` initialization in [`pipeline-factory.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/pipeline-factory.ts) and check `isDegraded()` status before debugging retrieval logic.
- Confirm `isFtsAvailable()` returns `true` and `nativeHybridSearch` capability exists to enable optimized hybrid paths.
- Ensure `embeddingService` generates valid `queryEmbedding` vectors and the request payload includes both `ftsQuery` and embedding data.
- Review `isolationFilter` parameters for overly restrictive session or agent scoping that eliminates valid memories.
- Trace the RRF merge logic in [`memory-search.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/memory-search.ts) when results appear incomplete or incorrectly ranked.
- Use SDK `retrievalConfig` flags to toggle between hybrid, BM25-only, and vector-only modes for isolation testing.

## Frequently Asked Questions

### Why does hybrid search return fewer results than BM25 alone?

Hybrid search applies `isolationFilter` constraints and RRF merging that may eliminate duplicates or low-scoring vector matches. If the `candidateK` value is too small or the filter is overly specific, valid BM25 hits get pruned during the merge phase. Increase `candidateK` or relax the `isolationFilter` to include broader context.

### How do I know if my store supports native hybrid search?

Call `vectorStore.getCapabilities().nativeHybridSearch` and verify `vectorStore.searchL1Hybrid` is defined. According to the `IMemoryStore` interface in [`MemoryCore/src/core/types.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryCore/src/core/types.ts), native support requires both the capability flag and the hybrid method implementation. If missing, the system falls back to separate FTS5 and vector queries.

### What causes "vectorStore is undefined" errors during retrieval?

This occurs when the store initialization fails in [`pipeline-factory.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/pipeline-factory.ts) (around line 311) and the system enters degraded mode. Check that the SQLite database file (`vectors.db`) is writable, disk space is available, and the `vectorStore.init` method completed without errors in the startup logs.

### How can I distinguish between embedding failures and FTS5 failures?

Set `retrievalConfig: { hybrid: false }` to test BM25-only retrieval. If results return successfully, the issue lies in the embedding generation (`embeddingService.embed`) or vector store connectivity. Check [`tdai-core.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/tdai-core.ts) for embedding service initialization errors and verify the query text successfully converts to embeddings before the hybrid merge phase.