How to Debug Memory Retrieval Issues with BM25 and Vector Hybrid Search

To debug BM25 and vector hybrid search failures in TencentDB Agent Memory, verify that the vector store initialized correctly with FTS5 support, confirm nativeHybridSearch capability is enabled, and trace the query through embedding generation, isolation filtering, and Reciprocal Rank Fusion (RRF) merging using debug logs in memory-search.ts.

The TencentDB Agent Memory service retrieves relevant conversational memories using a hybrid pipeline that combines BM25 full-text search (FTS5) with dense vector similarity. This multi-layered architecture (L3 → L2 → L1 → L0) falls back from fast keyword matches to computationally expensive embedding searches when necessary. When retrieval fails or returns irrelevant results, developers must trace the execution path from store initialization through query construction to result merging.

Verify Vector Store Initialization and Capabilities

First, ensure the underlying storage layer is healthy. The pipeline factory initializes the SQLite/FAISS store in MemoryCore/src/utils/pipeline-factory.ts around lines 311-354.

Check Store Initialization and Degraded Mode

If vectorStore.init fails, the system logs Store init failed… and operates in a degraded state where hybrid search is unavailable. Call vectorStore.isDegraded() to verify operational status before attempting retrieval.

Confirm FTS5 Index Availability

The hybrid path requires the FTS5 keyword index. In MemoryCore/src/core/tools/memory-search.ts at line 127, the code checks vectorStore.isFtsAvailable(). If this returns false, the system falls back to pure embedding search, bypassing BM25 entirely.

Validate Native Hybrid Search Support

For optimized retrieval, the store must advertise nativeHybridSearch and provide searchL1Hybrid / searchL0Hybrid methods. As implemented in memory-search.ts lines 149-151, absence of this capability forces the system to execute separate FTS5 and vector calls, merging results manually with RRF.

Debug Query Construction and Embedding Generation

Verify Embedding Service Connectivity

The embeddingService.embed call in MemoryCore/src/core/tdai-core.ts must successfully convert query text into embeddings. A missing or failed embedding aborts the vector branch entirely, causing the system to rely solely on BM25 if available.

Inspect Query Payload Structure

In MemoryCore/src/core/tools/conversation-search.ts lines 146-148, the hybrid query requires both ftsQuery (for BM25) and queryEmbedding fields. Ensure the request payload includes both the query text and optional ftQuery parameters to populate both search branches.

Review Isolation Filters

Hybrid search can be scoped by isolationFilter parameters such as session-id or agent-id. In memory-search.ts lines 151-154, overly strict filters prune valid results before the RRF merge. Verify filter values passed from the caller match expected memory boundaries and are not excessively restrictive.

Analyze Result Merging and Search Paths

Monitor Path Selection and Logging

The pipeline logs the chosen retrieval path (hybrid, embedding, or fts) at line 111 in memory-search.ts. Increase log verbosity to logger.debug to see exact branches taken during query execution and identify unexpected fallbacks.

Debug Reciprocal Rank Fusion Logic

After BM25 and vector branches return results, the system applies Reciprocal Rank Fusion (RRF). The mergeRrf implementation (lines 197-233 in memory-search.ts) handles scoring normalization and duplicate removal. Incorrect RRF weights or deduplication bugs can suppress valid hits or alter ranking order.

Common Failure Scenarios and Solutions

No results when BM25 alone works: This indicates nativeHybridSearch is disabled or searchL1Hybrid is missing. Upgrade to a store version supporting native hybrid (SQLite FTS5 + vector extensions) or disable hybrid mode by setting hybrid: false in retrievalConfig.

Too few results returned: An overly restrictive isolationFilter or insufficient candidateK value limits the candidate pool. Increase candidateK or relax filter constraints in the retrieval request to allow more candidates into the merge phase.

Wrong relevance ordering: RRF merging bugs or missing score normalization skew final rankings. Add debug prints of individual branch scores before the mergeRrf call in memory-search.ts to verify BM25 and vector scores are normalized correctly.

Vector store undefined errors: The store failed to initialize and entered degraded mode. Check startup logs in pipeline-factory.ts line 311 and verify the SQLite file (vectors.db) is writable and accessible at the configured path.

High latency on retrieval: Falling back to separate FTS and vector calls instead of native hybrid search doubles query time. Enable native hybrid support or pre-index vectors with FTS5 using vectorStore.enableHybrid() to use single-pass searchL1Hybrid.

Hybrid Search Implementation Details

The core retrieval function searchMemory in MemoryCore/src/core/tools/memory-search.ts selects the strategy based on capability detection:

// MemoryCore/src/core/tools/memory-search.ts
if (!vectorStore) {
  // No store → only fallback to static memory
}
const hasFts = vectorStore.isFtsAvailable();
if (vectorStore.getCapabilities().nativeHybridSearch && vectorStore.searchL1Hybrid) {
  // ✅ Native hybrid path
  results = await vectorStore.searchL1Hybrid(ftsQuery, queryEmbedding, candidateK, isolationFilter);
} else {
  // ❎ Separate searches with manual merge
  const ftsResults = await vectorStore.searchL1Fts(ftsQuery, candidateK);
  const vecResults = await vectorStore.searchL1Vector(queryEmbedding, candidateK, query);
  results = mergeRrf(ftsResults, vecResults);
}

This logic determines whether the system executes a single optimized query or branches into two separate searches requiring RRF reconciliation.

Practical Debugging Examples

Use the Python SDK to test hybrid versus non-hybrid retrieval:

from tencentdb_agent_memory import MemoryClient

client = MemoryClient(
    endpoint="https://memory.example.com",
    api_key="YOUR_API_KEY",
)

# Test hybrid search (BM25 + vector)

hybrid_response = client.retrieve_memory(
    query="How to reset my password?",
    retrieval_config={"hybrid": True, "candidateK": 20},
)

# Test BM25-only for comparison

bm25_response = client.retrieve_memory(
    query="How to reset my password?",
    retrieval_config={"hybrid": False, "candidateK": 20},
)

print(f"Hybrid hits: {len(hybrid_response['hits'])}")
print(f"BM25-only hits: {len(bm25_response['hits'])}")

Force pure BM25 in Node.js to isolate vector search issues:

import { MemoryClient } from "tencentdb-agent-memory";

const client = new MemoryClient({ endpoint: "https://memory.example.com" });

const result = await client.retrieveMemory({
  query: "billing cycle explanation",
  retrievalConfig: { hybrid: false, candidateK: 10 }, // force BM25 only
});

console.log("BM25 results:", result.hits);

Inspect store capabilities directly in debug mode:

// Inside a server extension or test harness
const store = core.getVectorStore();   // IMemoryStore instance
console.log("Hybrid supported:", store.getCapabilities().nativeHybridSearch);
console.log("FTS available:", store.isFtsAvailable());
console.log("Store degraded:", store.isDegraded());

Summary

• Verify vectorStore initialization in pipeline-factory.ts and check isDegraded() status before debugging retrieval logic.
• Confirm isFtsAvailable() returns true and nativeHybridSearch capability exists to enable optimized hybrid paths.
• Ensure embeddingService generates valid queryEmbedding vectors and the request payload includes both ftsQuery and embedding data.
• Review isolationFilter parameters for overly restrictive session or agent scoping that eliminates valid memories.
• Trace the RRF merge logic in memory-search.ts when results appear incomplete or incorrectly ranked.
• Use SDK retrievalConfig flags to toggle between hybrid, BM25-only, and vector-only modes for isolation testing.

Frequently Asked Questions

Why does hybrid search return fewer results than BM25 alone?

Hybrid search applies isolationFilter constraints and RRF merging that may eliminate duplicates or low-scoring vector matches. If the candidateK value is too small or the filter is overly specific, valid BM25 hits get pruned during the merge phase. Increase candidateK or relax the isolationFilter to include broader context.

How do I know if my store supports native hybrid search?

Call vectorStore.getCapabilities().nativeHybridSearch and verify vectorStore.searchL1Hybrid is defined. According to the IMemoryStore interface in MemoryCore/src/core/types.ts, native support requires both the capability flag and the hybrid method implementation. If missing, the system falls back to separate FTS5 and vector queries.

What causes "vectorStore is undefined" errors during retrieval?

This occurs when the store initialization fails in pipeline-factory.ts (around line 311) and the system enters degraded mode. Check that the SQLite database file (vectors.db) is writable, disk space is available, and the vectorStore.init method completed without errors in the startup logs.

How can I distinguish between embedding failures and FTS5 failures?

Set retrievalConfig: { hybrid: false } to test BM25-only retrieval. If results return successfully, the issue lies in the embedding generation (embeddingService.embed) or vector store connectivity. Check tdai-core.ts for embedding service initialization errors and verify the query text successfully converts to embeddings before the hybrid merge phase.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}