How BM25 Sparse Indexing Works in MemoryCore: Architecture and Implementation

BM25 sparse indexing in MemoryCore works by encoding text client-side into weighted term vectors using BM25LocalEncoder, storing these sparse vectors in Tencent Cloud VectorDB with an inverted index, and retrieving documents via fast term-level matching or hybrid search with Reciprocal Rank Fusion.

The TencentDB-Agent-Memory repository implements BM25 sparse indexing to enable efficient full-text retrieval without relying solely on dense embeddings. This approach generates sparse vectors on the client side and indexes them in Tencent Cloud VectorDB for high-performance term-based search operations.

BM25 Sparse Indexing Architecture

MemoryCore’s sparse indexing architecture separates encoding responsibilities from storage. The system uses a client-side BM25 encoder to convert raw text into sparse vectors—arrays of [termId, weight] pairs—which are then persisted in a specialized VectorDB collection configured for inverted index searches.

The implementation resides primarily in MemoryCore/src/core/store/tcvdb.ts, with encoding logic encapsulated in MemoryCore/src/core/store/bm25-local.ts. When dense embeddings are disabled, the system operates in pure sparse mode, using a dummy dense vector ([1]) only to satisfy API requirements while ignoring it during retrieval.

Step-by-Step Implementation

1. Collection Schema Configuration

When initializing a collection, MemoryCore defines a sparse_vector field with type sparseVector and configures an inverted index using the Inner Product (IP) metric. This schema enables fast term-level lookups essential for BM25 retrieval.

In tcvdb.ts, the _createCollectionWithVectorFallback method establishes this schema:

{
  fieldName: "sparse_vector",
  fieldType: "sparseVector",
  indexType: "inverted",
  metricType: "IP"
}

The inverted index structure allows VectorDB to efficiently match query terms against document term vectors without computing dense vector similarities.

2. Client-Side Text Encoding

The BM25LocalEncoder class handles all text-to-sparse-vector transformations. During both insertion and querying, the encoder generates BM25-weighted term vectors locally before transmitting them to the server.

For document insertion, the upsertL1Batch method invokes:

const sparse = await this.bm25Encoder.encodeTexts([record.content]);
doc.sparse_vector = sparse[0];

For search queries, the system calls bm25Encoder.encodeQueries([queryText]) to generate compatible sparse query vectors. This client-side approach reduces server computational load and allows custom BM25 parameter tuning without altering the VectorDB configuration.

3. Upserting Documents with Sparse Vectors

During the upsert operation in upsertL1, MemoryCore attaches the generated sparse vector to the document object. If dense embedding is disabled (indicated by embeddingEnabled: false), the system stores the sparse vector alongside a placeholder dense vector to maintain API compatibility.

The critical assignment occurs at:

doc.sparse_vector = sparse[0];

This line in tcvdb.ts (lines 94–98) ensures the BM25 vector persists in the sparse_vector field, making the document discoverable via term-level searches.

4. Search and Retrieval

MemoryCore supports both pure-sparse and hybrid retrieval modes. When search is invoked, the query text undergoes BM25 encoding, then the store executes hybridSearch (or a sparse-only variant) to retrieve results.

The search implementation combines sparse vectors with optional dense embeddings and applies Reciprocal Rank Fusion (RRF) for reranking when both modalities are present. The search payload specifies:

fieldName: "sparse_vector",
data: [sparseVec]

This configuration, found in tcvdb.ts (lines 1016–1045), directs VectorDB to match the query’s sparse vector against the inverted index, returning documents ranked by BM25 relevance scores.

5. Document Deletion

Deleting documents automatically removes their associated sparse vectors without requiring manual cleanup. The VectorDB store handles cascade deletion internally, ensuring index consistency when records are removed via the delete API.

Code Implementation Example

The following example demonstrates initializing a pure-BM25 store and performing sparse-only retrieval:

import { TcvdbMemoryStore } from "./MemoryCore/src/core/store/tcvdb";
import { BM25LocalEncoder } from "./MemoryCore/src/core/store/bm25-local";

// Initialize the store with client-side BM25 encoding
const store = new TcvdbMemoryStore({
  url: "https://vdb.example.com",
  username: "admin",
  apiKey: "******",
  database: "memory",
  embeddingEnabled: false,               // Pure sparse mode
  bm25Encoder: new BM25LocalEncoder(),   // Client-side encoder
});

// Upsert a record (sparse vector generated automatically)
await store.upsertL1({
  id: "msg-001",
  content: "How do I reset my password?",
  createdAt: new Date(),
  userId: "user-123"
});

// Perform sparse-only search
const results = await store.search({
  queryText: "reset password steps",
  topK: 5,
});
console.log(results);

Key Configuration Options

MemoryCore’s sparse indexing behavior is controlled through configuration flags in MemoryCore/src/config.ts:

• bm25Encoder: Accepts an instance of BM25LocalEncoder or BM25Client (for remote encoding via sidecar), determining where BM25 computation occurs.
• embeddingEnabled: When set to false, the system skips dense vector generation and relies exclusively on the sparse_vector field for retrieval, storing only the dummy vector [1] for API compliance.

According to the TencentDB-Agent-Memory source code, these settings determine whether the store invokes hybridSearch (combining dense and sparse signals) or routes queries through pure sparse vector matching.

Summary

• BM25 sparse indexing in MemoryCore uses client-side encoding via BM25LocalEncoder to generate term-weight vectors.
• Documents store sparse vectors in a dedicated sparse_vector field indexed with an inverted index and IP metric.
• The upsertL1 and upsertL1Batch methods in tcvdb.ts handle sparse vector persistence during document insertion.
• Hybrid search combines BM25 sparse retrieval with dense embeddings using Reciprocal Rank Fusion when both are enabled.
• Pure sparse mode operates without dense embeddings by utilizing a dummy vector placeholder, controlled via the embeddingEnabled configuration flag.

Frequently Asked Questions

How does MemoryCore generate BM25 sparse vectors?

MemoryCore generates BM25 sparse vectors using the BM25LocalEncoder class located in MemoryCore/src/core/store/bm25-local.ts. This client-side encoder converts raw text into arrays of [termId, weight] pairs by analyzing term frequency and inverse document frequency. The encoder is invoked during both document insertion (encodeTexts) and search queries (encodeQueries) to ensure consistent vector representations.

What is the purpose of the inverted index in BM25 sparse indexing?

The inverted index, configured with indexType: "inverted" and metricType: "IP" on the sparse_vector field, enables fast term-level matching without scanning entire document collections. According to the implementation in tcvdb.ts, this index structure allows Tencent Cloud VectorDB to efficiently retrieve documents containing specific query terms and compute relevance scores using inner product calculations between query and document term vectors.

Can MemoryCore use BM25 sparse indexing without dense embeddings?

Yes, MemoryCore supports pure BM25 sparse indexing by setting embeddingEnabled: false in the store configuration. In this mode, the system stores only the sparse vectors generated by BM25LocalEncoder and uses a dummy dense vector ([1]) solely to satisfy VectorDB API requirements. The search operation then relies exclusively on term-level matching through the sparse_vector field’s inverted index.

How does hybrid search combine BM25 sparse vectors with dense embeddings?

When both sparse and dense embeddings are enabled, MemoryCore’s hybridSearch method (implemented in tcvdb.ts) retrieves results using both modalities separately, then applies Reciprocal Rank Fusion (RRF) to rerank and combine the results. This fusion algorithm balances lexical matching (BM25 sparse vectors) with semantic similarity (dense embeddings) to improve retrieval accuracy across different query types.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}