# How BM25 Sparse Indexing Works in MemoryCore: Architecture and Implementation

> Discover how BM25 sparse indexing is implemented in MemoryCore. Learn about client-side encoding, weighted term vectors, and efficient document retrieval with Tencent Cloud VectorDB.

- Repository: [Tencent Cloud/TencentDB-Agent-Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory)
- Tags: architecture
- Published: 2026-08-22

---

**BM25 sparse indexing in MemoryCore works by encoding text client-side into weighted term vectors using `BM25LocalEncoder`, storing these sparse vectors in Tencent Cloud VectorDB with an inverted index, and retrieving documents via fast term-level matching or hybrid search with Reciprocal Rank Fusion.**

The **TencentDB-Agent-Memory** repository implements BM25 sparse indexing to enable efficient full-text retrieval without relying solely on dense embeddings. This approach generates sparse vectors on the client side and indexes them in Tencent Cloud VectorDB for high-performance term-based search operations.

## BM25 Sparse Indexing Architecture

MemoryCore’s sparse indexing architecture separates encoding responsibilities from storage. The system uses a **client-side BM25 encoder** to convert raw text into sparse vectors—arrays of `[termId, weight]` pairs—which are then persisted in a specialized VectorDB collection configured for inverted index searches.

The implementation resides primarily in [`MemoryCore/src/core/store/tcvdb.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryCore/src/core/store/tcvdb.ts), with encoding logic encapsulated in [`MemoryCore/src/core/store/bm25-local.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryCore/src/core/store/bm25-local.ts). When dense embeddings are disabled, the system operates in pure sparse mode, using a dummy dense vector (`[1]`) only to satisfy API requirements while ignoring it during retrieval.

## Step-by-Step Implementation

### 1. Collection Schema Configuration

When initializing a collection, MemoryCore defines a **`sparse_vector`** field with type **`sparseVector`** and configures an **inverted index** using the Inner Product (IP) metric. This schema enables fast term-level lookups essential for BM25 retrieval.

In [`tcvdb.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/tcvdb.ts), the `_createCollectionWithVectorFallback` method establishes this schema:

```typescript
{
  fieldName: "sparse_vector",
  fieldType: "sparseVector",
  indexType: "inverted",
  metricType: "IP"
}

```

The inverted index structure allows VectorDB to efficiently match query terms against document term vectors without computing dense vector similarities.

### 2. Client-Side Text Encoding

The **`BM25LocalEncoder`** class handles all text-to-sparse-vector transformations. During both insertion and querying, the encoder generates BM25-weighted term vectors locally before transmitting them to the server.

For document insertion, the `upsertL1Batch` method invokes:

```typescript
const sparse = await this.bm25Encoder.encodeTexts([record.content]);
doc.sparse_vector = sparse[0];

```

For search queries, the system calls `bm25Encoder.encodeQueries([queryText])` to generate compatible sparse query vectors. This client-side approach reduces server computational load and allows custom BM25 parameter tuning without altering the VectorDB configuration.

### 3. Upserting Documents with Sparse Vectors

During the upsert operation in `upsertL1`, MemoryCore attaches the generated sparse vector to the document object. If dense embedding is disabled (indicated by `embeddingEnabled: false`), the system stores the sparse vector alongside a placeholder dense vector to maintain API compatibility.

The critical assignment occurs at:

```typescript
doc.sparse_vector = sparse[0];

```

This line in [`tcvdb.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/tcvdb.ts) (lines 94–98) ensures the BM25 vector persists in the `sparse_vector` field, making the document discoverable via term-level searches.

### 4. Search and Retrieval

MemoryCore supports both pure-sparse and hybrid retrieval modes. When `search` is invoked, the query text undergoes BM25 encoding, then the store executes **`hybridSearch`** (or a sparse-only variant) to retrieve results.

The search implementation combines sparse vectors with optional dense embeddings and applies **Reciprocal Rank Fusion (RRF)** for reranking when both modalities are present. The search payload specifies:

```typescript
fieldName: "sparse_vector",
data: [sparseVec]

```

This configuration, found in [`tcvdb.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/tcvdb.ts) (lines 1016–1045), directs VectorDB to match the query’s sparse vector against the inverted index, returning documents ranked by BM25 relevance scores.

### 5. Document Deletion

Deleting documents automatically removes their associated sparse vectors without requiring manual cleanup. The VectorDB store handles cascade deletion internally, ensuring index consistency when records are removed via the delete API.

## Code Implementation Example

The following example demonstrates initializing a pure-BM25 store and performing sparse-only retrieval:

```typescript
import { TcvdbMemoryStore } from "./MemoryCore/src/core/store/tcvdb";
import { BM25LocalEncoder } from "./MemoryCore/src/core/store/bm25-local";

// Initialize the store with client-side BM25 encoding
const store = new TcvdbMemoryStore({
  url: "https://vdb.example.com",
  username: "admin",
  apiKey: "******",
  database: "memory",
  embeddingEnabled: false,               // Pure sparse mode
  bm25Encoder: new BM25LocalEncoder(),   // Client-side encoder
});

// Upsert a record (sparse vector generated automatically)
await store.upsertL1({
  id: "msg-001",
  content: "How do I reset my password?",
  createdAt: new Date(),
  userId: "user-123"
});

// Perform sparse-only search
const results = await store.search({
  queryText: "reset password steps",
  topK: 5,
});
console.log(results);

```

## Key Configuration Options

MemoryCore’s sparse indexing behavior is controlled through configuration flags in [`MemoryCore/src/config.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryCore/src/config.ts):

- **`bm25Encoder`**: Accepts an instance of `BM25LocalEncoder` or `BM25Client` (for remote encoding via sidecar), determining where BM25 computation occurs.
- **`embeddingEnabled`**: When set to `false`, the system skips dense vector generation and relies exclusively on the `sparse_vector` field for retrieval, storing only the dummy vector `[1]` for API compliance.

According to the TencentDB-Agent-Memory source code, these settings determine whether the store invokes `hybridSearch` (combining dense and sparse signals) or routes queries through pure sparse vector matching.

## Summary

- **BM25 sparse indexing** in MemoryCore uses client-side encoding via `BM25LocalEncoder` to generate term-weight vectors.
- Documents store sparse vectors in a dedicated **`sparse_vector`** field indexed with an **inverted index** and **IP metric**.
- The **`upsertL1`** and **`upsertL1Batch`** methods in [`tcvdb.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/tcvdb.ts) handle sparse vector persistence during document insertion.
- **Hybrid search** combines BM25 sparse retrieval with dense embeddings using Reciprocal Rank Fusion when both are enabled.
- Pure sparse mode operates without dense embeddings by utilizing a dummy vector placeholder, controlled via the `embeddingEnabled` configuration flag.

## Frequently Asked Questions

### How does MemoryCore generate BM25 sparse vectors?

MemoryCore generates BM25 sparse vectors using the `BM25LocalEncoder` class located in [`MemoryCore/src/core/store/bm25-local.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryCore/src/core/store/bm25-local.ts). This client-side encoder converts raw text into arrays of `[termId, weight]` pairs by analyzing term frequency and inverse document frequency. The encoder is invoked during both document insertion (`encodeTexts`) and search queries (`encodeQueries`) to ensure consistent vector representations.

### What is the purpose of the inverted index in BM25 sparse indexing?

The inverted index, configured with `indexType: "inverted"` and `metricType: "IP"` on the `sparse_vector` field, enables fast term-level matching without scanning entire document collections. According to the implementation in [`tcvdb.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/tcvdb.ts), this index structure allows Tencent Cloud VectorDB to efficiently retrieve documents containing specific query terms and compute relevance scores using inner product calculations between query and document term vectors.

### Can MemoryCore use BM25 sparse indexing without dense embeddings?

Yes, MemoryCore supports pure BM25 sparse indexing by setting `embeddingEnabled: false` in the store configuration. In this mode, the system stores only the sparse vectors generated by `BM25LocalEncoder` and uses a dummy dense vector (`[1]`) solely to satisfy VectorDB API requirements. The search operation then relies exclusively on term-level matching through the `sparse_vector` field’s inverted index.

### How does hybrid search combine BM25 sparse vectors with dense embeddings?

When both sparse and dense embeddings are enabled, MemoryCore’s `hybridSearch` method (implemented in [`tcvdb.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/tcvdb.ts)) retrieves results using both modalities separately, then applies **Reciprocal Rank Fusion (RRF)** to rerank and combine the results. This fusion algorithm balances lexical matching (BM25 sparse vectors) with semantic similarity (dense embeddings) to improve retrieval accuracy across different query types.