# How to Use Keystore to Handle Large In-Memory Indexes in FlexSearch

> Learn how to use the FlexSearch keystore option to handle massive in-memory indexes, expanding key limits to trillions without extra memory. Optimize your search performance today.

- Repository: [Nextapps GmbH/flexsearch](https://github.com/nextapps-de/flexsearch)
- Tags: best-practices
- Published: 2026-02-23

---

**Enable the `keystore` option in FlexSearch to expand the default 16 million key limit to billions or trillions of distinct terms without allocating extra memory per key.**

FlexSearch is a high-performance full-text search library that stores term and partial keys in an in-memory *keystore*. By default, this keystore supports up to 2²⁴ (approximately 16 million) distinct keys. When building indexes for massive datasets, you must configure the keystore to handle large in-memory indexes in FlexSearch by extending the address space.

## Understanding the FlexSearch Keystore Architecture

The keystore is the internal structure that maps distinct terms and partials to their posting lists. Understanding its limitations and extension mechanism is critical for scaling.

### Default Key Space Limitations

By default, the keystore can hold up to **2²⁴ distinct terms/partials** (approximately 16 million). This limit is hardcoded in the address space allocation within [`src/keystore.js`](https://github.com/nextapps-de/flexsearch/blob/main/src/keystore.js). When you attempt to index more unique terms than this limit allows, the index will fail to store additional keys.

### How Address Space Extension Works

The `keystore` configuration option accepts a value between **8 and 64**, representing the number of extra address bits to allocate. The total key space becomes **2^(24+bits)**:

- `keystore: 8` → 2³² keys (≈ 4.3 billion)
- `keystore: 16` → 2⁴⁰ keys (≈ 1 trillion)
- `keystore: 64` → 2⁸⁸ keys (theoretical upper bound)

According to the documentation in [`doc/keystore.md`](https://github.com/nextapps-de/flexsearch/blob/main/doc/keystore.md), enabling a larger keystore does not allocate additional RAM per key; it only expands the addressable range.

## Configuring Keystore for Large-Scale Indexes

Proper configuration ensures you can handle massive datasets without memory waste or performance degradation.

### Calculating Required Address Bits

To determine your `keystore` value:

1. Estimate your maximum distinct term count (including partials if using contextual indexes).
2. Calculate required bits: `bits = ceil(log2(estimated_terms) - 24)`.
3. Round up to the nearest valid option (8, 16, 32, or 64).

For most large-scale applications, `keystore: 8` (4.3 billion keys) provides sufficient headroom. Only extreme use cases require `keystore: 16` or higher.

### Recommended Configuration Settings

When enabling extended keystore support, combine it with performance optimizations:

```javascript
import FlexSearch from "flexsearch";

const index = new FlexSearch.Index({
  keystore: 16,        // 2^40 possible keys
  fastupdate: true,    // Reduces overhead during bulk inserts
  context: true        // Enables phrase searching
});

```

The `fastupdate` option is particularly important when handling large in-memory indexes, as it minimizes the cost of rebuilding posting lists after each insertion.

## Handling Massive Posting Lists

Beyond key count limits, individual terms may accumulate billions of document references. FlexSearch handles this through automatic scaling mechanisms.

### Automatic Proxy-Based Scaling

When a term’s posting list grows beyond **2³¹ entries**, FlexSearch automatically switches to a Proxy-based storage system. This is implemented in [`src/keystore.js`](https://github.com/nextapps-de/flexsearch/blob/main/src/keystore.js) and verified in [`test/keystore.js`](https://github.com/nextapps-de/flexsearch/blob/main/test/keystore.js).

The test suite artificially inflates posting lists to exceed the 2³¹ threshold:

```javascript
// From test/keystore.js
const index = new Index({
  fastupdate: true,
  keystore: 16,
  context: true
});

// Simulate massive posting list
let foo = index.map.get("fo");
foo[0].length = 2**31 - 10;

// Continue adding beyond 2^31
for (let i = foo[0].length; i <= 2**31 + 9; i++) {
  index.add(i, "foo bar");
}

```

This automatic scaling ensures you can exceed 2 billion documents per term without manual intervention or array overflow errors.

### Memory Management with cleanup()

After massive deletions or bulk updates, obsolete posting entries may remain in memory. The `cleanup()` method, available on index instances, prunes these obsolete postings and reduces the map size to the minimum required.

Call `cleanup()` after bulk removal operations:

```javascript
// After deleting millions of documents
index.cleanup();  // Reclaims memory from empty term buckets

```

## Code Examples

### Scaling to Hundreds of Millions of Terms

This example demonstrates configuring FlexSearch for extreme scale with 64-bit addressing:

```javascript
import FlexSearch from "flexsearch";

// 64-bit keystore → 2^88 possible keys (theoretical upper bound)
const index = new FlexSearch.Index({
  keystore: 64,
  fastupdate: true,
  context: true,
});

// Simulate indexing massive dataset
for (let i = 0; i < 10_000_000; i++) {
  index.add(i, `document number ${i} with many unique terms ${Math.random()}`);
}

// Compact after bulk load
index.cleanup();

```

**Key implementation details:**
- The `keystore` value represents **extra address bits**, not byte size.
- `fastupdate` maintains insertion performance at scale.
- `cleanup()` frees unused term slots after bulk operations.

### Testing Extended Limits

Verify your keystore configuration handles posting lists exceeding 2³¹ entries:

```javascript
import { Index } from "flexsearch";

const index = new Index({
  fastupdate: true,
  keystore: 16,  // 2^40 address space
  context: true
});

// Seed initial documents
index.add(0, "foo bar");
index.add(1, "foo bar");
index.add(2, "foo bar");

// Access internal posting lists (advanced usage)
let foo = index.map.get("fo");
let bar = index.ctx.get("fo").get("bar");

// Artificially inflate to near 2^31 limit
foo[0].length = 2**31 - 10;
bar[0].length = 2**31 - 10;

// Push beyond 2^31 to trigger Proxy-based storage
for (let i = foo[0].length; i <= 2**31 + 9; i++) {
  index.add(i, "foo bar");
}

console.log("Successfully stored >2^31 postings per term");

```

This test confirms that FlexSearch automatically switches to Proxy-based arrays when posting lists exceed 2³¹ entries, as implemented in [`src/keystore.js`](https://github.com/nextapps-de/flexsearch/blob/main/src/keystore.js) and validated in [`test/keystore.js`](https://github.com/nextapps-de/flexsearch/blob/main/test/keystore.js).

## Summary

- **Default limits**: FlexSearch supports 2²⁴ (≈16M) distinct terms by default, defined in the keystore address space.
- **Address extension**: Add extra bits via `keystore: 8` through `keystore: 64` to support billions or trillions of keys without per-key memory overhead.
- **Posting list scaling**: Automatic Proxy-based storage activates when individual terms exceed 2³¹ document references.
- **Performance optimization**: Combine `keystore` with `fastupdate: true` for bulk inserts and call `cleanup()` after deletions to reclaim memory.
- **Persistent storage incompatibility**: Do not enable `keystore` on SQLite, Redis, or other persistent backends, as they ignore the limit and may buffer excessively.

## Frequently Asked Questions

### What is the default key limit in FlexSearch?

By default, FlexSearch stores up to **2²⁴ distinct terms or partials** (approximately 16.7 million) in the in-memory keystore. This limit is hardcoded in the address space allocation within [`src/keystore.js`](https://github.com/nextapps-de/flexsearch/blob/main/src/keystore.js) and documented in [`doc/keystore.md`](https://github.com/nextapps-de/flexsearch/blob/main/doc/keystore.md). Once you exceed this threshold, the index cannot store additional unique keys unless you enable the extended keystore.

### Does enabling a larger keystore consume more RAM?

No. Enabling a larger keystore via the `keystore` option **does not allocate additional RAM per key**. According to the documentation in [`doc/keystore.md`](https://github.com/nextapps-de/flexsearch/blob/main/doc/keystore.md), the option only expands the *addressable range* by adding extra bits to the key space (2^(24+bits)). The memory overhead is negligible because the library does not pre-allocate space for the theoretical maximum; it merely allows the index to address more distinct terms as they are inserted.

### Can I use keystore with persistent storage backends?

No. The `keystore` option is designed exclusively for in-memory indexes. Persistent storage backends such as **SQLite, Redis, and other custom storages** ignore the keystore limit because they manage their own key spaces. Enabling `keystore` on a persistent index is unnecessary and may cause the internal buffer to stress before you call `index.commit()`, as noted in [`doc/keystore.md`](https://github.com/nextapps-de/flexsearch/blob/main/doc/keystore.md). Use `keystore` only when keeping the entire index in memory.

### How do I know if I need to increase the keystore size?

You need to increase the keystore size if you anticipate indexing **more than 16 million distinct terms or partials**. Calculate your requirements by estimating the cardinality of your vocabulary: if you expect billions of unique terms (common in large-scale log analysis or genomic data), set `keystore: 8` (4.3 billion keys) or higher. Monitor for insertion failures or memory errors during bulk indexing; if the index stops accepting new unique terms despite available system memory, you have likely hit the 2²⁴ default limit and must restart with a higher `keystore` value.