How to Implement Pagination with Offset/Limit in FlexSearch

FlexSearch supports native pagination via the offset and limit options in the search() method, which are processed directly in the intersection engine to achieve O(limit + offset) complexity without materializing the full result set.

Implementing efficient pagination with offset/limit in FlexSearch requires understanding how the library processes queries internally. The nextapps-de/flexsearch repository provides native pagination support that integrates directly into the intersection engine, allowing you to retrieve specific page ranges without loading the entire hit list into memory.

How FlexSearch Handles Pagination Internally

FlexSearch implements pagination through three coordinated stages in the search pipeline. Each stage is optimized to minimize memory allocation and CPU cycles when handling paginated queries.

API Entry Point and Option Normalization

The pagination logic begins in src/index/search.js at lines 55-67, within Index.prototype.search. This method normalizes the search options, extracting offset (defaulting to 0) and limit (defaulting to 0, meaning no hard cap) before passing them down the pipeline.

// The search method signature accepts pagination options directly
const results = index.search("query", {
  offset: 20,  // Skip first 20 results
  limit: 10    // Return next 10 results
});

When the legacy positional argument limit is used, it is converted to the options object format internally to maintain backward compatibility while ensuring the pagination logic remains consistent.

Early Exit in the Intersection Engine

The core performance optimization occurs in src/intersect.js, where the intersect and union functions handle result set merging. According to the source code at lines 84-92, the engine implements an early termination check: if (limit && count === length-1 && tmp.length-offset === limit). As soon as the requested page size is satisfied, the loop returns immediately, preventing unnecessary scanning of remaining posting lists.

The offset handling at lines 118-124 and 206-214 (in union) decrements the offset counter while iterating, discarding IDs on-the-fly. This means no separate "skip-first-N" pass is required, keeping the operation at O(limit + offset) rather than O(total hits).

Result Aggregation and Resolution Buckets

After term-wise intersection completes, the return_result function (referenced at lines 60-71 in src/index/search.js) applies the final slice. When working with resolution buckets (different relevance ranks), the engine first selects the bucket containing the final result (result[result_len-1]) before applying offset and limit. This resolution-aware slicing prevents materializing the full result array for large indexes.

Performance Characteristics of FlexSearch Pagination

FlexSearch’s pagination architecture provides several key performance advantages over client-side slicing:

• Early Termination: The intersection loop stops immediately once limit items are collected after applying offset, avoiding work on remaining posting lists when the requested page is near the beginning of the result set.
• Configurable Resolver Overhead: When SUPPORT_RESOLVER is disabled (resolve: false), FlexSearch returns plain ID arrays that are cheaper to slice. When the resolver is enabled, slicing occurs after the resolver produces final enriched objects, preserving the same asymptotic cost.
• Memory Efficiency: Because offset handling occurs during iteration rather than as a post-processing filter, the engine never allocates arrays for the skipped portion of the result set.

Implementation Examples

Basic Pagination with Offset and Limit

Use the options object to specify page boundaries. Calculate offset as (page - 1) * pageSize:

import FlexSearch from "flexsearch";

const index = new FlexSearch({
  encode: "icase",
  tokenize: "forward",
  threshold: 0,
  resolution: 9
});

// Add documents
index.add(1, "The quick brown fox jumps over the lazy dog");
index.add(2, "FlexSearch provides fast full-text search");
index.add(3, "Pagination with offset and limit is efficient");

// Retrieve page 2 with 10 items per page
const pageSize = 10;
const page = 2;
const results = index.search("search", {
  limit: pageSize,
  offset: (page - 1) * pageSize
});

Async Pagination with Persistent Backend

When using a persistent backend like SQLite, pagination works identically through the async interface:

// Assuming index configured with db: new FlexSearch.SqliteDB(...)
const asyncResults = await index.search("query", {
  limit: 5,
  offset: 10,
  async: true
});

Pagination with Document Resolution

To retrieve full documents instead of raw IDs while paginating:

// Store full objects
index.add(10, { title: "FlexSearch Guide", body: "High-performance search" });

// Search with resolver enabled
const enriched = index.search("guide", {
  limit: 3,
  offset: 0,
  resolve: true
});

console.log(enriched[0].title); // "FlexSearch Guide"

Summary

• Native Pipeline Integration: FlexSearch processes offset and limit directly in src/index/search.js and src/intersect.js, avoiding the need for manual result slicing.
• Linear Complexity: Pagination operates in O(limit + offset) time by decrementing counters during iteration and implementing early exit logic.
• Zero-allocation Skipping: The engine discards offset items on-the-fly during intersection/union operations without allocating memory for skipped results.
• Resolver Compatibility: Pagination works transparently with both raw ID results (resolve: false) and enriched document results (resolve: true).

Frequently Asked Questions

Does FlexSearch support cursor-based pagination?

No, FlexSearch implements traditional offset/limit pagination rather than cursor-based (seek-based) pagination. The offset parameter specifies how many results to skip from the beginning of the result set, which is processed efficiently in the intersection engine at src/intersect.js by decrementing counters during iteration rather than creating intermediate arrays.

What is the performance cost of using a large offset value?

While FlexSearch avoids materializing the full result set, very large offset values still require the intersection engine to traverse posting lists until the offset count is satisfied. This results in O(offset + limit) complexity, meaning deep pagination (e.g., offset 100,000) will be slower than shallow pagination, though still more efficient than retrieving all results and slicing them client-side.

Can I use pagination with boolean (union) queries?

Yes, pagination works with both intersection (AND) and union (OR) queries. The union function in src/intersect.js (lines 206-214) handles offset decrementing and limit checking during the merge operation, ensuring that pagination constraints are applied after the boolean logic resolves.

How does the resolution setting affect pagination?

The resolution setting controls scoring precision by creating relevance buckets. When paginating, return_result in src/index/search.js selects the appropriate resolution bucket before applying offset/limit, meaning pagination respects relevance ranking. Higher resolution values create more granular buckets but do not increase the computational cost of pagination itself.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}