# How to Implement Pagination in OpenSearch Search Results: Offset vs. Deep Pagination Methods

> Implement OpenSearch pagination effectively using offset and deep pagination methods. Learn to use PIT with search_after for efficient large datasets.

- Repository: [AWS Samples/sample-for-amazon-opensearch-service-tutorials-101](https://github.com/aws-samples/sample-for-amazon-opensearch-service-tutorials-101)
- Tags: tutorial
- Published: 2026-02-25

---

**Use offset-based pagination with `from` and `size` for the first 10 pages, then switch to Point-in-Time (PIT) with `search_after` for deep pagination to maintain consistent results and avoid performance degradation.**

The `aws-samples/sample-for-amazon-opensearch-service-tutorials-101` repository provides a foundational Lambda function for querying OpenSearch, but it currently lacks robust pagination support for production workloads. While the sample code in [`artifacts/search_lambda/opensearch_search.py`](https://github.com/aws-samples/sample-for-amazon-opensearch-service-tutorials-101/blob/main/artifacts/search_lambda/opensearch_search.py) uses a static `"size": 100` parameter, real-world applications require efficient pagination strategies to handle result sets ranging from hundreds to millions of documents without degrading query performance or consistency.

## Understanding OpenSearch Pagination Strategies

OpenSearch offers three primary mechanisms for paginating search results, each optimized for different use cases and result set depths.

### Offset-Based Pagination (from/size)

The simplest approach uses the `from` and `size` parameters, where `from` specifies the starting offset and `size` determines the number of hits to return. This method works well for shallow pagination, such as displaying the first 100 pages of results in a web interface. However, OpenSearch must sort and retrieve all documents up to the `from` offset, making deep pagination (e.g., page 10,000) increasingly expensive in terms of memory and CPU.

### Point-in-Time with search_after (Deep Pagination)

For reliable deep pagination, OpenSearch recommends using a **Point-in-Time (PIT)** context combined with the `search_after` parameter. A PIT creates a frozen snapshot of the index state at creation time, ensuring that subsequent queries see a consistent view of the data regardless of concurrent indexing operations. The `search_after` parameter uses the sort values from the last hit of the previous page to fetch the next set of results, offering O(1) performance regardless of how deep you paginate.

### Scroll API (Batch Processing)

The Scroll API maintains a server-side cursor for retrieving large result sets in batches, typically used for background data export or reindexing operations. While functional for pagination, Scroll contexts consume significant server resources and expire after a configured timeout, making them unsuitable for interactive user-facing pagination.

## Current Implementation in the AWS Tutorial Repository

The existing Lambda implementation in [`artifacts/search_lambda/opensearch_search.py`](https://github.com/aws-samples/sample-for-amazon-opensearch-service-tutorials-101/blob/main/artifacts/search_lambda/opensearch_search.py) handles multiple query types—including multi-match, wildcard, and vector searches—but implements a static pagination approach. Examining the multi-match block (lines 99-108) and wildcard block (lines 114-124) reveals a hardcoded `"size": 100` without any `from`, `search_after`, or PIT handling:

```python

# Simplified excerpt from opensearch_search.py

query = {
    "size": 100,  # Static limit

    "query": {
        "multi_match": {
            "query": attribute_value,
            "fields": [attribute_name]
        }
    }
}

```

This configuration returns only the first 100 matches regardless of the total result set size, preventing users from accessing deeper results without modifying the source code.

## Implementing Hybrid Pagination in the Lambda Function

To support production-grade pagination, extend the Lambda to implement a hybrid strategy that uses offset-based pagination for shallow pages and PIT with `search_after` for deep pagination.

### Adding Pagination Parameters

First, modify the `search_products` function to accept pagination inputs from the API Gateway event. Define constants to control the pagination strategy threshold:

```python
PAGE_LIMIT = 10          # Switch to PIT after this page

DEFAULT_SIZE = 100       # Fallback page size

def search_products(event):
    body = json.loads(event.get("body", "{}"))
    
    # Extract pagination parameters

    page = int(body.get("page", 1))
    page_size = int(body.get("pageSize", DEFAULT_SIZE))
    pit_id = body.get("pitId")
    search_after = body.get("searchAfter")
    
    # ... rest of implementation

```

### Offset-Based Implementation (Pages ≤ 10)

For the first 10 pages, use the standard `from` parameter. Always include a deterministic `sort` clause to ensure consistent ordering:

```python
DEFAULT_SORT = [
    {"created_at": "desc"},
    {"_id": "desc"}
]

if page <= PAGE_LIMIT:
    # Standard offset pagination

    query_body = {
        "from": (page - 1) * page_size,
        "size": page_size,
        "sort": DEFAULT_SORT,
        "query": build_query(body)  # Your existing query logic

    }
    
    response = ops_client.search(index=INDEX_NAME, body=query_body)

```

### Deep Pagination with PIT (Pages > 10)

When requesting page 11 or beyond, switch to the PIT mechanism. On the first deep page request, create a new PIT context; for subsequent requests, reuse the provided `pitId` and `searchAfter` values:

```python
else:
    # Deep pagination requires Point-in-Time

    if not pit_id:
        # First request into deep pagination - create PIT

        pit = ops_client.open_point_in_time(
            index=INDEX_NAME, 
            keep_alive="1h"
        )
        pit_id = pit["id"]
        
        # Initial query without search_after

        query_body = {
            "size": page_size,
            "sort": DEFAULT_SORT,
            "pit": {"id": pit_id, "keep_alive": "1h"},
            "query": build_query(body)
        }
    else:
        # Subsequent deep page - use search_after

        query_body = {
            "size": page_size,
            "sort": DEFAULT_SORT,
            "pit": {"id": pit_id, "keep_alive": "1h"},
            "search_after": search_after,
            "query": build_query(body)
        }
    
    response = ops_client.search(body=query_body)
    
    # Extract sort values for next page

    hits = response.get("hits", {}).get("hits", [])
    if hits:
        last_sort = hits[-1].get("sort")
    else:
        last_sort = None
    
    # Append pagination metadata to response

    response["pagination"] = {
        "pitId": pit_id,
        "nextSearchAfter": last_sort,
        "pageSize": page_size,
        "currentPage": page
    }

```

### Client-Side Integration

Frontend applications must store the `pitId` and `nextSearchAfter` values between requests when paginating beyond the offset limit:

```javascript
async function fetchSearchResults(page) {
  const payload = {
    type: "multi_match",
    attribute_name: "title",
    attribute_value: "laptop",
    page: page,
    pageSize: 20
  };
  
  // For deep pages, include tokens from previous response
  if (page > 10 && paginationState.pitId) {
    payload.pitId = paginationState.pitId;
    payload.searchAfter = paginationState.nextSearchAfter;
  }
  
  const response = await fetch('/search', {
    method: 'POST',
    body: JSON.stringify(payload)
  });
  
  const data = await response.json();
  
  // Store tokens for next deep page
  if (data.result.pagination) {
    paginationState.pitId = data.result.pagination.pitId;
    paginationState.nextSearchAfter = data.result.pagination.nextSearchAfter;
  }
  
  return data.result.hits.hits;
}

```

## Key Files and Architecture

The pagination implementation extends the following components in the `aws-samples/sample-for-amazon-opensearch-service-tutorials-101` repository:

- **[`artifacts/search_lambda/opensearch_search.py`](https://github.com/aws-samples/sample-for-amazon-opensearch-service-tutorials-101/blob/main/artifacts/search_lambda/opensearch_search.py)** – Core Lambda handler where pagination logic, PIT creation, and `search_after` handling are implemented. The `search_products` function requires modification to parse pagination parameters and construct appropriate query bodies.

- **[`search_tutorials/opensearch_proxy_stack.py`](https://github.com/aws-samples/sample-for-amazon-opensearch-service-tutorials-101/blob/main/search_tutorials/opensearch_proxy_stack.py)** – CDK stack defining the Lambda execution role. Ensure the IAM policy includes the `es:OpenPointInTime` and `es:ClosePointInTime` permissions required for PIT operations.

- **[`README.md`](https://github.com/aws-samples/sample-for-amazon-opensearch-service-tutorials-101/blob/main/README.md)** – Documentation should be updated to describe the hybrid pagination API contract, including the `page`, `pageSize`, `pitId`, and `searchAfter` request parameters.

## Summary

- The tutorial repository currently uses a static `size: 100` limit without pagination support in [`opensearch_search.py`](https://github.com/aws-samples/sample-for-amazon-opensearch-service-tutorials-101/blob/main/opensearch_search.py).
- **Offset pagination** (`from` + `size`) works efficiently for the first 10 pages but degrades significantly for deep pagination due to global sorting overhead.
- **Point-in-Time (PIT) with `search_after`** provides consistent, O(1) performance for deep pagination by maintaining a stable index snapshot and using sort values as cursors.
- A **hybrid approach** automatically selects offset pagination for shallow pages and PIT for deep pages, balancing simplicity with scalability.
- Implementation requires extending the Lambda to handle `page`, `pageSize`, `pitId`, and `searchAfter` parameters while ensuring deterministic sorting across all queries.

## Frequently Asked Questions

### What is the maximum offset limit in OpenSearch?

OpenSearch defaults to a maximum `from` + `size` value of 10,000 hits (controlled by the `index.max_result_window` setting). While you can increase this limit, doing so consumes significant heap memory per shard and degrades performance. For result sets beyond 10,000 documents, you should use Point-in-Time with `search_after` or the Scroll API instead of increasing the result window.

### Why does deep pagination using from/size become slow?

When using `from` and `size` for deep pages, OpenSearch must sort and retrieve all documents from the beginning of the result set up to the specified offset, then discard the initial `from` number of hits. This means page 10,000 requires sorting and ranking 10,000+ documents per shard, even though only `size` documents are returned. The computational cost grows linearly with the offset, causing timeouts and high memory usage for deep pagination.

### How long does a Point-in-Time context remain valid?

A Point-in-Time (PIT) context remains valid for the duration specified in the `keep_alive` parameter when creating the PIT (e.g., `"keep_alive": "1h"`). OpenSearch automatically closes PITs after this TTL expires, freeing associated resources. Clients should treat PIT IDs as temporary session tokens and be prepared to handle `resource_not_found` exceptions if the PIT expires before the user finishes paginating, potentially requiring a restart of the search with a new PIT.

### Can I use the Scroll API for user-facing pagination?

No, the Scroll API is designed for batch processing and background data export, not for real-time user interfaces. Scroll contexts consume significant server-side resources (file handles and memory) to maintain the search context, and they are optimized for processing large result sets sequentially rather than supporting random page access. For user-facing pagination that requires jumping between pages or maintaining sort consistency, use Point-in-Time with `search_after` instead, which provides better resource efficiency and supports the search-after pattern needed for deep pagination.