# How to Debug Bulk Indexing Errors and Handle Partial Failures in OpenSearch

> Debug OpenSearch bulk indexing errors by inspecting response items and implementing targeted retry logic for partial failures. Handle individual document failures efficiently.

- Repository: [AWS Samples/sample-for-amazon-opensearch-service-tutorials-101](https://github.com/aws-samples/sample-for-amazon-opensearch-service-tutorials-101)
- Tags: how-to-guide
- Published: 2026-02-25

---

**Bulk indexing errors in OpenSearch can be debugged by inspecting the `items` array in the response to identify specific failed documents, then implementing targeted retry logic that resubmits only the failed operations rather than aborting the entire batch.**

The `aws-samples/sample-for-amazon-opensearch-service-tutorials-101` repository demonstrates bulk indexing patterns for both standard and vector-based document ingestion. Understanding how to debug bulk indexing errors is critical because OpenSearch bulk operations can partially succeed—some documents may index correctly while others fail due to mapping conflicts, malformed data, or transient cluster issues.

## Understanding Bulk Indexing Responses in OpenSearch

When you submit a bulk request to OpenSearch, the response contains two critical fields: an `errors` boolean flag and an `items` array. The `errors` flag indicates whether any operation in the batch failed, while the `items` array provides granular details about each individual action.

In [`artifacts/index_lambda/opensearch_index.py`](https://github.com/aws-samples/sample-for-amazon-opensearch-service-tutorials-101/blob/main/artifacts/index_lambda/opensearch_index.py), the `bulk_index_documents` function (lines 20-56) currently implements a basic error check:

```python
response = ops_client.bulk(body=bulk_data)
if response.get("errors"):
    return failure_response(f"Bulk indexing errors: {response}")

```

This implementation treats any error as a total failure, returning immediately when `response["errors"]` is `True`. However, this masks the reality that bulk requests are often **partially successful**—OpenSearch may have successfully indexed 499 documents while rejecting one malformed document.

## Why Partial Failures Require Granular Inspection

Partial failures occur when individual operations within a bulk request fail independently. Common causes include:

- **Mapping conflicts**: Attempting to index a string into a numeric field
- **Document too large**: Exceeding the `http.max_content_length` setting
- **Version conflicts**: Optimistic concurrency control failures
- **Transient errors**: Temporary node unavailability or throttling

The current code in `vectorize_and_index_products` (lines 80-90 and 146-155) handles two separate bulk payloads—one for on-disk vectors and one for in-memory vectors—but uses the same coarse error checking:

```python
if bulk_data_on_disk:
    response = ops_client.bulk(body=bulk_data_on_disk)
    if response.get("errors"):
        return failure_response(f"Bulk indexing errors: {response}")

```

To properly debug bulk indexing errors, you must inspect the `items` array to identify which specific documents failed and why.

## Implementing Detailed Error Inspection

### Parsing the Items Array

Each element in the `items` array corresponds to the operation at the same index in the bulk payload. The structure includes the operation type (`index`, `create`, `update`, or `delete`), the `_id`, `status` code, and an `error` object if the operation failed.

Here is an enhanced helper function that extracts failed items and maps them back to source documents:

```python
def _extract_failed_items(response, original_docs):
    """Map OpenSearch error items back to the original document dicts."""
    failed = []
    # `items` preserves the order of the bulk payload

    for idx, item in enumerate(response.get("items", [])):
        # Each item has a single operation key, e.g. "index"

        op = next(iter(item.values()))
        if op.get("status", 200) >= 400:
            failed.append(original_docs[idx])
            LOG.error(
                "Bulk item failed – id=%s status=%s error=%s",
                op.get("_id"),
                op.get("status"),
                op.get("error"),
            )
    return failed

```

### Mapping Errors Back to Source Documents

To effectively debug bulk indexing errors, maintain a reference to the original document when building the bulk payload. This allows you to log the complete content of failed documents for forensic analysis:

```python

# When building the bulk payload, preserve the original doc reference

bulk_payload = []
original_mapping = []  # Parallel array to map back to source

for doc in documents:
    bulk_payload.append(
        {"index": {"_index": INDEX_NAME, "_id": doc["id"]}}
    )
    bulk_payload.append(doc)
    original_mapping.append(doc)

```

## Building a Resilient Retry Mechanism

### Retry Only Failed Documents

Rather than resubmitting the entire batch, implement logic that retries only the failed subset. This reduces load on the OpenSearch cluster and prevents unnecessary re-indexing of already-successful documents:

```python
import time

MAX_RETRIES = 3
BASE_BACKOFF = 1  # seconds

def bulk_index_with_retry(documents):
    """Bulk index with granular error handling and retry logic."""
    create_index()
    batch_size = 500
    
    for start in range(0, len(documents), batch_size):
        batch = documents[start:start + batch_size]
        failed_docs = _process_batch(batch)
        
        # Retry loop with exponential back-off

        retry = 0
        while failed_docs and retry < MAX_RETRIES:
            wait = BASE_BACKOFF * (2 ** retry)
            LOG.info("Retrying %d failed docs after %ds back-off", 
                    len(failed_docs), wait)
            time.sleep(wait)
            
            failed_docs = _process_batch(failed_docs)
            retry += 1
        
        if failed_docs:
            return failure_response(
                f"Failed to index {len(failed_docs)} documents after retries"
            )
    
    return success_response("All documents indexed successfully")

def _process_batch(docs):
    """Process a batch and return failed documents for retry."""
    bulk_payload = []
    for doc in docs:
        bulk_payload.append(
            {"index": {"_index": INDEX_NAME, "_id": f"{uuid.uuid4().hex}"}}
        )
        doc_clean = {k: v for k, v in doc.items() if k != "vector_embedding"}
        bulk_payload.append(doc_clean)
    
    response = ops_client.bulk(body=bulk_payload)
    if not response.get("errors"):
        return []  # All succeeded

    
    return _extract_failed_items(response, docs)

```

### Integrating with Vector Indexing Flows

The repository's `vectorize_and_index_products` function handles two distinct bulk operations: one for on-disk vector indices and one for in-memory indices. Apply the same resilient retry logic to both paths:

```python

# Inside vectorize_and_index_products

if bulk_data_on_disk:
    LOG.info("Indexing on-disk vectors – %d actions", len(bulk_data_on_disk)//2)
    resp = bulk_index_with_retry_logic(bulk_data_on_disk)  # Use enhanced helper

    if not resp.get("success"):
        return resp

if bulk_data_in_memory:
    LOG.info("Indexing in-memory vectors – %d actions", len(bulk_data_in_memory)//2)
    resp = bulk_index_with_retry_logic(bulk_data_in_memory)
    if not resp.get("success"):
        return resp

```

## Best Practices for Production Workloads

When implementing bulk indexing in production OpenSearch clusters, follow these guidelines to minimize errors and ensure data durability:

- **Use the `refresh` parameter wisely** – Set `refresh=False` during large bulk loads and trigger a manual refresh only after all retries complete. This significantly improves indexing throughput.
- **Implement circuit breakers** – If error rates exceed a threshold (e.g., >10% of documents failing), halt the pipeline to prevent cascading failures and alert operators.
- **Log structured error data** – Store full bulk responses (sanitized of sensitive data) in CloudWatch Logs or S3 for post-mortem analysis of partial failures.
- **Validate documents pre-flight** – Check field types and document sizes against index mappings before sending to OpenSearch, reducing preventable bulk errors.
- **Monitor cluster health** – Watch for `cluster_block_exception` or `es_rejected_execution_exception` in error responses, which indicate cluster resource constraints rather than document issues.

## Summary

- **Bulk indexing errors in OpenSearch** are indicated by the `errors` flag in the response, but this flag alone masks partial successes where some documents index correctly.
- **The `items` array** contains granular details for each operation, including status codes, document IDs, and specific error messages required for debugging.
- **The sample repository** currently aborts on any error in `bulk_index_documents` and `vectorize_and_index_products`, but you can enhance these functions to extract failed items and retry only those documents.
- **Exponential back-off** and targeted retry logic prevent unnecessary load on the cluster while ensuring data durability for transient failures.
- **Production implementations** should disable automatic refresh during bulk loads, implement circuit breakers, and log structured error data for forensic analysis.

## Frequently Asked Questions

### How do I identify which specific documents failed in a bulk indexing operation?

Inspect the `items` array in the bulk response. Each element corresponds to the operation at the same index in your bulk payload. Extract the operation details using `item.get("index")` or `item.get("create")`, then check the `status` field (HTTP status code) and the `error` object for failure details. The `_id` field within each item tells you exactly which document failed.

### What is the difference between the `errors` flag and the `items` array in an OpenSearch bulk response?

The `errors` boolean is a high-level indicator that returns `True` if any operation in the bulk request failed. However, it does not tell you which operations succeeded or failed. The `items` array provides granular, per-operation results including status codes, error messages, document IDs, and versioning information. You need to parse `items` to implement partial failure recovery.

### How should I handle transient errors like throttling or temporary node unavailability during bulk indexing?

Implement an exponential back-off retry mechanism that resubmits only the failed documents rather than the entire batch. Start with a base delay (e.g., 1 second) and double the wait time for each retry attempt (1s, 2s, 4s). Limit retries to 3-5 attempts to prevent infinite loops. Check for specific error types like `429 Too Many Requests` or `503 Service Unavailable` to distinguish transient errors from permanent data validation failures.

### Can I apply the same error handling logic to vector-based indexing in the sample repository?

Yes. The `vectorize_and_index_products` function in [`artifacts/index_lambda/opensearch_index.py`](https://github.com/aws-samples/sample-for-amazon-opensearch-service-tutorials-101/blob/main/artifacts/index_lambda/opensearch_index.py) performs two separate bulk operations—one for on-disk vectors and one for in-memory vectors. You can replace the simple `if response.get("errors")` checks with the granular inspection and retry logic described above. Apply the helper functions to both `bulk_data_on_disk` and `bulk_data_in_memory` payloads to ensure resilient vector indexing.