How to Use AWS4Auth for IAM Authentication with Amazon OpenSearch Service

AWS4Auth from the requests-aws4auth library signs HTTP requests using temporary IAM credentials, enabling secure, keyless authentication between AWS Lambda and Amazon OpenSearch Service without storing long-term access keys in code.

The aws-samples/sample-for-amazon-opensearch-service-tutorials-101 repository demonstrates production-ready patterns for connecting serverless applications to Amazon OpenSearch Service. This guide explains how AWS4Auth implements IAM-based request signing to authenticate Lambda functions securely.

Understanding AWS4Auth and IAM Authentication for OpenSearch

AWS4Auth implements the AWS Signature Version 4 signing process, which is required to authenticate HTTP requests to Amazon OpenSearch Service when using IAM-based access control. This approach works with both classic OpenSearch domains (service name es) and serverless collections (service name aoss).

The key advantage is zero credential storage. Instead of embedding access keys in your code or environment variables, AWS4Auth leverages the Lambda execution role's temporary credentials, which are rotated automatically by the AWS runtime.

Step-by-Step Implementation of AWS4Auth in Python

Retrieving Temporary IAM Credentials with boto3

The Lambda runtime provides temporary credentials through the boto3 credential chain. In artifacts/search_lambda/opensearch_search.py and artifacts/index_lambda/opensearch_index.py, the code retrieves these credentials at initialization:

import boto3

# Retrieve credentials from the Lambda execution role

credentials = boto3.Session().get_credentials()

The get_credentials() method returns an object containing access_key, secret_key, and token (the session token required for temporary credentials).

Creating the AWS4Auth Signer

With credentials in hand, you instantiate the AWS4Auth class from the requests-aws4auth package. This is implemented in lines 14-27 of both Lambda files:

from requests_aws4auth import AWS4Auth

SERVICE = "es"  # Use "aoss" for OpenSearch Serverless

REGION = "us-east-1"  # Or retrieve from environment

awsauth = AWS4Auth(
    credentials.access_key,
    credentials.secret_key,
    REGION,
    SERVICE,
    session_token=credentials.token,
)

Critical detail: The session_token parameter is essential when running in Lambda because the execution role uses temporary credentials. Omitting this parameter results in authentication failures.

Configuring the OpenSearch Client

The final step passes the awsauth signer to the OpenSearch client as the http_auth parameter. Lines 33-41 in the repository files show this configuration:

from opensearchpy import OpenSearch, RequestsHttpConnection

ops_client = OpenSearch(
    hosts=[{"host": ENDPOINT, "port": 443}],
    http_auth=awsauth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection,
    timeout=300,
)

The RequestsHttpConnection class is required because it respects the http_auth parameter and automatically adds the Authorization header (containing the AWS4-HMAC-SHA256 signature) to every HTTP request.

Complete Working Example from the Repository

Here is the consolidated pattern used in artifacts/search_lambda/opensearch_search.py and artifacts/index_lambda/opensearch_index.py:

import os
import boto3
import json
from opensearchpy import OpenSearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth

# Configuration from environment variables

ENDPOINT = os.getenv("OPENSEARCH_HOST")
REGION = os.getenv("AWS_REGION", "us-east-1")
INDEX_NAME = os.getenv("INDEX_NAME", "products")
SERVICE = "es"  # Change to "aoss" for serverless

# Initialize AWS4Auth with Lambda execution role credentials

credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(
    credentials.access_key,
    credentials.secret_key,
    REGION,
    SERVICE,
    session_token=credentials.token,
)

# Create OpenSearch client with IAM authentication

ops_client = OpenSearch(
    hosts=[{"host": ENDPOINT, "port": 443}],
    http_auth=awsauth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection,
    timeout=300,
)

def lambda_handler(event, context):
    # Example search implementation

    body = json.loads(event["body"])
    query = {
        "multi_match": {
            "query": body.get("searchTerm", ""),
            "fields": ["title^3", "description^2", "color"],
            "type": "best_fields"
        }
    }
    
    response = ops_client.search(index=INDEX_NAME, body={"query": query})
    return {
        "statusCode": 200,
        "body": json.dumps({"hits": response["hits"]["hits"]})
    }

The search_tutorials/opensearch_proxy_stack.py file in the same repository defines the CDK stack that provisions these Lambda functions and attaches the necessary IAM permissions to the execution role.

IAM Permissions and Security Considerations

For the signing process to succeed, the Lambda execution role must have appropriate permissions on the OpenSearch resource. According to the repository's CDK implementation in search_tutorials/opensearch_proxy_stack.py, the role requires:

• For classic domains: es:ESHttpGet, es:ESHttpPost, es:ESHttpPut, es:ESHttpDelete (or wildcard es:ESHttp*)
• For serverless collections: aoss:APIAccess and aoss:CollectionAdmin permissions

When the OpenSearch client makes a request, the AWS4Auth signer generates a signature based on the request parameters and the temporary credentials. The OpenSearch service validates this signature against the IAM policy attached to the credentials' role. If the signature is valid and the policy permits the action, the request is authorized.

Security best practice: This approach eliminates the need to store long-term AWS credentials in environment variables or code. The temporary credentials provided by the Lambda runtime are automatically rotated and have a limited lifetime, reducing the risk of credential exposure.

Summary

• AWS4Auth bridges boto3's credential handling with HTTP requests to OpenSearch, implementing AWS Signature Version 4.
• The signer requires temporary credentials from boto3.Session().get_credentials(), including the session_token for Lambda execution roles.
• Service name parameters differ by deployment type: use "es" for managed domains and "aoss" for serverless collections.
• The OpenSearch client must use RequestsHttpConnection as the connection class to properly apply the http_auth signer.
• IAM permissions on the Lambda execution role (es:ESHttp* or aoss:APIAccess) are validated through the signed request, enabling zero-trust authentication without stored credentials.

Frequently Asked Questions

What is the difference between using "es" and "aoss" as the service parameter in AWS4Auth?

The service parameter determines the AWS service namespace used in the Signature Version 4 calculation. Use "es" when connecting to a managed Amazon OpenSearch Service domain (the classic Elasticsearch-compatible service). Use "aoss" when connecting to Amazon OpenSearch Serverless collections. The signing algorithm remains identical, but the service name affects the canonical request string and the endpoint routing validation performed by AWS.

Why does AWS4Auth require a session_token when running in AWS Lambda?

AWS Lambda functions assume an IAM execution role at runtime, which generates temporary security credentials consisting of an access key ID, secret access key, and session token. The session token is required to validate that the credentials are temporary and to prevent replay attacks. When constructing AWS4Auth, you must pass session_token=credentials.token to include this token in the Authorization header; otherwise, OpenSearch will reject the request with a 403 Forbidden error.

Can I use AWS4Auth with Amazon OpenSearch Serverless?

Yes, AWS4Auth fully supports Amazon OpenSearch Serverless. You must set the service parameter to "aoss" instead of "es" when instantiating the signer. Additionally, your Lambda execution role must have permissions for aoss:APIAccess and appropriate collection-level permissions (such as aoss:CollectionAdmin or aoss:CollectionRead). The authentication flow and code structure remain identical to managed domain implementations.

Do I need to store AWS credentials in environment variables to use AWS4Auth?

No, storing long-term credentials in environment variables is unnecessary and discouraged when using AWS4Auth with Lambda. The library retrieves credentials dynamically from the boto3 credential provider chain, which automatically sources temporary credentials from the Lambda execution role via the instance metadata service. You only need to call boto3.Session().get_credentials() to access the runtime-provided credentials, eliminating the security risks associated with hardcoded or environment-stored access keys.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}