How OpenSRE Leverages Kubernetes for Infrastructure Integration: EKS Automation Without kubectl

OpenSRE integrates with Amazon EKS by programmatically assuming IAM roles, generating presigned authentication tokens, and constructing in-memory Kubernetes clients that bypass the need for local kubeconfig files or kubectl binaries.

OpenSRE, developed in the Tracer-Cloud/opensre repository, enables Site Reliability Engineers to investigate AWS Elastic Kubernetes Service (EKS) clusters directly from the SRE runtime. Unlike traditional CLI-based workflows, OpenSRE leverages Kubernetes for infrastructure integration by embedding the Kubernetes Python client and AWS Security Token Service (STS) to create transient, credential-backed API clients that fetch live cluster state without persistent configuration artifacts.

Architectural Overview of Kubernetes Integration in OpenSRE

The platform’s Kubernetes integration follows a four-stage pipeline that abstracts AWS authentication, client construction, and API consumption into reusable service layers.

Credential Acquisition via STS AssumeRole

When a source configuration contains an eks block, OpenSRE initiates authentication by invoking sts:AssumeRole to obtain temporary credentials for the specified IAM role. This logic resides in app/services/eks/eks_k8s_client.py. The module handles external IDs and session policies, ensuring secure cross-account access before any Kubernetes API calls are attempted.

In-Memory Kubernetes Client Construction

Using the temporary AWS credentials, the _generate_eks_token function constructs a presigned EKS authentication token that mimics the behavior of aws eks get-token. This token, combined with the cluster’s CA certificate, is injected into a kubernetes.client.Configuration object. The build_k8s_clients function then instantiates a kubernetes.client.ApiClient and returns CoreV1Api and AppsV1Api instances. Critically, no kubeconfig file is written to disk; the entire client lifecycle exists in memory, eliminating filesystem dependencies and security risks associated with credential files.

Tool Layer and API Abstraction

High-level investigation tools defined under app/tools/ consume the in-memory clients to perform specific operational queries. Each tool uses the @tool decorator for registration and calls build_k8s_clients to acquire API handles. Tools include:

• EKSListClustersTool – Discovers available clusters via Boto3.
• EKSPodLogsTool – Retrieves container logs using read_namespaced_pod_log.
• EKSNodeHealthTool – Aggregates node conditions, capacity, and pressure metrics via list_node.
• EKSListPodsTool – Enumerates pods with container states and restart counts.
• EKSListNamespacesTool – Lists namespaces with status and label metadata.

Discovery and Reasoning Integration

The remote-reasoning component in app/remote/reasoning.py orchestrates these tools during root-cause analysis. When an alert contains Kubernetes context (e.g., Datadog kubernetes_context fields), the reasoning engine automatically invokes the appropriate EKS tools to fetch live evidence, enabling AI-driven SRE investigations that reference actual cluster state rather than static logs.

Core Implementation: EKS Client Service

The app/services/eks/eks_k8s_client.py module serves as the central bridge between AWS IAM and the Kubernetes control plane. The build_k8s_clients function coordinates credential retrieval and client instantiation:

# app/services/eks/eks_k8s_client.py

def build_k8s_clients(
    cluster_name: str,
    role_arn: str,
    external_id: str = "",
    region: str = "us-east-1",
) -> tuple[CoreV1Api, AppsV1Api]:
    """
    Constructs in-memory CoreV1Api and AppsV1Api clients for the target EKS cluster.
    """
    # 1. Assume role via STS

    sts_client = boto3.client("sts", region_name=region)
    assume_kwargs = {"RoleArn": role_arn, "RoleSessionName": "opensre-session"}
    if external_id:
        assume_kwargs["ExternalId"] = external_id
    
    response = sts_client.assume_role(**assume_kwargs)
    credentials = response["Credentials"]
    
    # 2. Generate presigned EKS token

    token = _generate_eks_token(
        cluster_name, 
        region, 
        credentials["AccessKeyId"],
        credentials["SecretAccessKey"],
        credentials["SessionToken"]
    )
    
    # 3. Retrieve cluster CA certificate

    eks_client = boto3.client(
        "eks",
        region_name=region,
        aws_access_key_id=credentials["AccessKeyId"],
        aws_secret_access_key=credentials["SecretAccessKey"],
        aws_session_token=credentials["SessionToken"],
    )
    cluster_info = eks_client.describe_cluster(name=cluster_name)
    ca_cert = cluster_info["cluster"]["certificateAuthority"]["data"]
    
    # 4. Build in-memory Kubernetes configuration

    configuration = kubernetes.client.Configuration()
    configuration.host = f"https://{cluster_info['cluster']['endpoint']}"
    configuration.ssl_ca_cert = ca_cert
    configuration.api_key["authorization"] = f"Bearer {token}"
    
    api_client = kubernetes.client.ApiClient(configuration)
    core_v1 = kubernetes.client.CoreV1Api(api_client)
    apps_v1 = kubernetes.client.AppsV1Api(api_client)
    
    return core_v1, apps_v1

The _generate_eks_token helper creates a signed URL that Kubernetes accepts as a bearer token, eliminating the need for the AWS CLI or aws-iam-authenticator binary.

Practical Examples: Fetching Live Cluster Data

OpenSRE exposes Kubernetes functionality through discrete tools that return JSON-serializable results suitable for automated reasoning pipelines.

Retrieving Pod Logs

The EKSPodLogsTool fetches container logs without shelling out to kubectl logs. Located in app/tools/EKSPodLogsTool/__init__.py, it uses the CoreV1Api to stream logs directly:

# app/tools/EKSPodLogsTool/__init__.py

def get_eks_pod_logs(
    cluster_name: str,
    namespace: str,
    pod_name: str,
    role_arn: str,
    external_id: str = "",
    region: str = "us-east-1",
    tail_lines: int = 100,
    **_kwargs: Any,
) -> dict[str, Any]:
    """Fetch logs from a specific EKS pod."""
    # Build a CoreV1Api client without touching the file system

    core_v1, _ = build_k8s_clients(cluster_name, role_arn, external_id, region)

    # Directly call the Kubernetes SDK

    logs = core_v1.read_namespaced_pod_log(
        name=pod_name,
        namespace=namespace,
        tail_lines=tail_lines,
    )
    return {
        "source": "eks",
        "available": True,
        "cluster_name": cluster_name,
        "namespace": namespace,
        "pod_name": pod_name,
        "logs": logs,
        "error": None,
    }

Inspecting Node Health

The EKSNodeHealthTool aggregates node conditions, capacity metrics, and pressure indicators to diagnose cluster-wide infrastructure issues. Implemented in app/tools/EKSNodeHealthTool/__init__.py, it utilizes the list_node API:

# app/tools/EKSNodeHealthTool/__init__.py

def get_eks_node_health(
    cluster_name: str,
    role_arn: str,
    external_id: str = "",
    region: str = "us-east-1",
    **_kwargs: Any,
) -> dict[str, Any]:
    """Get health status of all EKS nodes."""
    core_v1, _ = build_k8s_clients(cluster_name, role_arn, external_id, region)

    nodes = core_v1.list_node()
    node_health = []
    for node in nodes.items:
        conditions = {c.type: c.status for c in (node.status.conditions or [])}
        addresses = {a.type: a.address for a in (node.status.addresses or [])}
        node_health.append({
            "name": node.metadata.name,
            "internal_ip": addresses.get("InternalIP"),
            "ready": conditions.get("Ready"),
            "memory_pressure": conditions.get("MemoryPressure"),
            "disk_pressure": conditions.get("DiskPressure"),
        })
    # Return a concise report used by the reasoning engine

    return {
        "source": "eks",
        "available": True,
        "cluster_name": cluster_name,
        "nodes": node_health,
        "total_nodes": len(node_health),
        "not_ready_count": sum(1 for n in node_health if n["ready"] != "True"),
        "error": None,
    }

Key Files and Module Structure

Understanding the file layout is essential for contributors extending OpenSRE’s Kubernetes capabilities:

Component	Role	Source Path
EKS K8s Client	Builds in-memory Kubernetes clients, handles STS token generation	app/services/eks/eks_k8s_client.py
EKS Boto3 Client	Thin wrapper around Boto3 EKS client for cluster discovery	app/services/eks/eks_client.py
List Clusters Tool	Discovers available clusters in the AWS account	app/tools/EKSListClustersTool/__init__.py
Pod Logs Tool	Retrieves logs from a specific pod	app/tools/EKSPodLogsTool/__init__.py
Node Health Tool	Reports node conditions, capacity, and pressure metrics	app/tools/EKSNodeHealthTool/__init__.py
List Pods Tool	Enumerates pods with container states and restart counts	app/tools/EKSListPodsTool/__init__.py
List Namespaces Tool	Lists namespaces with status and labels	app/tools/EKSListNamespacesTool/__init__.py
Remote Reasoning	Invokes Kubernetes tools as part of root-cause analysis	app/remote/reasoning.py

Summary

• OpenSRE leverages Kubernetes for infrastructure integration by constructing ephemeral, in-memory API clients that communicate directly with EKS control planes.
• Zero local dependencies: The platform eliminates kubectl and kubeconfig files by using Boto3 STS to assume IAM roles and generating presigned EKS tokens within app/services/eks/eks_k8s_client.py.
• Unified tool interface: High-level tools in app/tools/ expose pod logs, node health, namespace listings, and deployment status as JSON-serializable functions, enabling seamless integration with automated reasoning engines.
• Secure by design: All AWS credentials are temporary session tokens, and Kubernetes bearer tokens are generated at runtime without persisting to disk.

Frequently Asked Questions

Does OpenSRE require kubectl to be installed locally?

No. OpenSRE bypasses kubectl entirely by using the official kubernetes Python client library. The build_k8s_clients function in app/services/eks/eks_k8s_client.py constructs a kubernetes.client.Configuration object programmatically, injecting the presigned EKS token and CA certificate directly into memory. This design removes filesystem dependencies and version conflicts associated with local CLI tools.

How does OpenSRE handle AWS authentication for EKS clusters?

OpenSRE uses Boto3 to call sts:AssumeRole for the IAM role specified in the source configuration. It retrieves temporary credentials (Access Key, Secret Key, and Session Token) and passes them to _generate_eks_token, which creates a signed URL compatible with the EKS authentication webhook. This token is then used as a Bearer token in the Kubernetes API client, ensuring secure, short-lived access without long-term credential storage.

Which Kubernetes resources can OpenSRE inspect?

OpenSRE exposes tools to inspect pods, nodes, namespaces, deployments, and container logs. Specific implementations include EKSPodLogsTool for container stdout/stderr, EKSNodeHealthTool for node conditions and capacity pressure, EKSListPodsTool for pod status and restart counts, EKSListNamespacesTool for namespace metadata, and analogous tools for deployment and replica set inspection. These tools utilize the CoreV1Api and AppsV1Api clients from the Kubernetes Python SDK.

Is OpenSRE limited to AWS EKS, or does it support other Kubernetes distributions?

The current implementation in Tracer-Cloud/opensre specifically targets AWS EKS through the app/services/eks/ module, leveraging STS and EKS-specific authentication mechanisms. However, the architectural pattern—instantiating kubernetes.client.ApiClient with programmatic credentials—could theoretically extend to other distributions by substituting the token generation logic with provider-specific methods (e.g., Azure AD tokens for AKS or GCP tokens for GKE). Currently, the codebase is optimized for EKS integration according to the source analysis.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}