How the Pod Certificate Controller Manages mTLS Certificates in Agent Substrate

The Pod Certificate Controller automates mTLS certificate lifecycle by watching PodCertificateRequest objects, deterministically assigning them to controller replicas via rendezvous hashing, and signing short-lived X.509 certificates through specialized signers that inject SPIFFE identities and update ClusterTrustBundles.

The podcertcontroller component in the agent-substrate/substrate repository is a lightweight, in-process Kubernetes controller designed to issue and rotate mutual TLS certificates for pods. It operates by reconciling custom PodCertificateRequest resources and maintaining the trust infrastructure required for secure pod-to-pod and pod-to-control-plane communication.

Certificate Signer Architecture

The controller supports two distinct signing implementations to handle different identity use cases.

Pod Identity Signer

The Pod Identity signer issues certificates that cryptographically identify individual pods. Located in cmd/podcertcontroller/internal/podidentitysigner/podidentitysigner.go, this signer generates X.509 certificates containing SPIFFE URIs, client authentication key usage, and optional server authentication extensions for components like atelet or worker-pool pods that must accept TLS connections.

Service-DNS Signer

The Service-DNS signer handles certificates for Kubernetes Service DNS names rather than individual pod identities. Implemented in cmd/podcertcontroller/internal/servicednssigner/servicednssigner.go, this signer enables components to present a stable DNS-based TLS identity regardless of which pod instance handles the request.

Certificate Issuance Workflow

The podcertcontroller processes certificate requests through a deterministic five-stage pipeline defined in cmd/podcertcontroller/internal/signercontroller/signercontroller.go.

1. Watching PodCertificateRequest Objects

The controller initializes a shared informer using certinformersv1beta1.NewFilteredPodCertificateRequestInformer (lines 77-99 in signercontroller.go) to watch for create, update, and delete events on PodCertificateRequest resources. Events feed into an in-memory workqueue that decouples API server notifications from processing logic.

2. Rendezvous Hashing for Replica Assignment

To ensure exactly-once processing in horizontally scaled deployments, the controller implements rendezvous hashing in cmd/podcertcontroller/internal/rendezvous/rendezvous.go. Each replica evaluates hasher.AssignedToThisReplica against the PCR name to deterministically decide ownership, guaranteeing that a request is processed by exactly one controller instance regardless of replica count.

3. Certificate Signing Process

Once assigned, the controller invokes the appropriate signer's handler.MakeCert method. In the Pod Identity signer (podidentitysigner.go), the process executes:

• Pod validation (lines 21-29): Retrieves the target pod and validates its UID against the request
• Identity construction (lines 35-49): Determines certificate lifetime and constructs the SPIFFE URI
• Extension injection (lines 69-78): Calls substratex509.AddPodIdentityToCertificate to embed pod-specific X.509 extensions
• Chain assembly (lines 82-92): Signs the leaf certificate with the local CA and assembles the PEM-encoded chain

4. Updating PCR Status

After signing, the controller updates the PodCertificateRequest status subresource (lines 102-124 in podidentitysigner.go) to include:

• pcr.Status.CertificateChain: The complete PEM-encoded certificate chain
• pcr.Status.NotBefore and pcr.Status.NotAfter: Validity bounds
• Condition marking the request as Issued

5. Maintaining ClusterTrustBundles

The ensureBundles method in signercontroller.go (lines 8-54) runs a continuous reconciliation loop to create or update ClusterTrustBundle resources. These bundles contain the controller's CA certificates and are labeled for discovery, allowing other cluster components to validate the mTLS certificates issued by the controller.

Requesting and Verifying mTLS Certificates

Pods request certificates by creating PodCertificateRequest objects. The controller's response contains the signed certificate chain, while trust validation relies on ClusterTrustBundles.

Creating a Certificate Request

pcr := &certsv1beta1.PodCertificateRequest{
    ObjectMeta: metav1.ObjectMeta{
        Name:      "my-pod-abc123",
        Namespace: "default",
    },
    Spec: certsv1beta1.PodCertificateRequestSpec{
        PodName:            "my-pod",
        PodUID:             podUID,
        ServiceAccountName: "default",
        ServiceAccountUID:  saUID,
        NodeName:           nodeName,
        NodeUID:            nodeUID,
        MaxExpirationSeconds: pointer.Int32Ptr(3600), // 1 hour max
        SignerName:         podidentitysigner.Name,
    },
}

Retrieving Issued Certificates

After the controller processes the request, the certificate material appears in the status:

fmt.Println(pcr.Status.CertificateChain) // PEM‑encoded cert chain
fmt.Println(pcr.Status.NotBefore)        // Certificate validity start
fmt.Println(pcr.Status.NotAfter)         // Certificate expiration

Validating Certificates with Trust Bundles

Downstream components retrieve the CA chain to validate peer certificates:

ctb, err := kubeClient.CertificatesV1beta1().
    ClusterTrustBundles().
    Get(context.Background(), "podidentity.podcert.ate.dev:identity:primary-bundle", metav1.GetOptions{})
if err != nil {
    log.Fatalf("failed to get trust bundle: %v", err)
}
roots := []byte(ctb.Spec.TrustBundle) // PEM‑encoded CA chain

Key Source Files and Entrypoints

File	Responsibility
cmd/podcertcontroller/main.go	Entrypoint that initializes the local CA pool and starts the signer controller
cmd/podcertcontroller/internal/signercontroller/signercontroller.go	Core reconciliation logic, PCR watching, and ClusterTrustBundle maintenance
cmd/podcertcontroller/internal/podidentitysigner/podidentitysigner.go	SPIFFE-aware certificate generation for pod identities
cmd/podcertcontroller/internal/servicednssigner/servicednssigner.go	DNS-based certificate signing for Kubernetes Services
cmd/podcertcontroller/internal/rendezvous/rendezvous.go	Deterministic request-to-replica assignment algorithm

Summary

• The Pod Certificate Controller automates mTLS certificate lifecycle management through the PodCertificateRequest custom resource.
• Rendezvous hashing ensures exactly-once processing across horizontally scaled controller replicas.
• Two specialized signers handle distinct use cases: Pod Identity (SPIFFE-based) and Service-DNS (stable service identities).
• The controller maintains ClusterTrustBundles to distribute CA certificates for cluster-wide trust validation.
• Certificates are short-lived (configurable via MaxExpirationSeconds) and include pod-specific X.509 extensions for fine-grained identity.

Frequently Asked Questions

How does the controller ensure a certificate request is processed exactly once?

The controller uses rendezvous hashing implemented in cmd/podcertcontroller/internal/rendezvous/rendezvous.go. Each replica evaluates hasher.AssignedToThisReplica against the PCR name; only the replica that owns the hash processes the request, preventing duplicate issuance during scale-out events.

What identity format do the issued certificates use?

Pod Identity certificates embed a SPIFFE URI in the Subject Alternative Name extension, constructed from the pod's namespace, service account, and unique identifiers. The substratex509.AddPodIdentityToCertificate function injects these extensions during signing.

How do other pods validate these mTLS certificates?

The controller maintains a ClusterTrustBundle resource containing the CA certificate chain. Clients retrieve this bundle (e.g., podidentity.podcert.ate.dev:identity:primary-bundle) from the Kubernetes API and use it to verify the certificate chain presented by peers.

Can the controller issue server certificates for pods accepting TLS connections?

Yes. While the default Pod Identity signer primarily issues client authentication certificates, it optionally adds the server authentication Extended Key Usage (EKU) for special components like atelet or worker-pool pods that must terminate TLS connections.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}