# How the Pod Certificate Controller Manages mTLS Certificates in Agent Substrate

> Discover how the Pod Certificate Controller automates mTLS certificate management in Agent Substrate by watching requests, assigning them to replicas, and signing short-lived X.509 certificates with SPIFFE identities.

- Repository: [Agent Substrate/substrate](https://github.com/agent-substrate/substrate)
- Tags: internals
- Published: 2026-08-22

---

**The Pod Certificate Controller automates mTLS certificate lifecycle by watching PodCertificateRequest objects, deterministically assigning them to controller replicas via rendezvous hashing, and signing short-lived X.509 certificates through specialized signers that inject SPIFFE identities and update ClusterTrustBundles.**

The `podcertcontroller` component in the `agent-substrate/substrate` repository is a lightweight, in-process Kubernetes controller designed to issue and rotate mutual TLS certificates for pods. It operates by reconciling custom `PodCertificateRequest` resources and maintaining the trust infrastructure required for secure pod-to-pod and pod-to-control-plane communication.

## Certificate Signer Architecture

The controller supports two distinct signing implementations to handle different identity use cases.

### Pod Identity Signer

The **Pod Identity signer** issues certificates that cryptographically identify individual pods. Located in [`cmd/podcertcontroller/internal/podidentitysigner/podidentitysigner.go`](https://github.com/agent-substrate/substrate/blob/main/cmd/podcertcontroller/internal/podidentitysigner/podidentitysigner.go), this signer generates X.509 certificates containing SPIFFE URIs, client authentication key usage, and optional server authentication extensions for components like `atelet` or worker-pool pods that must accept TLS connections.

### Service-DNS Signer

The **Service-DNS signer** handles certificates for Kubernetes Service DNS names rather than individual pod identities. Implemented in [`cmd/podcertcontroller/internal/servicednssigner/servicednssigner.go`](https://github.com/agent-substrate/substrate/blob/main/cmd/podcertcontroller/internal/servicednssigner/servicednssigner.go), this signer enables components to present a stable DNS-based TLS identity regardless of which pod instance handles the request.

## Certificate Issuance Workflow

The `podcertcontroller` processes certificate requests through a deterministic five-stage pipeline defined in [`cmd/podcertcontroller/internal/signercontroller/signercontroller.go`](https://github.com/agent-substrate/substrate/blob/main/cmd/podcertcontroller/internal/signercontroller/signercontroller.go).

### 1. Watching PodCertificateRequest Objects

The controller initializes a shared informer using `certinformersv1beta1.NewFilteredPodCertificateRequestInformer` (lines 77-99 in [`signercontroller.go`](https://github.com/agent-substrate/substrate/blob/main/signercontroller.go)) to watch for create, update, and delete events on `PodCertificateRequest` resources. Events feed into an in-memory workqueue that decouples API server notifications from processing logic.

### 2. Rendezvous Hashing for Replica Assignment

To ensure exactly-once processing in horizontally scaled deployments, the controller implements rendezvous hashing in [`cmd/podcertcontroller/internal/rendezvous/rendezvous.go`](https://github.com/agent-substrate/substrate/blob/main/cmd/podcertcontroller/internal/rendezvous/rendezvous.go). Each replica evaluates `hasher.AssignedToThisReplica` against the PCR name to deterministically decide ownership, guaranteeing that a request is processed by exactly one controller instance regardless of replica count.

### 3. Certificate Signing Process

Once assigned, the controller invokes the appropriate signer's `handler.MakeCert` method. In the Pod Identity signer ([`podidentitysigner.go`](https://github.com/agent-substrate/substrate/blob/main/podidentitysigner.go)), the process executes:

- **Pod validation** (lines 21-29): Retrieves the target pod and validates its UID against the request
- **Identity construction** (lines 35-49): Determines certificate lifetime and constructs the SPIFFE URI
- **Extension injection** (lines 69-78): Calls `substratex509.AddPodIdentityToCertificate` to embed pod-specific X.509 extensions
- **Chain assembly** (lines 82-92): Signs the leaf certificate with the local CA and assembles the PEM-encoded chain

### 4. Updating PCR Status

After signing, the controller updates the `PodCertificateRequest` status subresource (lines 102-124 in [`podidentitysigner.go`](https://github.com/agent-substrate/substrate/blob/main/podidentitysigner.go)) to include:

- `pcr.Status.CertificateChain`: The complete PEM-encoded certificate chain
- `pcr.Status.NotBefore` and `pcr.Status.NotAfter`: Validity bounds
- Condition marking the request as `Issued`

### 5. Maintaining ClusterTrustBundles

The `ensureBundles` method in [`signercontroller.go`](https://github.com/agent-substrate/substrate/blob/main/signercontroller.go) (lines 8-54) runs a continuous reconciliation loop to create or update `ClusterTrustBundle` resources. These bundles contain the controller's CA certificates and are labeled for discovery, allowing other cluster components to validate the mTLS certificates issued by the controller.

## Requesting and Verifying mTLS Certificates

Pods request certificates by creating `PodCertificateRequest` objects. The controller's response contains the signed certificate chain, while trust validation relies on ClusterTrustBundles.

### Creating a Certificate Request

```go
pcr := &certsv1beta1.PodCertificateRequest{
    ObjectMeta: metav1.ObjectMeta{
        Name:      "my-pod-abc123",
        Namespace: "default",
    },
    Spec: certsv1beta1.PodCertificateRequestSpec{
        PodName:            "my-pod",
        PodUID:             podUID,
        ServiceAccountName: "default",
        ServiceAccountUID:  saUID,
        NodeName:           nodeName,
        NodeUID:            nodeUID,
        MaxExpirationSeconds: pointer.Int32Ptr(3600), // 1 hour max
        SignerName:         podidentitysigner.Name,
    },
}

```

### Retrieving Issued Certificates

After the controller processes the request, the certificate material appears in the status:

```go
fmt.Println(pcr.Status.CertificateChain) // PEM‑encoded cert chain
fmt.Println(pcr.Status.NotBefore)        // Certificate validity start
fmt.Println(pcr.Status.NotAfter)         // Certificate expiration

```

### Validating Certificates with Trust Bundles

Downstream components retrieve the CA chain to validate peer certificates:

```go
ctb, err := kubeClient.CertificatesV1beta1().
    ClusterTrustBundles().
    Get(context.Background(), "podidentity.podcert.ate.dev:identity:primary-bundle", metav1.GetOptions{})
if err != nil {
    log.Fatalf("failed to get trust bundle: %v", err)
}
roots := []byte(ctb.Spec.TrustBundle) // PEM‑encoded CA chain

```

## Key Source Files and Entrypoints

| File | Responsibility |
|------|----------------|
| [`cmd/podcertcontroller/main.go`](https://github.com/agent-substrate/substrate/blob/main/cmd/podcertcontroller/main.go) | Entrypoint that initializes the local CA pool and starts the signer controller |
| [`cmd/podcertcontroller/internal/signercontroller/signercontroller.go`](https://github.com/agent-substrate/substrate/blob/main/cmd/podcertcontroller/internal/signercontroller/signercontroller.go) | Core reconciliation logic, PCR watching, and ClusterTrustBundle maintenance |
| [`cmd/podcertcontroller/internal/podidentitysigner/podidentitysigner.go`](https://github.com/agent-substrate/substrate/blob/main/cmd/podcertcontroller/internal/podidentitysigner/podidentitysigner.go) | SPIFFE-aware certificate generation for pod identities |
| [`cmd/podcertcontroller/internal/servicednssigner/servicednssigner.go`](https://github.com/agent-substrate/substrate/blob/main/cmd/podcertcontroller/internal/servicednssigner/servicednssigner.go) | DNS-based certificate signing for Kubernetes Services |
| [`cmd/podcertcontroller/internal/rendezvous/rendezvous.go`](https://github.com/agent-substrate/substrate/blob/main/cmd/podcertcontroller/internal/rendezvous/rendezvous.go) | Deterministic request-to-replica assignment algorithm |

## Summary

- The **Pod Certificate Controller** automates mTLS certificate lifecycle management through the `PodCertificateRequest` custom resource.
- **Rendezvous hashing** ensures exactly-once processing across horizontally scaled controller replicas.
- Two specialized signers handle distinct use cases: **Pod Identity** (SPIFFE-based) and **Service-DNS** (stable service identities).
- The controller maintains **ClusterTrustBundles** to distribute CA certificates for cluster-wide trust validation.
- Certificates are short-lived (configurable via `MaxExpirationSeconds`) and include pod-specific X.509 extensions for fine-grained identity.

## Frequently Asked Questions

### How does the controller ensure a certificate request is processed exactly once?

The controller uses **rendezvous hashing** implemented in [`cmd/podcertcontroller/internal/rendezvous/rendezvous.go`](https://github.com/agent-substrate/substrate/blob/main/cmd/podcertcontroller/internal/rendezvous/rendezvous.go). Each replica evaluates `hasher.AssignedToThisReplica` against the PCR name; only the replica that owns the hash processes the request, preventing duplicate issuance during scale-out events.

### What identity format do the issued certificates use?

Pod Identity certificates embed a **SPIFFE URI** in the Subject Alternative Name extension, constructed from the pod's namespace, service account, and unique identifiers. The `substratex509.AddPodIdentityToCertificate` function injects these extensions during signing.

### How do other pods validate these mTLS certificates?

The controller maintains a **ClusterTrustBundle** resource containing the CA certificate chain. Clients retrieve this bundle (e.g., `podidentity.podcert.ate.dev:identity:primary-bundle`) from the Kubernetes API and use it to verify the certificate chain presented by peers.

### Can the controller issue server certificates for pods accepting TLS connections?

Yes. While the default Pod Identity signer primarily issues client authentication certificates, it optionally adds the **server authentication** Extended Key Usage (EKU) for special components like `atelet` or worker-pool pods that must terminate TLS connections.