Best Practices for Deploying Oracle AI Applications on Kubernetes

Deploy Oracle AI applications on Kubernetes using Infrastructure-as-Code with Terraform, environment-specific Kustomize overlays, secure OCI Vault secret injection, and automated CI/CD pipelines to ensure production-grade security, scalability, and observability.

Deploying Oracle AI workloads on Oracle Container Engine for Kubernetes (OKE) requires a cloud-native approach that balances security with operational efficiency. The reference implementation in the oracle-devrel/oracle-ai-developer-hub repository demonstrates a production-grade deployment pattern for generative AI applications. This guide covers the essential best practices for deploying Oracle AI applications on Kubernetes, from infrastructure provisioning to runtime security and observability.

Infrastructure as Code with Terraform

Provision all OKE resources using Terraform to ensure repeatable, version-controlled infrastructure. In apps/oci-generative-ai-jet-ui/deploy/terraform/oke.tf, the cluster configuration automatically selects the latest supported Kubernetes version via local.cluster_k8s_latest_version, ensuring you run on a patched, supported release.

Define your node pool capacity explicitly using the worker_pools block. The reference implementation specifies worker_pool_size: 2 with VM.Standard.E5.Flex shapes, providing predictable compute capacity for AI inference workloads.

# Initialise Terraform providers

terraform init

# Create the OKE cluster, VCN, subnets, and security lists

terraform apply --auto-approve

Environment Management with Kustomize

Manage multiple deployment environments using Kustomize overlays to keep a single source of truth for base resources while customizing per-environment settings. Store environment-specific manifests under deploy/k8s/overlays/ (e.g., prod), adjusting namespaces, replica counts, and image tags without duplicating YAML.

Apply the production configuration using the overlay path:

# Apply the production overlay

kubectl apply -k deploy/k8s/overlays/prod

The kustomization.yaml in apps/oci-generative-ai-jet-ui/deploy/k8s/overlays/prod/ maps the base resources to production-specific patches, enabling clean separation between development and production environments.

Secure Secrets Management

Never hardcode credentials in container images. Instead, use the scripts/setenv.mjs helper script to generate genai.json, which assembles OCI Vault secrets (API keys and endpoints) for injection as environment variables. This decouples sensitive credentials from your codebase and runtime images.

Generate the secure environment configuration:

# Produce genai.json containing OCI_GENAI_API_KEY and OCI_GENAI_ENDPOINT

npx zx scripts/setenv.mjs

For database connections, configure private endpoints so traffic between your pods and Oracle AI Database never traverses the public internet, eliminating exposure to external threats.

Networking and Ingress Configuration

Deploy a managed NGINX Ingress controller with TLS termination to handle external traffic securely. The repository includes a ready-made configuration in apps/oci-generative-ai-jet-ui/deploy/k8s/ingress/ingress-controller.yaml, which uses the Kubernetes-recommended app.kubernetes.io/* label scheme for service discovery and monitoring.

Ensure all ingress resources follow labeling standards for consistent network policy application and load balancer health check configuration.

Resource Management and Scaling

Define explicit CPU and memory limits in your deployment manifests to prevent resource starvation and ensure predictable performance. The reference OKE node pool configuration provides a fixed capacity envelope that aligns with your AI workload requirements.

When scaling, leverage the node pool architecture defined in oke.tf to maintain adequate capacity for both inference and training workloads without over-provisioning.

CI/CD Automation

Automate the build and deployment pipeline using the repository's helper scripts. The scripts/release.mjs script builds Docker images and pushes them to Oracle Container Registry, while scripts/kustom.mjs regenerates Kustomize overlay files with updated image tags.

Integrate these into your CI/CD pipeline (GitHub Actions, OCI Code Repository, or GitLab CI) to achieve fully automated rollouts:

# Build and push container images

npx zx scripts/release.mjs

# Regenerate overlay configurations

npx zx scripts/kustom.mjs

Point kubectl to the generated kubeconfig before deployment:

export KUBECONFIG="$(pwd)/deploy/terraform/generated/kubeconfig"
kubectl cluster-info
kubectl apply -k deploy/k8s/overlays/prod

Retrieve the external load balancer IP to verify service exposure:

kubectl get service -n backend -o jsonpath='{.items[?(@.spec.type=="LoadBalancer")].status.loadBalancer.ingress[0].ip}'

Observability and Monitoring

Enable the OKE Monitoring add-on to leverage OCI Logging and Metrics for cluster-wide observability. The deployed backend service writes interaction records to the autonomous database, providing a built-in audit trail via queries like SELECT * FROM interactions;.

Monitor node pool health, pod resource utilization, and ingress traffic patterns to maintain service level objectives for your AI applications.

Summary

• Use Terraform (oke.tf) to provision OKE clusters with automatic version selection and defined node pools (VM.Standard.E5.Flex, worker_pool_size: 2)
• Implement Kustomize overlays (deploy/k8s/overlays/prod) for environment-specific configurations without code duplication
• Secure credentials via scripts/setenv.mjs and OCI Vault injection, avoiding hardcoded secrets
• Configure private endpoints for database connectivity and managed NGINX Ingress with TLS for external traffic
• Automate deployments with scripts/release.mjs and scripts/kustom.mjs integrated into CI/CD pipelines
• Enable observability through OKE Monitoring add-ons and database audit trails

Frequently Asked Questions

How do I manage secrets for Oracle AI applications in Kubernetes?

Use the scripts/setenv.mjs script in the repository to generate genai.json, which pulls credentials from OCI Vault and formats them for injection as environment variables. This approach decouples secrets from container images and source code, adhering to the principle of least privilege.

What is the recommended way to handle different deployment environments?

Store base Kubernetes manifests in deploy/k8s/base/ and create environment-specific overlays in deploy/k8s/overlays/ (e.g., prod). Use Kustomize to apply patches for namespace, replica count, and image tag variations, then deploy with kubectl apply -k deploy/k8s/overlays/prod.

How do I ensure my OKE cluster runs the latest Kubernetes version?

The Terraform configuration in apps/oci-generative-ai-jet-ui/deploy/terraform/oke.tf uses local.cluster_k8s_latest_version to automatically select the newest supported Kubernetes version during provisioning. Run terraform plan regularly to detect version updates and apply changes through your Infrastructure-as-Code workflow.

What networking configuration is recommended for database connectivity?

Configure private endpoints for Oracle AI Database connections to ensure traffic remains within the Oracle Cloud Infrastructure network and never traverses the public internet. Combine this with security lists and network policies defined in your Terraform (oke.tf) to enforce zero-trust networking between pods and data stores.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}