How to Debug Deployment Issues in the Starship Kubernetes Environment

To debug Starship Kubernetes deployment issues, systematically verify cluster prerequisites, inspect Helm release status, examine pod logs and events with kubectl, validate port-forwarding configuration, and run end-to-end tests to isolate failures across the multi-chain infrastructure stack.

Starship by Hyperweb.io orchestrates multi-chain development networks entirely within Kubernetes clusters. When deployments fail, you must trace backward through the installation layers—cluster setup, Helm chart deployment, StatefulSet pod initialization, and service exposure—to identify configuration errors or resource constraints. This guide provides a systematic approach using the actual source files from the hyperweb-io/starship repository.

Understand the Starship Deployment Architecture

Starship deploys infrastructure through five distinct layers defined in the repository scripts:

1. Cluster preparation – scripts/cluster-setup.sh creates the ingress and cert-manager Helm releases if missing (see lines 14-60).
2. Helm chart installation – scripts/install.sh builds a Helm install command from your config.yaml and streams chain-specific script files into the chart using --set-file (see lines 75-80).
3. Pod startup – The Helm chart creates a StatefulSet for each chain component (genesis, comet-mock, relayers, registry, explorer).
4. Port forwarding – scripts/port-forward.sh reads the same config.yaml and executes kubectl port-forward for every exposed service (see lines 44-119).
5. End-to-end validation – The test suite in starship/tests/e2e/README.md verifies functionality against the running cluster (lines 30-65).

When debugging, inspect the Kubernetes resources created at each layer to isolate the failure point.

Verify Cluster Prerequisites

Before troubleshooting Starship-specific components, confirm the underlying Kubernetes infrastructure is ready. The scripts/cluster-setup.sh file contains the logic for verifying these prerequisites.

Run these verification commands:

• Check Kubernetes connectivity: kubectl version --short ensures kubectl can communicate with the cluster.
• Verify Helm functionality: helm version confirms the package manager is operational.
• Inspect required namespaces: kubectl get ns should list ingress and cert-manager namespaces created by the setup_ingress and setup_cert_manager functions.
• Validate ingress controller: helm list -n ingress confirms the controller installation.
• Check cert-manager pods: kubectl get pods -n cert-manager verifies certificate management infrastructure.

If any namespace is missing, re-run scripts/cluster-setup.sh to install the ingress controller and cert-manager CRDs with --set crds.enabled=true.

Inspect Helm Release Health

If cluster prerequisites are satisfied but chains are not deploying, examine the Helm release state. The scripts/install.sh generates the install command at lines 75-80, typically creating a release named starship-localnet.

Execute these diagnostic commands:

helm status starship-localnet
helm get values starship-localnet

If Helm reports "release not found", the installation failed or used a different release name. Re-run the install script with an explicit name flag:

./scripts/install.sh -c configs/two-chain.yaml -n starship-localnet

When helm status shows a failed or pending state, examine the underlying pod events before attempting a reinstall.

Examine Pod and Service Health

Kubernetes pods and services expose the actual runtime state of your Starship deployment. Use kubectl to identify startup failures in the chain StatefulSets.

List all Starship resources:

kubectl get pods -A | grep starship
kubectl get svc -A | grep starship

Common failure patterns include:

• Pods stuck in Init or CrashLoopBackOff – Usually indicates an incorrect config.yaml (e.g., missing port definitions) or missing chain scripts. Validate your YAML against the sample in README.md (lines 77-98) and re-run scripts/install.sh.
• Services with no endpoints – The underlying pod never reached a Ready state. Run kubectl describe pod <pod> to inspect events like image pull errors or volume mount failures.
• Missing registry service – Occurs when registry.enabled: false is set in the configuration. Enable it in config.yaml or forward the registry manually.

For detailed diagnostics, use:

kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> -c <container>

The describe output reveals scheduling and resource events, while logs display chain-specific startup errors from the container runtime.

Validate Port Forwarding Configuration

The scripts/port-forward.sh script bridges your local environment to the cluster services. If you cannot reach chain RPC or REST endpoints, verify the port-forwarding layer.

Run the script with the same configuration used during installation:

./scripts/port-forward.sh -c configs/two-chain.yaml

If the script fails to connect:

1. Confirm port definitions – Ensure config.yaml defines the required ports under chains[].ports (e.g., rpc, rest).
2. Check pod naming conventions – The script replaces underscores with hyphens in chain IDs (line 69). Verify your id field in config.yaml follows this convention to match the actual pod names like osmosis-genesis-0.

Run End-to-End Tests for Validation

Execute the comprehensive test suite to verify the entire deployment stack:

make test HELM_FILE=configs/two-chain.yaml

The Makefile in examples/upgrade-test invokes a Go test harness that communicates with the forwarded services. If tests fail:

• Check for HTTP connection errors – Indicates missing or misconfigured port-forwarding.
• Compare with CI output – Review .github/workflows/pr-tests.yaml to see how the same configs behave in continuous integration.

Resolve Common Deployment Gotchas

Issue	Diagnosis	Remedy
Helm install hangs indefinitely	Helm waits for pod readiness (--wait) but a pod never becomes Ready.	Run helm status to identify the failing pod; correct resource limits or configuration errors in config.yaml, then reinstall.
Port-forward script cannot find pods	Chain ID mismatch between config and actual pod names (underscores vs hyphens).	Ensure your config.yaml uses IDs compatible with the hyphen-replacement logic at line 69 of scripts/port-forward.sh.
Ingress resources unreachable	Ingress controller not installed or incorrect ingressClassName.	Verify setup_ingress completed successfully in scripts/cluster-setup.sh, then inspect kubectl get ingress.
TLS certificate errors	Missing cert-manager CRDs or wrong namespace.	Re-run scripts/cluster-setup.sh to install CRDs with --set crds.enabled=true.
Pods stuck in Pending state	Insufficient node resources (CPU/memory).	Scale up your local Kubernetes cluster (Docker Desktop or Kind) or adjust resource requests in the Helm values.

Quick Debugging Checklist

Execute these commands in sequence to isolate deployment failures:

# 1. Verify cluster health

kubectl get nodes
kubectl version --short
helm version

# 2. Inspect namespaces and releases

kubectl get ns
helm list -A

# 3. Check Helm release status

helm status starship-localnet

# 4. Examine Starship pods and services

kubectl get pods -A | grep starship
kubectl get svc -A | grep starship

# 5. Diagnose specific pod failures

kubectl describe pod <pod> -n <ns>
kubectl logs <pod> -n <ns>

# 6. Establish port forwarding

./scripts/port-forward.sh -c configs/two-chain.yaml

# 7. Validate functionality

make test HELM_FILE=configs/two-chain.yaml

Summary

• Verify prerequisites using scripts/cluster-setup.sh to ensure ingress and cert-manager are installed before deploying Starship.
• Inspect Helm releases with helm status and helm get values to confirm chart installation parameters defined in scripts/install.sh.
• Diagnose pod failures via kubectl describe and kubectl logs to identify CrashLoopBackOff causes or resource constraints.
• Validate port-forwarding by running scripts/port-forward.sh with the correct config.yaml and ensuring chain IDs match pod naming conventions (hyphens vs underscores).
• Execute E2E tests using the Makefile targets to verify the entire deployment stack functionality.

Frequently Asked Questions

How do I check if my Starship Helm release installed correctly?

Run helm status starship-localnet replacing starship-localnet with your specific release name. If the status shows deployed, examine the pod health with kubectl get pods. If the release is failed or pending, use helm get values to verify the configuration matches your config.yaml and check scripts/install.sh (lines 75-80) for the exact command generation logic.

Why are my Starship pods stuck in CrashLoopBackOff?

This typically indicates invalid configuration in config.yaml or missing chain-specific scripts. Run kubectl logs <pod-name> to view the specific startup error. Validate your YAML structure against the sample in the root README.md (lines 77-98) and ensure all required ports are defined under chains[].ports before reinstalling with scripts/install.sh.

How do I fix port-forwarding script errors when it cannot find pods?

The scripts/port-forward.sh script converts underscores to hyphens in chain IDs at line 69 to match Kubernetes pod naming conventions. Ensure your config.yaml uses chain IDs that align with this transformation (e.g., cosmos_hub becomes cosmos-hub in pod names). Verify the pods exist with kubectl get pods | grep <chain-id> before running the port-forward script.

What should I do if the Helm install command hangs indefinitely?

Helm hangs when waiting for pod readiness (--wait flag) but a pod never becomes Ready. In a separate terminal, run kubectl get pods to identify which pod is failing, then use kubectl describe pod <name> to check for image pull errors, resource limits, or volume mount issues. Fix the underlying cause in your configuration or cluster resources, then run helm uninstall starship-localnet before attempting a fresh install.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}