Cluster Bootstrap Monitor

Purpose

Run ./bootstrap.sh and monitor the cluster deployment to complete success or failure. This skill enforces continuous monitoring - no intermediate reports, no partial success.

Exit Conditions (ONLY 2 ALLOWED)

1. Complete Success: ALL of the following must be true:

   • All terraform layers applied successfully (00-persistent-auth, 01-infrastructure, 02-services)
   • All nodes Ready (kubectl get nodes)
   • All Flux kustomizations reconciled (flux get kustomizations)
   • All HelmReleases ready (flux get helmreleases -A)
   • Core services responding (Vault, Authentik, PowerDNS, Harbor, Gitea)
   • Health checks passing (HTTP endpoints, DNS resolution)

2. Irrecoverable Failure: Error that will not self-heal:

   • Terraform apply failed with non-transient error
   • Pod in CrashLoopBackOff for > 10 minutes
   • PVC stuck Pending for > 5 minutes (storage issue)
   • Certificate generation failed (cert-manager)
   • Database connection failures after pod restarts

Monitoring Protocol

Polling Intervals

• Minimum: 10 seconds between checks
• Maximum: 5 minutes between checks
• Adjust based on what's happening (faster during active deployments)

Transient Error Handling

When an error occurs that might recover:

1. Note the timestamp when first observed
2. Pre-commit to a timeout (e.g., "HelmRelease flux-system/harbor: expected to reconcile within 5 minutes")
3. Continue monitoring until timeout expires
4. Only declare failure if still broken after timeout

Health Checks to Perform

# Nodes
kubectl get nodes

# Flux GitOps
flux get kustomizations
flux get helmreleases -A

# Pods in bad states
kubectl get pods -A | grep -v Running | grep -v Completed

# PVC status
kubectl get pvc -A

# Core services
curl -sk https://auth.test-cluster.agentydragon.com/api/v2/ | head -c 100
curl -sk https://registry.test-cluster.agentydragon.com/api/v2/ | head -c 100
curl -sk https://git.test-cluster.agentydragon.com/ | head -c 100
curl -sk https://vault.test-cluster.agentydragon.com/v1/sys/health | head -c 100

# DNS (PowerDNS)
dig @10.0.3.3 auth.test-cluster.agentydragon.com +short

# Certificates
kubectl get certificates -A

Expected Timeouts

Reference these when deciding if an error is transient:

Command Reference

# Run bootstrap
cd /home/agentydragon/code/cluster && ./bootstrap.sh

# Use direnv for kubectl/talosctl
direnv exec /home/agentydragon/code/cluster kubectl get nodes
direnv exec /home/agentydragon/code/cluster flux get kustomizations

# Check Flux logs
kubectl logs -n flux-system deployment/kustomize-controller --tail=50
kubectl logs -n flux-system deployment/helm-controller --tail=50

# Check specific service
kubectl describe helmrelease <name> -n <namespace>
kubectl logs -n <namespace> deployment/<name> --tail=50

DO NOT

• Return to user with partial progress reports ("infrastructure is up, looks good!")
• Declare success before ALL health checks pass
• Give up on transient errors before timeout expires
• Assume "pods are Running" means "service is healthy"