Cluster Administration

Executive Summary

Production-grade Kubernetes cluster administration covering the complete lifecycle from initial deployment to day-2 operations. This skill provides deep expertise in cluster architecture, high availability configurations, upgrade strategies, and operational best practices aligned with CKA/CKS certification standards.

Core Competencies

1. Cluster Architecture Mastery

Control Plane Components

┌─────────────────────────────────────────────────────────────────┐
│                      CONTROL PLANE                               │
├─────────────┬─────────────┬──────────────┬────────────────────┤
│ API Server  │ Scheduler   │ Controller   │ etcd               │
│             │             │ Manager      │                    │
│ - AuthN     │ - Pod       │ - ReplicaSet │ - Cluster state    │
│ - AuthZ     │   placement │ - Endpoints  │ - 3+ nodes for HA  │
│ - Admission │ - Node      │ - Namespace  │ - Regular backups  │
│   control   │   affinity  │ - ServiceAcc │ - Encryption       │
└─────────────┴─────────────┴──────────────┴────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                      WORKER NODES                                │
├─────────────────┬─────────────────┬─────────────────────────────┤
│ kubelet         │ kube-proxy      │ Container Runtime           │
│ - Pod lifecycle │ - iptables/ipvs │ - containerd (recommended)  │
│ - Node status   │ - Service VIPs  │ - CRI-O                     │
│ - Volume mount  │ - Load balance  │ - gVisor (sandboxed)        │
└─────────────────┴─────────────────┴─────────────────────────────┘

Production Cluster Bootstrap (kubeadm)

# Initialize control plane with HA
sudo kubeadm init \
  --control-plane-endpoint "k8s-api.example.com:6443" \
  --upload-certs \
  --pod-network-cidr=10.244.0.0/16 \
  --service-cidr=10.96.0.0/12 \
  --apiserver-advertise-address=0.0.0.0 \
  --apiserver-cert-extra-sans=k8s-api.example.com

# Join additional control plane nodes
kubeadm join k8s-api.example.com:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash> \
  --control-plane \
  --certificate-key <cert-key>

# Join worker nodes
kubeadm join k8s-api.example.com:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash>

2. Node Management

Node Lifecycle Operations

# View node details with resource usage
kubectl get nodes -o wide
kubectl top nodes

# Label nodes for workload placement
kubectl label nodes worker-01 node-type=compute tier=production
kubectl label nodes worker-02 node-type=gpu accelerator=nvidia-a100

# Taint nodes for dedicated workloads
kubectl taint nodes worker-gpu dedicated=gpu:NoSchedule

# Cordon node (prevent new pods)
kubectl cordon worker-03

# Drain node safely (for maintenance)
kubectl drain worker-03 \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=300 \
  --timeout=600s

# Return node to service
kubectl uncordon worker-03

Node Problem Detector Configuration

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-problem-detector
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: node-problem-detector
  template:
    metadata:
      labels:
        app: node-problem-detector
    spec:
      containers:
      - name: node-problem-detector
        image: registry.k8s.io/node-problem-detector/node-problem-detector:v0.8.14
        securityContext:
          privileged: true
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        volumeMounts:
        - name: log
          mountPath: /var/log
          readOnly: true
        - name: kmsg
          mountPath: /dev/kmsg
          readOnly: true
      volumes:
      - name: log
        hostPath:
          path: /var/log
      - name: kmsg
        hostPath:
          path: /dev/kmsg
      tolerations:
      - operator: Exists
        effect: NoSchedule

3. High Availability Configuration

HA Architecture Pattern

                    ┌─────────────────┐
                    │  Load Balancer  │
                    │ (HAProxy/NLB)   │
                    └────────┬────────┘
                             │
        ┌────────────────────┼────────────────────┐
        │                    │                    │
        ▼                    ▼                    ▼
┌───────────────┐    ┌───────────────┐    ┌───────────────┐
│ Control Plane │    │ Control Plane │    │ Control Plane │
│     Node 1    │    │     Node 2    │    │     Node 3    │
├───────────────┤    ├───────────────┤    ├───────────────┤
│ API Server    │    │ API Server    │    │ API Server    │
│ Scheduler     │    │ Scheduler     │    │ Scheduler     │
│ Controller    │    │ Controller    │    │ Controller    │
│ etcd          │◄──►│ etcd          │◄──►│ etcd          │
└───────────────┘    └───────────────┘    └───────────────┘
        │                    │                    │
        └────────────────────┴────────────────────┘
                             │
                    ┌────────┴────────┐
                    │  Worker Nodes   │
                    │  (N instances)  │
                    └─────────────────┘

etcd Backup & Restore

# Backup etcd
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot-$(date +%Y%m%d).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Verify backup
ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-snapshot-*.db --write-out=table

# Restore etcd (disaster recovery)
ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-snapshot-*.db \
  --data-dir=/var/lib/etcd-restored \
  --name=etcd-0 \
  --initial-cluster=etcd-0=https://10.0.0.10:2380 \
  --initial-advertise-peer-urls=https://10.0.0.10:2380

# Automated backup CronJob
kubectl apply -f - <<EOF
apiVersion: batch/v1
kind: CronJob
metadata:
  name: etcd-backup
  namespace: kube-system
spec:
  schedule: "0 */6 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: backup
            image: bitnami/etcd:3.5
            command:
            - /bin/sh
            - -c
            - |
              etcdctl snapshot save /backup/etcd-\$(date +%Y%m%d-%H%M).db
            env:
            - name: ETCDCTL_API
              value: "3"
            volumeMounts:
            - name: backup
              mountPath: /backup
            - name: etcd-certs
              mountPath: /etc/kubernetes/pki/etcd
              readOnly: true
          volumes:
          - name: backup
            persistentVolumeClaim:
              claimName: etcd-backup-pvc
          - name: etcd-certs
            hostPath:
              path: /etc/kubernetes/pki/etcd
          restartPolicy: OnFailure
          nodeSelector:
            node-role.kubernetes.io/control-plane: ""
          tolerations:
          - key: node-role.kubernetes.io/control-plane
            effect: NoSchedule
EOF

4. Cluster Upgrades

Upgrade Strategy Decision Tree

Upgrade Required?
│
├── Minor Version (1.29 → 1.30)
│   ├── Review release notes for breaking changes
│   ├── Test in staging environment
│   ├── Upgrade control plane first
│   │   └── One node at a time
│   └── Upgrade workers (rolling)
│
├── Patch Version (1.30.0 → 1.30.1)
│   ├── Generally safe, security fixes
│   └── Can upgrade more aggressively
│
└── Major changes in components
    ├── Test thoroughly
    ├── Have rollback plan
    └── Consider blue-green cluster

Production Upgrade Process

# Step 1: Upgrade kubeadm on control plane
sudo apt-mark unhold kubeadm
sudo apt-get update && sudo apt-get install -y kubeadm=1.30.0-00
sudo apt-mark hold kubeadm

# Step 2: Plan the upgrade
sudo kubeadm upgrade plan

# Step 3: Apply upgrade on first control plane
sudo kubeadm upgrade apply v1.30.0

# Step 4: Upgrade kubelet and kubectl
kubectl drain control-plane-1 --ignore-daemonsets
sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=1.30.0-00 kubectl=1.30.0-00
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubelet
kubectl uncordon control-plane-1

# Step 5: Upgrade additional control planes
sudo kubeadm upgrade node
# Then upgrade kubelet/kubectl as above

# Step 6: Upgrade worker nodes (rolling)
for node in $(kubectl get nodes -l node-role.kubernetes.io/worker -o name); do
  kubectl drain $node --ignore-daemonsets --delete-emptydir-data
  # SSH to node and upgrade packages
  kubectl uncordon $node
  sleep 60  # Allow pods to stabilize
done

5. Resource Management

Namespace Resource Quotas

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
  namespace: team-backend
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi
    limits.cpu: "40"
    limits.memory: 80Gi
    persistentvolumeclaims: "10"
    requests.storage: 500Gi
    pods: "50"
    services: "20"
    secrets: "50"
    configmaps: "50"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: team-backend
spec:
  limits:
  - type: Container
    default:
      cpu: 500m
      memory: 512Mi
    defaultRequest:
      cpu: 100m
      memory: 128Mi
    min:
      cpu: 50m
      memory: 64Mi
    max:
      cpu: 4
      memory: 8Gi
  - type: PersistentVolumeClaim
    min:
      storage: 1Gi
    max:
      storage: 100Gi

Cluster Autoscaler Configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cluster-autoscaler
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cluster-autoscaler
  template:
    metadata:
      labels:
        app: cluster-autoscaler
    spec:
      serviceAccountName: cluster-autoscaler
      containers:
      - name: cluster-autoscaler
        image: registry.k8s.io/autoscaling/cluster-autoscaler:v1.30.0
        command:
        - ./cluster-autoscaler
        - --v=4
        - --stderrthreshold=info
        - --cloud-provider=aws
        - --skip-nodes-with-local-storage=false
        - --expander=least-waste
        - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/my-cluster
        - --balance-similar-node-groups
        - --scale-down-enabled=true
        - --scale-down-delay-after-add=10m
        - --scale-down-unneeded-time=10m
        - --scale-down-utilization-threshold=0.5
        resources:
          limits:
            cpu: 100m
            memory: 600Mi
          requests:
            cpu: 100m
            memory: 600Mi

Integration Patterns

Uses skill: docker-containers

• Container runtime configuration
• Image management on nodes
• Registry authentication

Coordinates with skill: security

• RBAC for cluster admins
• Node security hardening
• Audit logging configuration

Works with skill: monitoring

• Cluster health dashboards
• Control plane metrics
• Node resource alerting

Troubleshooting Guide

Decision Tree: Cluster Health Issues

Cluster Health Problem?
│
├── API Server unreachable
│   ├── Check: systemctl status kube-apiserver
│   ├── Check: /var/log/kube-apiserver.log
│   ├── Verify: etcd connectivity
│   └── Verify: certificates not expired
│
├── Node NotReady
│   ├── Check: kubelet status on node
│   ├── Check: container runtime status
│   ├── Verify: node network connectivity
│   └── Check: disk pressure, memory pressure
│
├── Pods Pending (no scheduling)
│   ├── Check: kubectl describe pod
│   ├── Verify: node resources available
│   ├── Check: taints and tolerations
│   └── Verify: PVC bound (if using volumes)
│
└── etcd Issues
    ├── Check: etcdctl endpoint health
    ├── Check: etcd member list
    ├── Verify: disk I/O performance
    └── Check: cluster quorum

Debug Commands Cheatsheet

# Cluster-wide diagnostics
kubectl cluster-info dump --output-directory=/tmp/cluster-dump
kubectl get componentstatuses
kubectl get nodes -o wide
kubectl get events --sort-by='.lastTimestamp' -A

# Control plane health
kubectl get pods -n kube-system
kubectl logs -n kube-system kube-apiserver-<node>
kubectl logs -n kube-system kube-scheduler-<node>
kubectl logs -n kube-system kube-controller-manager-<node>

# etcd health
ETCDCTL_API=3 etcdctl endpoint health \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Node diagnostics
kubectl describe node <node-name>
kubectl get node <node-name> -o yaml | grep -A 10 conditions
ssh <node> "journalctl -u kubelet --since '1 hour ago'"

# Certificate expiration check
kubeadm certs check-expiration

# Resource usage
kubectl top nodes
kubectl top pods -A --sort-by=memory

Common Challenges & Solutions

Success Criteria

Resources

• Official Kubernetes Documentation
• Kubernetes Cluster Administration
• kubeadm Reference
• etcd Operations Guide