How to simulate a Worker Node failure in your Kubernetes Cluster without direct access

I pondered: How can I simulate a node outage due to misconfiguration using only kubectl? This approach can be valuable for preparing training sessions or technical interviews.

Suppose I aim to alter the kubelet service configuration to prevent it from starting.

First, I need a method to access the node and modify the kubelet service configuration. I can utilize the kubectl debug command to create an ephemeral container that shares its Linux namespaces with the node. While this feature is commonly used for debugging distroless containers, it can also be employed for node debugging.

I also would like to be able to use commands like systemctl as if i were directly on the node.
Let's try with the general profile:

$ kubectl debug node/node01 --profile=general --image=busybox -it
# Creating debugging pod node-debugger-node01-2zpm8 with container debugger on node node01.
# If you don't see a command prompt, try pressing enter.

/ $ df -h
# Filesystem                Size      Used Available Use% Mounted on
# overlay                  18.3G      5.9G     12.4G  32% /
# tmpfs                    64.0M         0     64.0M   0% /dev
# /dev/vda1                18.3G      5.9G     12.4G  32% /host
# ...

/ $ cat /host/etc/os-release | head -n1
# PRETTY_NAME="Ubuntu 24.04.1 LTS"

/ $ ps aux | grep kubelet
# 736 root      0:42 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/# bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/# lib/kubelet/config.yaml --container-runtime-endpoint=unix:///var/run/containerd/# containerd.sock --pod-infra-container-image=registry.k8s.io/pause:3.10 # --container-runtime-endpoint unix:///run/containerd/containerd.sock # --cgroup-driver=systemd --eviction-hard imagefs.available<5%,memory.# available<100Mi,nodefs.available<5% --fail-swap-on=false
# 18233 root      0:00 grep kubelet

Yeah ! I can access the host filesystem and view its processes, but I still can't use systemctl commands. Let's chroot into the host filesystem and intentionally disrupt the kubelet service:

/ $ chroot /host
node01:/ $ sed -i 's/config\.yaml/conf1g.yaml/g' /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
node01:/$ systemctl daemon-reload
node01:/$ systemctl restart kubelet

I was immediately ejected from the node, and after a few seconds—the duration of the controller manager's grace period—the node was marked as NotReady.

Let's encapsulate this process into a concise one-liner to simulate a failure on a random node in the cluster:

$ kubectl debug $(kubectl get nodes -l '!node-role.kubernetes.io/control-plane' -o name | shuf -n1) \
  --profile=general --image=busybox -- \
  chroot /host sh -c "sed -i 's/config\\.yaml/conf1g.yaml/g' /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf && \
  systemctl daemon-reload && \
  systemctl restart kubelet"

The key takeaway from this test is the critical importance of exercising caution when assigning the pods/ephemeralcontainers permission to users. Granting this permission allows users to inject ephemeral containers into existing pods/nodes, which can bypass standard admission controls and security policies such as Pod Security Admission, OPA Gatekeeper, or Kyverno. This capability poses a significant security risk, as it enables the creation of privileged containers without undergoing the usual validation processes, potentially compromising the cluster's security posture.