On Kubernetes, it’s easy to end up doing a thing that leaves you with a bunch of stuff that won’t delete. I have just made a right mess of removing Rancher:
$ kg ns
NAME STATUS AGE
cattle-capi-system Terminating 24h
cattle-fleet-clusters-system Terminating 24h
cattle-fleet-local-system Terminating 24h
cattle-fleet-system Terminating 24h
cattle-global-data Terminating 24h
cattle-impersonation-system Terminating 24h
cattle-local-user-passwords Terminating 24h
cattle-system Terminating 24h
cattle-turtles-system Terminating 24h
cattle-ui-plugin-system Terminating 24h
[...]
The problem here is almost always a finalizer; something that k8s has to do before deleting a thing, and it is sat waiting on that thing deleting.
It’s pretty quick and easy with a deployment to find the pod’s finalizer and delete it, but Rancher’s pulled in all sorts of exotic things in a huge number of namespaces and I don’t like the sort of manual work this sounds like.
I do like a nested loop, though, so here’s a script that’ll find anything that currently exists in a Terminating namespace and delete its finalizer before deleting the resource:
#! /bin/bash
kubectl get ns | grep Terminating | awk '{print $1}'| while read ns; do
echo "N amespace: $ns"
kubectl api-resources --verbs=list --namespaced -o name | while read thing; do
kubectl get --no-headers --show-kind --ignore-not-found -n $ns $thing
kubectl get --no-headers --show-kind --ignore-not-found -n $ns $thing | awk '{print $1}' | while read full_name; do
echo "Found $thing in ns $ns: $full_name"
kubectl patch "$full_name" -p '{"metadata":{"finalizers":[]}}' --type=merge
kubectl delete "$full_name"
done
done
done