Kubernetes poweruser tips
Table of Contents
Fetch revisions for a StatefulSet
#
Revisions for StatefulSet
& DaemonSet
are stored in ControllerRevision
objects.
kubectl get controllerrevisions --all-namespaces
These revisions allow rolling back StatefulSet
by running
kubectl rollout undo
Different from Deployment
which capture the state in ReplicaSet
kudos Sheogorath
Get events for a certain Pod #
kubectl get events --field-selector involvedObject.name=<pod-name>
Get events for other objects #
kubectl get events --field-selector involvedObject.kind=<resource-name>,involvedObject.name=<object-name>
- resource-name could be
Pod
,Job
,CronJob
etc. - object-name the name of resource
Delete a PVC stuck in “Terminating” state #
You’ve probably already tried to forcefully delete the PVC
kubectl delete pvc <pvc-name> --force --grace-period=0
PVC needs to be unmounted from the node in order to finalize deletion and we do that by annotating the object with the removal of finalizer field.
kubectl patch pvc <pvc-name> -p '{"metadata":{"finalizers":null}}'
Also kudos Dean Lewis
Delete a Pod stuck in “Terminating” state #
This is a dangerous operation. Be prepared in case you might corrupt the ETCD database and have a backup. These resources might be in handy:
If you did everything in order to delete a Pod, meaning delete its dependencies and it’s still not terminating, last resort is to attempt a removal from ETCD
Identify the IPs of your ETCD replicas #
Let’s assume we have these IPs
10.11.68.181
10.11.68.226
10.11.68.164
and we want to terminate the Pod oops-im-stuck-1234
Santity check: list the Pod #
ETCDCTL_API=3 etcdctl --endpoints=https://10.11.68.181:2379,https://10.11.68.226:2379,https://10.11.68.164:2379 \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
get --prefix "/registry/pods/default/oops-im-stuck-1234"
Delete the Pod from ETCD #
ETCDCTL_API=3 etcdctl --endpoints=https://10.11.68.181:2379,https://10.11.68.226:2379,https://10.11.68.164:2379 \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
del "/registry/pods/default/oops-im-stuck-1234"