Accidentally deleted a Kubernetes service ? Helm Rollback to the Rescue
Updated: Oct 14, 2019
Introduction
In a kubernetes cluster setup where terraform, ansible and helm automate all our deployments we often start deleting random things to see how resilient our cluster is, if our monitoring picks it up, and how we can rectify things in the easiest way possible (with a minimal risk/impact on production environments).
That ‘woopsie’ moment
So, during a new monitoring setup, I was happily deleting pods left and right, to see how fast our monitoring could pick it up. After a short break I came back to my shell, still running one of our favourites; the wonderful k9s.
Without thinking I gave it a quick ctrl-d to delete whatever pod I happened to be on (sftp). After the deletion confirmation (which of course I didn’t read) I realised that instead of the sftp pod, I deleted the sftp service.
Woopsie.
First attempt
No problem, I thought, just retrigger a circleci deploy workflow, that’ll fix it. However, to my surprise, our terraform plan didn’t pick up that one of the helm charts had a resource missing!
Now what ?
Am I sure that the service really has been deleted ? Who am I to question Terraform/Helm, the Mighty Gods of Omnipotence. Let’s check our status:
> helm status sftp
This confirmed that indeed the service resource was MISSING:
The Solution
Helm keeps a record of all the yaml files per revision, so you can always go back to a previous release. I wonder if I can go ‘back’ to the current revision, acting as some sort of a ‘refresh’ of the revision. Turns out, that is exactly what it does.
> helm history sftp
Let’s remember that revision number : 1
> helm rollback sftp 1
“Rollback was a success.”
You don’t say. Don’t be mad if I don’t take your word for it, dear Helm.
> helm status sftp
There you have it. Quick and easy fix, with minimal (to no) risk.
About us
The kuori team is trying to bring technological revolution to companies in an affordable, qualitative and safe manner. We love technologies, but have years of experience as consultants teaching us that it’s very hard to embed these beautiful ecosystems in most companies. Our managed ecosystems are catered exactly to your needs, and enable you to focus on what’s really important.
Or are you a techie like us ? We’re always up for a good chat and share experiences, or write a blog post about requested topics.