Search
  • Carlo Wouters

Accidentally deleted a Kubernetes service ? Helm Rollback to the Rescue

Updated: Oct 14, 2019

Introduction


In a kubernetes cluster setup where terraform, ansible and helm automate all our deployments we often start deleting random things to see how resilient our cluster is, if our monitoring picks it up, and how we can rectify things in the easiest way possible (with a minimal risk/impact on production environments).


That ‘woopsie’ moment


So, during a new monitoring setup, I was happily deleting pods left and right, to see how fast our monitoring could pick it up. After a short break I came back to my shell, still running one of our favourites; the wonderful k9s.


Without thinking I gave it a quick ctrl-d to delete whatever pod I happened to be on (sftp). After the deletion confirmation (which of course I didn’t read) I realised that instead of the sftp pod, I deleted the sftp service.


Woopsie.


[the oopsie woopsie meme]


First attempt

No problem, I thought, just retrigger a circleci deploy workflow, that’ll fix it. However, to my surprise, our terraform plan didn’t pick up that one of the helm charts had a resource missing!



Now what ?

Am I sure that the service really has been deleted ? Who am I to question Terraform/Helm, the Mighty Gods of Omnipotence. Let’s check our status:


> helm status sftp


This confirmed that indeed the service resource was MISSING:




The Solution


Helm keeps a record of all the yaml files per revision, so you can always go back to a previous release. I wonder if I can go ‘back’ to the current revision, acting as some sort of a ‘refresh’ of the revision. Turns out, that is exactly what it does.


> helm history sftp




Let’s remember that revision number : 1


> helm rollback sftp 1


“Rollback was a success.”

You don’t say. Don’t be mad if I don’t take your word for it, dear Helm.


> helm status sftp





There you have it. Quick and easy fix, with minimal (to no) risk.


About us


The kuori team is trying to bring technological revolution to companies in an affordable, qualitative and safe manner. We love technologies, but have years of experience as consultants teaching us that it’s very hard to embed these beautiful ecosystems in most companies. Our managed ecosystems are catered exactly to your needs, and enable you to focus on what’s really important.


Or are you a techie like us ? We’re always up for a good chat and share experiences, or write a blog post about requested topics.

772 views

Address

Interleuvenlaan 27B
Leuven, 3001
Belgium

Contact

Follow

  • LinkedIn Social Icon
  • Twitter Social Icon