About a year ago, it was a promising morning, everything seemed fine. Then I got a message from my manager:
“Take down the entire staging environment. Let’s test on dev to save costs.”
I was in a hurry to get it done. What happened next was a nightmare.
Our infrastructure was hosted on Microsoft Azure, and like most DevOps engineers, I had structured everything neatly into Resource Groups (RGs) — one for each environment.
In my rush, I went straight to the console and deleted the entire staging RG…
Unknown to me, one of our developers had created a production Azure Storage Account inside that same staging RG.
A few seconds later, production was down.
In fact, all environments were down.
I panicked. It felt like the sky was falling. For a moment, I wanted to just disappear.
After a few moments of chaos (and a call with a teammate that didn’t help much), I forced myself to calm down. That’s when a thought hit me:
“Azure can’t be that careless. There must be a way to recover deleted storage.”
So, I started digging through documentation and community forums… and after some research, I finally found a way to restore the deleted storage account.
That day, I wrote a note and pinned it to my mirror:
“Never delete a Resource Group without thoroughly inspecting all the resources inside.”
The lesson?
Always enable Deletion Protection (or soft delete, versioning, etc.) on critical resources.
It might just save you from a heart attack, or at least from “shitting your pants.” 💩😅
We learn every day.
#DevOps #Azure #CloudEngineering #MistakeToMastery #LearningCurve #DeletionProtection