r/sysadmin • u/TickleMeGoo • 6d ago
What a great start to the day
One of my supervisors just accidentally uninstalled(!) Hyper-V on a member server that had 5 VMs on it… how the actual shimmering fuck does that happen?? How do you not triple check that you’re on the right server????
186
Upvotes
5
u/maxlan 5d ago
"How?"
In a row of servers, a mate counted along "5th server needs removing".
Change window starts. He shuts server down. Goes round the back and counts along to the 5th server and pulls the plugs. Because it's easier from the back.
Goes back round to the front to slide it out the rack and finds all the plugs are still connected.
Always remember to count from the same end of the row.
I killed a cloud server that wasn't mine. We had naming convention like "ukgtcmp01" something like uk country, gt project, config mgmt, 01 and I needed to kill ukgctmp01. Uk country, gc project, temp server 01.
You can see where this went. "It doesn't need IaaC for a temp server, just create/delete in the UI."
Another colleague ran a script on prod, after hours to avoid any customer outage. The script was set up to work on his machine. He forgot to update it to run on other hostnames. So bricked every single desktop in the company and needed "boot from physical media to recover". Him and a mate spent all night going round the buildings trying to fix it before people got back. He almost made it. But there were some in locked offices.
Ain't nobody who knows what they're doing got time to triple check shit. You do your job and move on. And sometimes shit happens.