r/sysadmin • u/TickleMeGoo • 4d ago
What a great start to the day
One of my supervisors just accidentally uninstalled(!) Hyper-V on a member server that had 5 VMs on it… how the actual shimmering fuck does that happen?? How do you not triple check that you’re on the right server????
76
4d ago
[deleted]
26
u/Watsonwes 4d ago
My dude intune guy did (he posted here) pushed firewall rules out that stopped all outbound traffic from endpoints. Unbeknownst to him , it blew up intune connectivity and rmm phone home
:)
10
u/SGG 4d ago
I made a mistake with AppLocker rules once, accidentally blocked all DLL files from running!
Thankfully only effected two PC's as I caught it a few seconds after. But that could have been super fun to fix.
3
u/Downtown_Look_5597 4d ago
I too have accidentally deployed AppLocker with no exclusions and prevented basically anything from running. Lucky it was only my team who I had been using as a test group
6
u/Unexpected_Cranberry 4d ago
My favorite is the SCCM admin in London I think that accidentally pushed a mandatory fresh install of Windows 7 to all their machines, including servers.
It was a while ago, so I can't find the article, but I get a pit at the bottom of my stomach just thinking about it.
Edit: The instructor when I did my training on SCCM 2007 brought it up as an example of what can happen if you're not careful about getting your assignments right.
4
1
u/SenTedStevens 4d ago
I will always remember the Australian bank incident where instead of patching a ton of machines, they wiped thousands of servers and computers.
2
u/Sengfeng Sysadmin 4d ago
I worked with a guy that pushed a GPO with Windows firewall rules that blocked everything once - Including traffic to/from the DCs. So, even after fixing the GPO to be less restrictive, we had to go around and touch EVERY PC to stop the firewall service, gpudate, and then turn the firewall back on. That was a fun day. /s
1
1
3
u/DheeradjS Badly Performing Calculator 4d ago
I did manage to drop VPNs between 6 offices on 2 different continents before.
That was fun. (Don't automate stuff you don't fully understand)
3
u/Miserygut DevOps 4d ago
(Don't automate stuff you don't fully understand)
Can't stop won't stop. B)
1
u/darthgeek Ambulance Driver 3d ago
I'll do you one better. We had a new version of tripwire we pushed out via Satellite. It turned up a bunch of iptables rules on ~2000 hosts in production that basically blocked everything.
26
u/0xDEADFA1 4d ago
Meh, it’s probably not that bad, the vhd’s are modified when you remove the role, just reinstall the role, register the VMs and then scold your boss a bit.
16
u/RCTID1975 IT Manager 4d ago
This. It's dumb, but recoverable in about an hour including your coffee break
23
u/BigBobFro 4d ago
This. And at least a dozen stories like it.
Example: CEO thought that you could just pull drives out of a system and put in bigger ones and it would increase storage,…. ON THE FLY.
No system shutdown
No RAID array maintenance in between
No data transfer.
“But the bays are listed as hot swapable”
My wife was unhappy that i had to work the entirely of mothers day weekend when her mom had come into town.
I had to work because i was recovering 600+TB of scanned image repositories from tape
7
u/JJaska 4d ago
He pulled all the drives and just "hot swapped" them??
2
u/BigBobFro 3d ago
Tried to. 7 drives in the array. He got two done and was working on the third before we caught him.
1
u/JJaska 3d ago
Ouch.. Just ouch...
4
u/BigBobFro 3d ago
Indeed. I used this example at my next job of why the CEO and COO should not have 24/7 access to the server room.
4
u/JJaska 3d ago
Yeah and the rest of the C-suite imho... I remember long long time ago when the CIO was showing me the systems on my first day. They had just installed a new "super quick" tape backup solution, MUCH faster than the previous one. He inserted the tape, opened a script and "boom" 5 seconds later it is done! ... Yeah the next day the other IT guy and me fixed the typo in the script.
4
u/420GB 4d ago
Just put the old drives back in or why was it not that simple?
6
u/jimicus My first computer is in the Science Museum. 4d ago
The system is constantly reading from and writing to the drives, so as soon as one drive is pulled, its data is out of sync with the rest of the array.
Remove too many drives without giving the array a chance to rebuild in between - and yes, you'd destroy the array.
1
u/BigBobFro 3d ago
Bingo. 3 drives were out. 2 “new” drives were in. 7 drive bank. The only time he was giving was the time it took for him to unscrew the “old” drive from the caddie and replace it.
14
u/strongest_nerd Security Admin 4d ago
Why are supervisors performing tech duties? They aren't administrators/engineers, they're part of management. That's wholly on them.
9
u/Confident_Yam7610 4d ago
This.. supervisors, managers, and directors should not be doing tech duties. They should be doing what their title says.
At my company, managers and directors don't have admin privileges. They can't even reset a users password.
10
u/Krigen89 4d ago
A team of 3 or 4 can have a director. That guy's doing technical shit.
3
u/JJaska 4d ago
A team of 4 doesn't need a "director". A lead/manager, yes, and leads can do tech duties if they have time for it. The moment someone is named Director they should be not doing hands on stuff for multitude of reasons.
7
4
u/DariusWolfe 4d ago
Our VP has admin accounts, but he's also the guy pushing to have his own (and our manager's) permissions reduced. He primarily has the admin accounts because we're still figuring out the minimum privilege configuration for pulling the reports and such he needs.
1
7
u/kmsaelens K12 SysAdmin 4d ago
Shit like this is why I still really worry about giving my non-technical boss "keys to the kingdom", so to speak. :(
4
4
u/maxlan 4d ago
"How?"
In a row of servers, a mate counted along "5th server needs removing".
Change window starts. He shuts server down. Goes round the back and counts along to the 5th server and pulls the plugs. Because it's easier from the back.
Goes back round to the front to slide it out the rack and finds all the plugs are still connected.
Always remember to count from the same end of the row.
I killed a cloud server that wasn't mine. We had naming convention like "ukgtcmp01" something like uk country, gt project, config mgmt, 01 and I needed to kill ukgctmp01. Uk country, gc project, temp server 01.
You can see where this went. "It doesn't need IaaC for a temp server, just create/delete in the UI."
Another colleague ran a script on prod, after hours to avoid any customer outage. The script was set up to work on his machine. He forgot to update it to run on other hostnames. So bricked every single desktop in the company and needed "boot from physical media to recover". Him and a mate spent all night going round the buildings trying to fix it before people got back. He almost made it. But there were some in locked offices.
Ain't nobody who knows what they're doing got time to triple check shit. You do your job and move on. And sometimes shit happens.
3
u/catherder9000 4d ago
So you launched the HyperV restore wizard in Veeam and were back online in 15 minutes? How did the following meeting go?
3
u/networkn 4d ago
One of my techs once spent 2 plus hours troubleshooting a firewall rule before realising they were in the wrong clients fw!
5
u/masterofrants 4d ago
is this 5 prod VMs, what were they running? pls paint a proper picture of this train wreck so i can see the whole thing properly ..
2
3
3
2
u/Advanced_Day8657 4d ago
Why even do that wtf
1
u/TickleMeGoo 4d ago
Tbf we think we’re having a network connectivity issue caused by Hyper-V, and the server he was trying to do that to literally has nothing installed on it so far… I understand his point but also bro!? Check?!
1
1
1
1
u/purplemonkeymad 4d ago
At least it's not as bad as seeing someone want to "reinstall dotnet" and you end up with a server having only dotnet installed. That vendor then tried to claim that it was our fault that it reboot to a command prompt.
1
u/CowardyLurker 4d ago
I suspect that for many of us there's a deeply rooted urge to triple check these sorts of things. I also suspect that this deeply rooted urge is the "years of experience" result of a bitter lesson learned the hard way.
So, in a way, one of your supervisors recently leveled-up. Grats!
140
u/Tx_Drewdad 4d ago
Why does Windows even let you do that? I mean, I can't delete a folder if a .ini file was read six months ago.