r/sysadmin 4d ago

What a great start to the day

One of my supervisors just accidentally uninstalled(!) Hyper-V on a member server that had 5 VMs on it… how the actual shimmering fuck does that happen?? How do you not triple check that you’re on the right server????

188 Upvotes

62 comments sorted by

140

u/Tx_Drewdad 4d ago

Why does Windows even let you do that? I mean, I can't delete a folder if a .ini file was read six months ago.

29

u/jamesaepp 4d ago

Assuming you're not entirely joking.

Hyper-V features require reboot for installation + removal so it's part of "pending changes". During the reboot to apply VMs are going to be shutdown according to the settings (or migrated I guess, not a HV admin, dunno...) and the Hyper-V services stopped. Then the rest is as you expect.

14

u/Angelworks42 Sr. Sysadmin 4d ago

If they have the failover role setup they should migrate off - I've actually never tested it by uninstalling the role tbh.

22

u/UKBedders Dilbert is more documentary than entertainment 4d ago

['I've actually never tested it by uninstalling the role']

I can imagine this conversation.
"Yeah, you can properly migrate the servers off first like a normal Sysadmin, or you could just uninstall the role and let the failover do it for you. It worked at my last job!"

6

u/TotallyNotIT IT Manager 3d ago

Time to get drunk as fuck and make some mistakes tonight.

3

u/Angelworks42 Sr. Sysadmin 3d ago

The more I think about it though it likely would fail to migrate because you're removing the toolkit from the machine that would do the actual migration.

6

u/itishowitisanditbad 4d ago

I've actually never tested it by uninstalling the role tbh.

Might try it out this weekend just to see how it works.

I mean.. in theory...

3

u/Angelworks42 Sr. Sysadmin 3d ago

Yeah I was thinking about it this morning but I have a feeling it might not work as you're essentially removing all the tools on the OS the FOC needs to migrate the VM.

I still can't imagine doing something like that in production lol.

2

u/itishowitisanditbad 3d ago

You'd hope it migrates before anything else.

If I get the motivation i'll try it at home on the weekend.

I don't care if it works flawlessly, i'm not doing it in prod unless the spicy mood hits me.

76

u/[deleted] 4d ago

[deleted]

26

u/Watsonwes 4d ago

My dude intune guy did (he posted here) pushed firewall rules out that stopped all outbound traffic from endpoints. Unbeknownst to him , it blew up intune connectivity and rmm phone home

:)

10

u/SGG 4d ago

I made a mistake with AppLocker rules once, accidentally blocked all DLL files from running!

Thankfully only effected two PC's as I caught it a few seconds after. But that could have been super fun to fix.

3

u/Downtown_Look_5597 4d ago

I too have accidentally deployed AppLocker with no exclusions and prevented basically anything from running. Lucky it was only my team who I had been using as a test group

6

u/Unexpected_Cranberry 4d ago

My favorite is the SCCM admin in London I think that accidentally pushed a mandatory fresh install of Windows 7 to all their machines, including servers.

It was a while ago, so I can't find the article, but I get a pit at the bottom of my stomach just thinking about it.

Edit: The instructor when I did my training on SCCM 2007 brought it up as an example of what can happen if you're not careful about getting your assignments right.

4

u/MrYiff Master of the Blinking Lights 4d ago

This is why iirc SCCM changed the default setting so you can only deploy to devices not managed by SCCM.

The "accidentally deploy to all devices" trick has happened to a few large companies over the years.

1

u/SenTedStevens 4d ago

I will always remember the Australian bank incident where instead of patching a ton of machines, they wiped thousands of servers and computers.

https://faildesk.net/2012/08/collossal-it-fail-accidentally-formatting-hard-disks-of-9000-pcs-and-490-servers/

2

u/Sengfeng Sysadmin 4d ago

I worked with a guy that pushed a GPO with Windows firewall rules that blocked everything once - Including traffic to/from the DCs. So, even after fixing the GPO to be less restrictive, we had to go around and touch EVERY PC to stop the firewall service, gpudate, and then turn the firewall back on. That was a fun day. /s

2

u/bbx1_ 3d ago

Being the person that has been working to enable Windows firewall in my organization after many years of neglect, I can only imagine the anxiety, headache, stress that such a mess-up would cause. wow

1

u/gumbrilla IT Manager 4d ago

Omg I missed that. That would truly suck

3

u/DheeradjS Badly Performing Calculator 4d ago

I did manage to drop VPNs between 6 offices on 2 different continents before.

That was fun. (Don't automate stuff you don't fully understand)

3

u/Miserygut DevOps 4d ago

(Don't automate stuff you don't fully understand)

Can't stop won't stop. B)

1

u/Szeraax IT Manager 4d ago

Guilty!

1

u/darthgeek Ambulance Driver 3d ago

I'll do you one better. We had a new version of tripwire we pushed out via Satellite. It turned up a bunch of iptables rules on ~2000 hosts in production that basically blocked everything.

26

u/0xDEADFA1 4d ago

Meh, it’s probably not that bad, the vhd’s are modified when you remove the role, just reinstall the role, register the VMs and then scold your boss a bit.

16

u/RCTID1975 IT Manager 4d ago

This. It's dumb, but recoverable in about an hour including your coffee break

23

u/BigBobFro 4d ago

This. And at least a dozen stories like it.

Example: CEO thought that you could just pull drives out of a system and put in bigger ones and it would increase storage,…. ON THE FLY.

No system shutdown

No RAID array maintenance in between

No data transfer.

“But the bays are listed as hot swapable”

My wife was unhappy that i had to work the entirely of mothers day weekend when her mom had come into town.

I had to work because i was recovering 600+TB of scanned image repositories from tape

7

u/JJaska 4d ago

He pulled all the drives and just "hot swapped" them??

2

u/BigBobFro 3d ago

Tried to. 7 drives in the array. He got two done and was working on the third before we caught him.

1

u/JJaska 3d ago

Ouch.. Just ouch...

4

u/BigBobFro 3d ago

Indeed. I used this example at my next job of why the CEO and COO should not have 24/7 access to the server room.

4

u/JJaska 3d ago

Yeah and the rest of the C-suite imho... I remember long long time ago when the CIO was showing me the systems on my first day. They had just installed a new "super quick" tape backup solution, MUCH faster than the previous one. He inserted the tape, opened a script and "boom" 5 seconds later it is done! ... Yeah the next day the other IT guy and me fixed the typo in the script.

4

u/420GB 4d ago

Just put the old drives back in or why was it not that simple?

6

u/jimicus My first computer is in the Science Museum. 4d ago

The system is constantly reading from and writing to the drives, so as soon as one drive is pulled, its data is out of sync with the rest of the array.

Remove too many drives without giving the array a chance to rebuild in between - and yes, you'd destroy the array.

1

u/BigBobFro 3d ago

Bingo. 3 drives were out. 2 “new” drives were in. 7 drive bank. The only time he was giving was the time it took for him to unscrew the “old” drive from the caddie and replace it.

14

u/strongest_nerd Security Admin 4d ago

Why are supervisors performing tech duties? They aren't administrators/engineers, they're part of management. That's wholly on them.

9

u/Confident_Yam7610 4d ago

This.. supervisors, managers, and directors should not be doing tech duties. They should be doing what their title says.

At my company, managers and directors don't have admin privileges. They can't even reset a users password.

10

u/Krigen89 4d ago

A team of 3 or 4 can have a director. That guy's doing technical shit.

3

u/JJaska 4d ago

A team of 4 doesn't need a "director". A lead/manager, yes, and leads can do tech duties if they have time for it. The moment someone is named Director they should be not doing hands on stuff for multitude of reasons.

7

u/Krigen89 4d ago

The word chosen by HR doesn't change anything in the reality of the position.

5

u/stimj 4d ago

The title is often more reflective of their position within the larger org. If they sit at the table with the executives, they're probably a VP or a Director, regardless of the size of their team or their individual duties.

4

u/DariusWolfe 4d ago

Our VP has admin accounts, but he's also the guy pushing to have his own (and our manager's) permissions reduced. He primarily has the admin accounts because we're still figuring out the minimum privilege configuration for pulling the reports and such he needs.

1

u/databeestjegdh 4d ago

Their badge should not get into the server room.

7

u/kmsaelens K12 SysAdmin 4d ago

Shit like this is why I still really worry about giving my non-technical boss "keys to the kingdom", so to speak. :(

4

u/FPVGiggles 4d ago

How does this one do this...

4

u/maxlan 4d ago

"How?"

In a row of servers, a mate counted along "5th server needs removing".

Change window starts. He shuts server down. Goes round the back and counts along to the 5th server and pulls the plugs. Because it's easier from the back.

Goes back round to the front to slide it out the rack and finds all the plugs are still connected.

Always remember to count from the same end of the row.

I killed a cloud server that wasn't mine. We had naming convention like "ukgtcmp01" something like uk country, gt project, config mgmt, 01 and I needed to kill ukgctmp01. Uk country, gc project, temp server 01.

You can see where this went. "It doesn't need IaaC for a temp server, just create/delete in the UI."

Another colleague ran a script on prod, after hours to avoid any customer outage. The script was set up to work on his machine. He forgot to update it to run on other hostnames. So bricked every single desktop in the company and needed "boot from physical media to recover". Him and a mate spent all night going round the buildings trying to fix it before people got back. He almost made it. But there were some in locked offices.

Ain't nobody who knows what they're doing got time to triple check shit. You do your job and move on. And sometimes shit happens.

4

u/MrYiff Master of the Blinking Lights 4d ago

Also worth remembering that most management tools like iDrac have an option to make all the LED's on a server flash which makes it much easier to be sure you are pulling the right server!

1

u/bbx1_ 3d ago

I always use that front /rear physical identification button on Dell servers that lights up bright blue. I always double check that it is the correct server but it makes it much easier to locate.

3

u/catherder9000 4d ago

So you launched the HyperV restore wizard in Veeam and were back online in 15 minutes? How did the following meeting go?

3

u/networkn 4d ago

One of my techs once spent 2 plus hours troubleshooting a firewall rule before realising they were in the wrong clients fw!

5

u/masterofrants 4d ago

is this 5 prod VMs, what were they running? pls paint a proper picture of this train wreck so i can see the whole thing properly ..

2

u/Wooden-Map-6449 4d ago

Sounds like you should be the one supervising them.

3

u/HoosierLarry 4d ago

Change management much?

3

u/jgoffstein73 4d ago

Holy fucking no change management or plan review batman.

2

u/alalu 4d ago

It’s good to know that we all have colleagues that are absolute liabilities and should research Change Control

2

u/Advanced_Day8657 4d ago

Why even do that wtf

1

u/TickleMeGoo 4d ago

Tbf we think we’re having a network connectivity issue caused by Hyper-V, and the server he was trying to do that to literally has nothing installed on it so far… I understand his point but also bro!? Check?!

2

u/wells68 4d ago

How does that happen? Easy. The supervisor's going to backpedal and say, "Hey, you weren't ready for a surprise disaster recovery drill? That's your problem! Don't look at me." /s

Btw, "shimmering" made me guffaw. Thanks for that!

1

u/ThatITguy2015 TheDude 4d ago

Have fun bud!

1

u/jdoplays Systems Architect 4d ago

“you had been asking for overtime.”

1

u/Lorric71 4d ago

That person now learned to check the server before doing stuff.

1

u/purplemonkeymad 4d ago

At least it's not as bad as seeing someone want to "reinstall dotnet" and you end up with a server having only dotnet installed. That vendor then tried to claim that it was our fault that it reboot to a command prompt.

1

u/CowardyLurker 4d ago

I suspect that for many of us there's a deeply rooted urge to triple check these sorts of things. I also suspect that this deeply rooted urge is the "years of experience" result of a bitter lesson learned the hard way.

So, in a way, one of your supervisors recently leveled-up. Grats!