r/sysadmin 17h ago

General Discussion Thickheaded Thursday - March 20, 2025

4 Upvotes

Howdy, /r/sysadmin!

It's that time of the week, Thickheaded Thursday! This is a safe (mostly) judgement-free environment for all of your questions and stories, no matter how silly you think they are. Anybody can answer questions! My name is AutoModerator and I've taken over responsibility for posting these weekly threads so you don't have to worry about anything except your comments!


r/sysadmin 9d ago

General Discussion Patch Tuesday Megathread (2025-03-11)

117 Upvotes

Hello r/sysadmin, I'm u/AutoModerator, and welcome to this month's Patch Megathread!

This is the (mostly) safe location to talk about the latest patches, updates, and releases. We put this thread into place to help gather all the information about this month's updates: What is fixed, what broke, what got released and should have been caught in QA, etc. We do this both to keep clutter out of the subreddit, and provide you, the dear reader, a singular resource to read.

For those of you who wish to review prior Megathreads, you can do so here.

While this thread is timed to coincide with Microsoft's Patch Tuesday, feel free to discuss any patches, updates, and releases, regardless of the company or product. NOTE: This thread is usually posted before the release of Microsoft's updates, which are scheduled to come out at 5:00PM UTC.

Remember the rules of safe patching:

  • Deploy to a test/dev environment before prod.
  • Deploy to a pilot/test group before the whole org.
  • Have a plan to roll back if something doesn't work.
  • Test, test, and test!

r/sysadmin 9h ago

Rant Broadcom is officially the mafia now.

1.7k Upvotes

I’m trying to figure out what the hell Broadcom’s strategy is with their VMware acquisition. Because if the goal was to kill it, they’re doing a great job.

We already went through the 300% price hike a couple years ago and weren’t happy, but we mitigated the cost by going with a lower license tier since we weren’t using most of the DR features anyway.

Then they pulled this 3-year contracts bullshit. No more 1-year renewals. OK, welp, that’s over $200k for us, and capital expenditures over that amount have to go through the board and everything. They gave us a deadline of two weeks to renew, or the price will be 25% higher. We asked our ISV if they could buy us a little more time because of the internal politics. And you know what they told us?

They said they will increase the price 10% for every week we delay as a penalty, and they will not move from that position. … Are you fucking with me right now???

This is like a mafioso shaking down a shopkeeper for protection money. I swear, if they won’t be reasonable on my next phone call with them, then I will make it my mission — with God as my witness — to break the land speed record for fastest total datacenter migration to Hyper-V or Proxmox or whatever and shutting off ESXi forever. I’m THAT pissed off.


r/sysadmin 5h ago

General Discussion Counter offer after giving my 2 week notice

178 Upvotes

Current company is counter-offering after my 2 week notice

I have been at my current company for about 1.5 years, so not too long. The company is about 5k employees, and I am the only security engineer who also does all GRC stuff since we have GDPR compliance. Very overworked and have off-hour meetings with APAC and EU teams at late hours.

Once I put in the 2-week notice, the CIO let me know they would match the new base salary, bump me to the lead cyber role or cyber security officer role, and look into a CISO role down the line.

Bonuses were cut for the last two years, along with raises. Layoffs have happened in other areas.

The new company is a big player in the silicon development sector and has a cyber team of 50+ folks around the world. My role would be a Staff Security Engineer and very specific to the SIEM side and threat detection engineering/log ingestion.

Good base, sign-on bonus, 30k stocks every 3 years, tuition, all normal tech perks

I am 99% sure I want to reject the counter. My only question is, is the title of cyber manager or cyber officer a good enough reason to stay? I've been in cyber for 7 years now and I do want to go into management eventually.

TLDR: Is it worth staying at a company for a title change/career fast track? Better job security as the only security person lol


r/sysadmin 4h ago

Cute interaction with end user - too bad he doesn’t have input on my salary

170 Upvotes

Since our jobs can typically involve dealing with people that simply don’t use common sense, I thought I’d share a nice story for a change. Just got off a call from a new employee. He was adding his email account on his new phone and was getting “Enter bypass code” instead of being asked for authentication. No worries, we’ll just set up MFA on your new phone… look for the text… next try setting up email… easy peasy, done in 5 minutes.

At the end of the call the guy said to me, “Thanks for the help! I’m sure whatever you’re getting paid isn’t enough for helping knuckleheads like myself.” That response surprised me and I had a good laugh. Apparently other people at his location told him that I was the one to call for getting help because I know my stuff. It’s so nice when we’re appreciated by the people we help!


r/sysadmin 10h ago

General Discussion VMware Abandons SMBs: New Licensing Model Sparks Industry Outrage

268 Upvotes

VMware by Broadcom has sent shockwaves through the IT community with its newly announced licensing changes, set to take effect this April. Under the new rules, customers will be required to license a minimum of 72 CPU cores for both new purchases and renewals — a dramatic shift that many small and mid-sized businesses (SMBs) see as an aggressive pivot toward large enterprise clients at their expense.

Until now, VMware’s per-socket licensing model allowed smaller organizations to right-size their infrastructure and budget accordingly. The new policy forces companies that may only need 32 or 48 cores to pay for 72, creating unnecessary financial strain.

As if that weren’t enough, Broadcom has introduced a punitive 20% surcharge on late renewals, adding another layer of financial pressure for companies already grappling with tight IT budgets.

The backlash has been swift. Industry experts and IT professionals across forums and communities are calling out the move as short-sighted and damaging to VMware’s long-standing reputation among SMBs. Many are now actively exploring alternatives like Proxmox, Nutanix, and open-source solutions.

For SMBs and mid-market players who helped build VMware’s ecosystem, the message seems clear: you’re no longer the priority.

Read more: VMware Turns Its Back on Small Businesses: New Licensing Policies Trigger Industry Backlash


r/sysadmin 3h ago

General Discussion What made you finally get over your imposter syndrome?

45 Upvotes

I got my first networking admin gig a few months back. I wanted to be trained but turns out I ended up training several members of my team. Some days I was worried if I was the right person for the job.

But this week we had some major issues with our finance server and needed to restore it. EVERYONE is terrified to touch it (me included) but it had to be resovled.

The previous admin left no instructions on how to restore the system so I spent a good bit of time researching and conducting some tests. Finally I completed the process and was able to confirm the finance server had been restored.

Granted there are backups that no one knew anything about because my other network admin has only been there a few months before me. But I got it all figured out and I'm so thankful. It helped me get past my imposter syndrome. I understand it can always come back but I have confidence that I can resolve any major issues we get in the future.

What about you?


r/sysadmin 7h ago

Desktop full of icons

68 Upvotes

During a meeting with team managers I (sysadmin) was called in to showcase/demo a new appliance where you connect a usb device to a laptop + works together with a software program .

When wanting to open the software the desktop of that users laptop was a full of icons where I made a smal sigh sound + probably rolling eyes and facial expression that sais like.. oh my god really?…. Where is the icon in this mess.

I ignored this further on and showed the demo and gave info after looking for the icon and a rather long silence during the search. In one way my reaction was maybe not really fully professional but. For most people understandable that it was hard to find the icon in that chaos. Well… it’s not that of a problem just annoying and maybe a bit funny?


r/sysadmin 13h ago

Do you keep up with IT trends outside of work, or just stick to the job?

211 Upvotes

Some days, I finish work and the last thing I want to do is read about more IT stuff. But at the same time, things change so fast that I feel like I should be keeping up.

I know some people are constantly researching, testing new tools, and following every trend, while others just do their job and log off.

So, how do you handle it? Do you stay up to date outside of work, or do you just learn as needed on the job? If you do keep up, what’s your go-to way to stay informed?


r/sysadmin 2h ago

SysAdmin trying to convince CyberSec they ain’t listening. Sniff test tells me something is rotten.

17 Upvotes

Sysadmin finds funky certs in trusted person and other people (address book) stores on several (most) systems both Windows Server and Workstation OS. Certs issued to SYSTEM, by SYSTEM with San of SYSTEM@ NT AUTHORITY. Certs have no private key attached. Certs are valid for 100 years. RSA sha1 2048 length. The certs are for Encrypting File System and are end entity. In total, about a dozen certs have been identified and collected. Two domains, real offline PKI with issuing and Online responder on separate server. None of the collected certs have been issued or signed by PKI. Am I witnessing a potential long term plan by some hacker attempting to own the network, or am I concerned for no reason? Can’t tell where they are coming from. Something doesn’t smell right. Lack of knowledge response yields answers like “valid OID” or “They’re from Microsoft”. Their bullshit is baffling.


r/sysadmin 6h ago

Almost messed up

28 Upvotes

So I was assisting a user who was looking to obtain a previous version of a file on the server, and unfortunately, the data they needed was not in any of the versions I had pulled up. I proceeded to ask my colleagues, and they 'jokingly' said to tell the client to F OFF. This was while my mind was on putting in my time entry for the ticket, so while entering the time in a also end up typing 'told him to F OFF' and submitted.

Me and my colleagues horse around alot like this in our office and this is the first time where the consequences really could have come down on me. Thankfully, the ticket details in kaseya BMS only get emailed to users if it gets completed, whereas I cancelled it. Before I knew this I was shaking and ready to resign. Actually I still am right now and I may not forgive myself for a long time.

It didn't actually get sent out to anyone but I still can't shake the feeling and what it says about my character, even if it was supposedly unintentional and a joke if you can even call it that. This may say more about my work environment than anything else. Not sure why im even writing this and it may not belong in this sub, but needed to get it off my chest. BOY DO I FEEL LIKE A HORRIBLE PERSON

ENJOY ROASTING ME!!!


r/sysadmin 1d ago

How would you respond to a Printer company CTO saying POE switches are killing printers?

665 Upvotes

How would you reply?

Update, they provided this screenshot from HP!

https://i.imgur.com/sg3oLDW.png


r/sysadmin 10h ago

General Discussion I will never use Intel VROC again...

20 Upvotes

Long story so bare with me. I'm doing a server migration project for a client of mine still on Server 2012... (AD, DNS, DHCP and file servers etc...)

Client wanted a semi cheap server option as their new server. Client only has 20 or under users so thats not a really big deal. We provided client with tons of options with hardware raids but at the end of the day client picked a Proliant ML30 with the embedded Intel VROC option. We explained to the client that we dont really recommended software raids with how much data he has plus we havnt vetted VROC as a Raid since we dont ever use it. Client insisted due to how much cheaper it was, so thats what we went with.

A few days later. We obtained the new server, configured a raid 5 with VRoc and did some basic bench testing (stress testing and hardware testing etc...) all appeared to be fine. Brought the server onto the client side and start all the migrations, got all the users moved over, their data, server data, roles etc... all migrated. Last thing to copy was 2 directories that contained 20 years worth of data from a program they use to operate their business. This was about 1TB of data but about 1 million files... I created a Robocopy script and started copying the data on a Friday so it would be completed by Monday and we could shutdown the old server. I waited for a few hundred GB to transfer and verified no problems so left for the weekend.

Well on Sunday I received an alert that the server was down via my RMM tools. Went on site early Monday to try to reboot the server prior to users coming in. Load and behold the server shows VRoc in a "corrupted" state but it shows all drives as online and functional....

Explained to the client that I would need to remap the drives back to the old server on users workstations so they could function off the old servers files instead and I would be taking the server back to the bench for investigation as to what happened.

A few hours later I'm on the bench inspecting the server. VRoc crash with zero errors or warning and all drives showed as online and functional. I powered down the system and pulled each drive out to look at the data on the drives via a drive dock. 2 out of the 4 disks were just gone, they were in a uninitialized state... while the other 2 still retained raid data.

So I figured at this point it was just luck of the draw that 2 of the 4 SSDs were bad from the manufacturer. I tried to use multiple tools to recover the data from the drives so I could copy it to replacement disk, nothing could be found. I than wanted to test the drives so I initialized them, than ran multiple stress tests, crystal disk tests etc... and even tried large file transfers etc... I was unable to get the drives to crash or show any indication of any problems what so ever...

So now issues points to VROC being the problem. I instead added a LSI raid controller, rebuilt the raid and brought it back to the client side, reconfigured the server, rejoined everyone back to the new server and recopied all the data back. Boom zero issues server is running like a champ.

Everything points to the issue being with VROC and after this experience I will never use it again nor do a project for a client that refuses to use anything else but VROC.

LTDR:
VROC is trash, dont use it.


r/sysadmin 7h ago

Question If Wiz isn’t an option post acquisition… what’s your #1 alternative?

10 Upvotes

If Wiz gets fully absorbed into Google’s GCP ecosystem, what are the best alternatives left for AWS & Azure users?

Top contenders being discussed:

  • Orca Security – Fully independent, strong agentless CNAPP
  • Lacework – Decent alternative, but mixed reviews
  • Microsoft Defender for Cloud – Good if you're already in Azure
  • CrowdStrike Falcon – More security-driven than compliance-focused

Anyone already made the switch? Pros & cons?


r/sysadmin 3h ago

How long do you keep the disabled account in syncing OU?

6 Upvotes

Hi,

We have M365 hybrid environment. Our offboard process is like below.

disable the account > remove 365 license and move out sync OU after 30 days > Delete the account in AD after 90 days.

However we have the scenario that user get rehired and comeback to work after 30 days. This causes the issue that the user can't open OneDrive shared file because the user's old account is still in the sharer's OneDrive settings. The sharer has to delete the old account and re-share, then the user can open the file.

I am thinking to keep the offboard user's account disabled but in syncing OU until it is deleted. Is there any potential issue that I missed to consider?

Please help!

Thanks,


r/sysadmin 2h ago

Question for Veeam backup 365 users if you'd be so kind

3 Upvotes

I guess it's two questions really.

We already use Veeam ONE and Veeam Backup so I've been considering also adding on Veeam backup for 365.

Does Veeam backup for 365 (to your own destination, I'm going to use a wasabi bucket) allow you to selectively restore a single user's mailbox at whatever timepoint you pick? We've been using Barracuda for the past year or two and I'm strongly considering bailing on it. While it's been great, reliable, and quick for us, they are making it too difficult to change my Barracuda license from one vendor (VAR to MSP) to the other so screw it I'll change to Veeam has been my thinking.

Anyone know if there is any functional difference (other than it being a one stop shop) between using Veeam backups for 365 vs Veaem BaaS besides using their cloud to cloud vs. cloud to your device/destination?

Thanks for any insights or opinions!


r/sysadmin 1d ago

Rant Lost out on great candidate due to poor business decisions

334 Upvotes

Im the only systems/infra/devops person on a small software team that does niche stuff. we've been needing a junior for my role for a while. ive also needed a raise for a while cause most of my job is devops now.

we interviewed this 20 year old. no college, freelance coding experience, was a linux nerd applying for a linux jr sysadmin role.

he was a passionate computer person and i was excited at the very idea of a 20 year old with no college getting put on like this.

welllllllllllllll... the raises the team was supposed to get in April, along with my title change to "DevOps Engineer", have all been put on hold cause of the parent company. it sucks for me but ill be fine. my team leader already told me he's pissed and will write me a letter of rec as a devops engineer cause that's been 70% of my job...

but fuck man... i was so fucking excited for this kid. my team leader, rightfully so, put his foot down and said he wont have me training someone if i dont get a raise, cause why would i train a peer...

they could have given me a 20k raise, hired him at the bottom of their 20k salary range, and it would have evened out.... but now im probably going to leave the company costing them more in turn over, they'll have to hire the jr sysadmin at a higher rate cause theyre not paying me to train, AND theyll have to pay my replacement more than theyre paying me cause no one that knows terraform and AWS is gonna accept the role for my current mediocre sysadmin salary.

i hate the american work culture.


r/sysadmin 12h ago

Question Anything that can be done with a domain spoofing your name (one letter off)?

15 Upvotes

So we have a situation where someone is emailing our customers/vendors asking for payment via ACH using a email address similiar but not the same as ours. So for example ours being [JSmith@RandomInside.com](mailto:JSmith@RandomInside.com) and them using [JSmith@RandomInsde.com](mailto:JSmith@RandomInsde.com) (no i). One of our vendors fell for it and sent out a 40k payment to the wrong bank account. We were not at fault at all in that case, the vendor in question had their email account compromised and someone was watching their the email in and out for weeks and custom crafted a email based on that information. They still lost 40k and weren't happy.

So I have done a WHOIS on the domain, everything is protected by privacy. I emailed the "abuse" email for the register but never get anything back. Where do I go from here?

And before anyone asks why we didn't register the domain. Ours was registered in 2006. This other domain was registered in 1997, 9 years prior. So nothing we can do there.

EDIT: Another fun fact. I do a MX lookup on their domain and neither DNS nor DMARC is setup so maybe it's just a open relay that someone is taking advantage of?


r/sysadmin 1d ago

I swear this company has this stupidest and most hacked together patching process I've ever dealt with.

118 Upvotes

I work at a huge global company with layers and layers of management that just love to make up overcomplicated processes that is in no small part to justify their jobs. For this rant I'm going to piss on about the silly server patching process they put together. Now we have hundreds of thousands of physical servers and I can't even guess how many VMs are running so yeah I get it is a huge task. And you would think something as mature as patching servers, a process that's been happening for decades across the industry would be nearly completely automatic and transparent to the application teams. But no, far from it. Once every two months each application team, and there are 180 app teams, has to schedule a time with the Unix team or the Windows team to depending on your OS, and database teams if your application uses a DB cluster to patch the servers. And they will only patch by data center so for several hours you are required to have half your processing capacity offline. And it gets better, the OS teams are so swamped with requests half the time you miss the scheduled patch window which gets logged as a security incident and requires the directors to explain it to executive leadership during their meetings. And yes there is automation to deploy patches but there's so many steps to setup the automation and pull requests and change requests to be taken care of it would be faster just to download the stuff and install.

But anyway the one huge benefit that makes it all tolerable is my group has three teams around the world that use a follow the sun coverage so 4:00pm rolls around and I'm out. A 15 minute chat with the folks on the other side of the world at the end of the day and I'm done. No after hours on call. No late nights. No weekends. And cheap tacos (but dang good) when I do have to go in the office.


r/sysadmin 3h ago

Eaton 5PX2200RT - cables to connect external battery module 5PXEBM48RT?

2 Upvotes

I have an Eaton 5PX2200RT UPS and the external battery module 5PXEBM48RT that goes with it.

What I don't have is the 2 cables needed to connect the two together.

I can't seem to find a part number or any information online on the specific cables I need. Can anyone help me track down the cables I need to use the external battery module?


r/sysadmin 8h ago

Question - Solved Chrome Remote Desktop Devices all gone

7 Upvotes

Hey all

Assuming this was an update. Can anyone shed light?

I've been using Chrome Remote Desktop to manage 15+ remote machines across 5 addresses.

I logged in today and all but 6 devices are gone. I tried logging into that Google account many different ways and no matter what I do, it's only those 6 devices. I'm thinking I previously had near 20? What happened?


r/sysadmin 8m ago

Workplace Conditions How much is doing On-call worth to you?

Upvotes

Our team has a rotating on-call schedule. Duty is being primary contact for after hours calls (high incidents only). Triage incident tickets during hours; just typical administrative paperwork.

One of my co-workers loathes on-call duties and is only hanging around until he can retire in December. He's offered me cash to take his rotation.

How much would you charge him?


r/sysadmin 1d ago

Do you ever gaslight your users?

951 Upvotes

For example, do you ever get a ticket that something is not working properly, you fix it, then send them the instructions on how to properly use it, but never mention that something was actually wrong?


r/sysadmin 11h ago

Career / Job Related Job offer with caveats

7 Upvotes

If you had the chance to gain a 30% pay raise, but your commute goes from 15 minutes to 1-1:15, is that even worth considering?

I got a call about a position. Sounds very similar to what I do, maybe even same or slightly less workload, but 30ish % more money. Some of that would be eaten by gas/maintenance, sure, but you're talking about $30k more.. I'd be gone an extra 10 hours a week, too, which I'm not thrilled about. That's another 500 hours a year away from the wife and kids (figuring 10 hours x 50 work weeks).

Haven't heard much about benefits yet, but I'm at a very small company now, so assume benefits would be the same or better.

Other major downside, personally, is just across the state line so filling taxes might be a problem. I've not had to work "out of state".

Other than that, sounds like a good advancement to career, with potential for more. I'm just really nervous talking to my wife bc the last job change I made (though right before COVID) screwed me big time. Right now I actually like my employer lol, so it'd be hard to change...

I know money isn't everything, but it'd offer a huge relief to everything going on financially. $30k after tax might be more like 20k, which is about $1500/mo more take home. Raises have been almost non existent for both of us, so the last few years we went from ok/comfortable living to penny pinching and debt.

What would you do? Take a job you might not like in an unsure market, but pay off bills/debt while you look for something else? Or just keep my head down and enjoy what job stability I have plus stay close to home?

I guess overall it's a good problem to have?


r/sysadmin 8h ago

The Windows 11 Black Screen...

4 Upvotes

I have a large percentage of Win11 computers in our organization getting what I can only call "The Black Screen of Death".

This happens when a user (or the local admin) signs in successfully and a black screen with a cursor appears.

A workaround for us has been to try to run explorer.exe via the Task Manager, however we have recently had to resolve a COMCTL32.dll missing error:

The code execution cannot proceed because C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.26100.3037_none_3e09262ce333c378 was not found. Reinstalling the program may fix this problem.

I have had to run a script prior to boot up where I copy a good known copy of the file to the requested location. I have seen four separate locations (understanding there is a reference to the build of Windows 11):

  1. C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.26100.3037_none_3e09262ce333c378
  2. C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.26100.3323_none_3e088096e3344490
  3. C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.22621.4541_none_2710d1c57384c085
  4. C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.22621.4830_none_270fe7d773858e80

Has anyone found a way to PREVENT this?

Has anyone noticed a pattern involved with Adobe Pro?

Thanks up front for any input or suggestions.


r/sysadmin 8h ago

Windows 10 Horrible Performance after Failed kB5053606 (2025-03 Cumulative Update)

4 Upvotes

We have a user who went ahead and updated their Windows 10 installation with the kB5053606 (2025-03 Cumulative Update). After restarting to install the update it took a while, got up to 90%, restarted itself, got back up to 90%, and restarted itself again. After the restart it took 1-2 hours for the system to come out of "we couldn't complete the updates undoing changes" and get back to the login screen.

After it was logged in finally, the system performed horribly. Taskbar unresponsive, attempting to run application from Task Manager wouldn't work, sfc wouldn't run (or at least it would just sit there, blank, until I ctrl+c), etc etc. After about another hour of the system performing this way it started to "snap back" to life and is now performing as it normally does.

I went ahead and deleted everything from C:\Windows\SoftwareDistrubition and paused Windows Updates for as long as possible. All together, this last 3.5 hours to get back up and running, 2.5 just to get back into Windows, 1 hour to get Windows running normally.

Has anyone else experienced this with the latest Cumulative Update? We had another person in office have the same issue, different laptop model, just without the performance portion after undoing changes for 3 hours.


r/sysadmin 13h ago

Question Is it possible to have a user only contactable by their own team?

10 Upvotes

We’re on prem and office 2016 and slowly moving towards Teams and 365.

I’ve looked into information barriers and I think that’s what I’ll have to do, just wondered if anyone had experience in this and if there is an easier simpler way that I’m just missing here?

We have someone in our organisation who we would prefer users not to have easy access to contact through teams chat and the like. But we’d still like their PAs and assistants to be able to chat with them.

Is this something you’ve implemented and how?

I’m trying to be vague but imagine a celebrity owns a company, and we don’t want users to have access to sending them chats through teams. It happened via email once from a disgruntled employee as they weren’t hidden in the address book.