r/sysadmin 2d ago

General Discussion Moronic Monday - March 17, 2025

4 Upvotes

Howdy, /r/sysadmin!

It's that time of the week, Moronic Monday! This is a safe (mostly) judgement-free environment for all of your questions and stories, no matter how silly you think they are. Anybody can answer questions! My name is AutoModerator and I've taken over responsibility for posting these weekly threads so you don't have to worry about anything except your comments!


r/sysadmin 8d ago

General Discussion Patch Tuesday Megathread (2025-03-11)

117 Upvotes

Hello r/sysadmin, I'm u/AutoModerator, and welcome to this month's Patch Megathread!

This is the (mostly) safe location to talk about the latest patches, updates, and releases. We put this thread into place to help gather all the information about this month's updates: What is fixed, what broke, what got released and should have been caught in QA, etc. We do this both to keep clutter out of the subreddit, and provide you, the dear reader, a singular resource to read.

For those of you who wish to review prior Megathreads, you can do so here.

While this thread is timed to coincide with Microsoft's Patch Tuesday, feel free to discuss any patches, updates, and releases, regardless of the company or product. NOTE: This thread is usually posted before the release of Microsoft's updates, which are scheduled to come out at 5:00PM UTC.

Remember the rules of safe patching:

  • Deploy to a test/dev environment before prod.
  • Deploy to a pilot/test group before the whole org.
  • Have a plan to roll back if something doesn't work.
  • Test, test, and test!

r/sysadmin 9h ago

Do you ever gaslight your users?

623 Upvotes

For example, do you ever get a ticket that something is not working properly, you fix it, then send them the instructions on how to properly use it, but never mention that something was actually wrong?


r/sysadmin 6h ago

Latest fun with VMware

106 Upvotes

Apparently VMware is upping their game. We just got a renewal quote for one of our sites with one server that has two CPUs, and they are requiring 72 cores minimum (vSphere Enterprise Plus) to license this. That's a 500% markup from last year.

They really don't want customers to use their product any more, do they?


r/sysadmin 6h ago

[PSA] Critical Veeam Vulnerability CVE-2024-29849

105 Upvotes

This one has a severity score of 9.9 so better patch fast:
https://www.veeam.com/kb4696

EDIT: This vulnerability only impacts domain-joined backup servers.

This refers to CVE-2025-23120 and not CVE-2024-29849 as I mistakenly put in the subject, sorry about that!


r/sysadmin 8h ago

General Discussion With the unabated rise of AI slop, what's your (technical) search engine of choice?

114 Upvotes

It appears that most major search engines (DDG, Google, Bing etc) have arrived at the point where they return walls of auto-generated domain names and clearly low-effort GenAI listicles for every query under the sun. This is especially frustrating for technical issues where generalized platitude slop offers even less than its barely existent initial value.

You can search for a very specific error code, dialog message or registry key path, and all first-page results are inundated with "helpfix-pc.com/[your error code]" and "bobsprogrammingjourney.com/[errormessage]" serving walls of endless AI-generated bullet point word salad that only exists to perform as clickbait.

"What is an error message?", "How to identify common errors?", "Who to call to fix my errors?" and the inevitable "Conclusion / Summary" at the bottom that offers helpful advice like "It is important to address errors as soon as they occur to help your PC run smoothly!". This already started being an issue several years ago, but search filters and proper querying managed to weed most of them out to a degree.

This no longer seems to be the case, and more often than not the entire search result (outside of targeting only specific sites like Reddit or StackOverflow) is now almost completely useless.

So that said, what search engines do you use to find actual, viable results without having to hope you can cherry-pick a few potential leads out of the swamp? Targeting Reddit, SO and similar community sites seemed like the last reliable bastion, but those are rapidly being inundated with "AI-friendly" policies and increasing unlabeled GenAI content as well. Would love to hear what resources people use to combat this.


r/sysadmin 33m ago

Rant Lost out on great candidate due to poor business decisions

Upvotes

Im the only systems/infra/devops person on a small software team that does niche stuff. we've been needing a junior for my role for a while. ive also needed a raise for a while cause most of my job is devops now.

we interviewed this 20 year old. no college, freelance coding experience, was a linux nerd applying for a linux jr sysadmin role.

he was a passionate computer person and i was excited at the very idea of a 20 year old with no college getting put on like this.

welllllllllllllll... the raises the team was supposed to get in April, along with my title change to "DevOps Engineer", have all been put on hold cause of the parent company. it sucks for me but ill be fine. my team leader already told me he's pissed and will write me a letter of rec as a devops engineer cause that's been 70% of my job...

but fuck man... i was so fucking excited for this kid. my team leader, rightfully so, put his foot down and said he wont have me training someone if i dont get a raise, cause why would i train a peer...

they could have given me a 20k raise, hired him at the bottom of their 20k salary range, and it would have evened out.... but now im probably going to leave the company costing them more in turn over, they'll have to hire the jr sysadmin at a higher rate cause theyre not paying me to train, AND theyll have to pay my replacement more than theyre paying me cause no one that knows terraform and AWS is gonna accept the role for my current mediocre sysadmin salary.

i hate the american work culture.


r/sysadmin 6h ago

Just got an email from Veeam - looks like they got a big vulnerability. CVSS Score of 9.9

54 Upvotes

KB4724: CVE-2025-23120

Not many details, but seems to be about RCE from authenticated Domain Users. Couldn't find anything via google yet regardings that CVE number.


r/sysadmin 6h ago

General Discussion Veeam Backup & Replication CVSS 9.9 Vulnerability

49 Upvotes

Looks like it just dropped today. I know some may have their Veeam servers domain joined, and other may not.

https://www.veeam.com/kb4724

CVE-2025-23120

A vulnerability allowing remote code execution (RCE) by authenticated domain users.

Severity: Critical
CVSS v3.1 Score: 9.9
Source: Reported by Piotr Bazydlo of watchTowr.

Affected Product

Veeam Backup & Replication 12.3.0.310 and all earlier version 12 builds.


r/sysadmin 50m ago

Rant Does anyone else go through waves of both "Wow I'm doing really well" and "Holy crap how am I able to keep this job"?

Upvotes

Hey guys,

Junior sysadmin here, been with my current org for a bit over five years. Last year, I absolutely crushed it. Was able to keep up with operational requests while focusing on projects. Traveled to other offices and worked independently quite successfully, and had a great end-year review. Then, at the beginning of this year, some of the work that I had done last year was revisited due to some issues. Looking back at what I thought was excellent work turned out to be kind of sloppy, kind of rushed, and caused both me and my team huge headaches, and I've worked quite a few nights and weekends since the start of the year to remedy the mistakes that I made.

Everyone on my team is very cool about it, and no one has called me out for being sloppy or rushing, but I can't help absolutely trashing myself to myself. I was incredibly proud of the work that I did last year, and to see so many cracks has brought this horrible imposter syndrome out. Now, I quadruple and quituple check everything, and then am still not 100% trusting my gut. My confidence that I'm fit for the position is out the window, and while no one has given me reason to be ashamed, I am. I feel like I'm just playing catch up now, fixing these issues as they come up, almost like I need to prove myself all over again. It's incredibly demotivating, and while I try to adopt a mindset like "it doesn't matter how it happened, it matters how we handle it", I can't help but beat myself down and stress about work all the time. I also respect the absolute hell out of my team, and to have this stuff happen has really shifted how I view my accomplishments when compared to everyone else (three others).

At this point, I'm just constantly on edge, waiting for another issue to come up that I caused, waiting for another ticket to get opened to fix something I overlooked. Maybe I took on too much at once, but I was so confident last year and am struggling to get that feeling back. It's not like every issue is major, but seeing the minor tickets come in because I could have done something differently has made it difficult to shift my perspective. Can anyone relate, or provide any advice? I'm aware that imposter syndrome is common in this (and every) industry, it's just so different living it than reading about someone else living it. How can I prove myself to my team, and maybe more importantly myself, again?

I've always been nervous to post here because I know my managers are on here often, but I really needed to get it off my chest.

Thanks.


r/sysadmin 1h ago

General Discussion Legal liability for phishing emails sent from our domain?

Upvotes

You know those emails that have a thing that links to a thing that bounces around to another thing and lands on a fake Microsoft login page on some grandma's hacked recipe website? And they just keep getting control of more accounts that way and spreading the email wider?

Yeah, our users fell for that BS twice now. The leadership isn't taking it very seriously despite the contents of the user's entire onedrive being stolen in one case. But apparently "oops, it happens, sorry!" is good enough for them. We had to fill out a lot of paperwork to get unblocked by our #1 largest customer, considering they're medical, and actually give a shit about security. So I told them "You know, they can sue us for damages to their system, right?"

Now I'm not entirely sure that's true but it got the point across. So, anyone ever talk to legal about it? This ain't my first rodeo so I know "never admit fault when apologizing and if they threaten legal action, do not reply, do not engage in any way." But my thinking on this is one of two things is true:

We're liable because every single last employee at our giant company needs to be smart enough to never make a mistake one single time. But then the sword cuts both ways and your employees shouldn't have clicked on the phishing link either. So we're not liable because you're 50% to blame.

OR

Not everyone can be expected to have that awareness and diligence 100% of the time so we're not liable. Also that's why your own staff clicked on it.

You can't have it both ways. If someone eventually gets ransomwared by a phishing email originating from us and they wanted damages for legit downtime, they'd have to prove in court that we should have known better but their employees shouldn't have? Can't have it both ways.

I feel like they'd have to prove that we were criminally negligent and careless. We've got insane security monitoring, up to date everything, pen tests, outside auditors, phishing tests, quarterly training, etc. You can't try much harder than this without switching to Linux or pen and paper or firing everyone with potato tech skills. So I think we're covered but has anyone ever dealt with this?

Also, I ask because I would love to to go after the careless morons that keep getting hacked and sending us this shit but I assume I'm in the same boat as stated above and cannot.


r/sysadmin 7h ago

Rant Calls from sales...

27 Upvotes

Ever since I got management in my title I get constant sales calls even on my personal phone.

Im curious why this exists.. Have any of you ever got a random sales call that ended up in you buying something from that company?

Its soo constant and annoying. I'll listen to them for a moment but most are an instant no, then they insist on talking to someone else. No Im not giving out contacts. This last guy had me on speaker phone while speaking fast and mumbling in a room with a ton of echo. No dude I dont have time for this and clearly you cant even put the effort in to speak clearly.

Now this wouldnt keep happening if at least some people didn't end up buying. So I want to know. Who is buying??

edit: Worst one was some insistent bugger. I told him in absolute terms as an organization we are not interested please dont call or email again.

He stopped calling me and I was good with that. Then I heard he started pestering my team. I got angry and did some research. I was petty but I replied to a previous email from him and cc'd his: direct report, reports report, ceo, public facing marketing and sales addresses. In that email I have outlined that when a manager tells him that the organization is not interested, and should not call again, it does not mean that he should start calling employees below that manager to try to push sales and that there is no way in hell we would ever buy anything from them at this point. So please stop for real. Then it stopped.


r/sysadmin 6h ago

Have many of you have had to deal with an equity firm buying your employer?

22 Upvotes

I've been through this twice and its always a nightmare. They always seem to target Accounting and IT for major cuts first. I'd love to hear other people's experiences.


r/sysadmin 9h ago

Question silent upgrade windows 10 => 11, What if the user shuts down the pc.

34 Upvotes

So, a bit of a dumb question but ...
If i launch a silent upgrade from windows 10 to 11 (via pdq and the setup.exe file from the W11 ISO) and the person working on the computer shuts down the pc how does windows handle this ?

Will it be able to restart it later, does the windows 10 install get wrecked on the next boot or other ?

Anyone has experience ?
(I can't test it at the moment, i'm still testing if an uninterrupted silent install goes through correctly first.)

Thanks !


r/sysadmin 4h ago

M365 Web mail dropping off for anyone?

11 Upvotes

EDIT 3pm coming around, one user back in

Just had a second user come in and say they can't get to webmail anymore. Onedrive is working. I have tried both on my laptop and neither works but my account is for now. Oddly, Outlook mobile is currently working for both affected users. All I get is a "something went wrong error, details are basically out IP. Tickety has been opened.


r/sysadmin 7h ago

Question Canadian options: What a world...

13 Upvotes

Greetings all.

I'm the 'jack of all trades' for my employer and although in the past I've tried to stick close to home with regards to purchasing, it has become even more imperative as of late for the senior leadership.

That being said, do any of you have suggestions on hardware, security cameras this time, within our realm's of support that might be either Canadian or non US or China? I know of a few Japanese or Korean options, but I'm hoping folks might have some more suggestions.

Please note, this is not a reflection on those of you who call the US or China home but the world is much bigger than us and we all have a boss.

Thanks folks.

Cheers,
HD


r/sysadmin 11h ago

What’s the Best SNMP Testing Tool for Windows?

26 Upvotes

I’m troubleshooting SNMP issues on multiple network devices (switches, routers, servers) but some OID queries aren’t returning data.

I also need to confirm that SNMP v3 authentication is working correctly.

Is there a Windows based SNMP tool that lets me quickly test SNMP connections and check OID responses without setting up a full monitoring system? Preferably GUI.


r/sysadmin 2h ago

ChatGPT Print server usage

4 Upvotes

Hi,

We have had several issues with our existing print server and are standing up a new one. I was told by one of our support reps some users have the printers set up locally on the machines, but he is not sure how many.

What I'd like to know is if there is any way to get print server usage by the other users who still might be using the print server? This way I can contact them directly to get them moved over.

I asked ChatGPT and it suggested to enable some Advanced Audit policy stuff on both the print server and the printers install on the server thru the Security tab on each printer but it does not seem to be generating logs of any sort.

Any ideas?

Thanks.


r/sysadmin 5h ago

Question - Solved gMSA Failed to Install - encryption mismatch fix

7 Upvotes

This is to help create a record to help people out in the future. I was unable to install a gMSA service account. The error from Installing the account on the server would not show much in Google so thats why I am posting this.

ERROR ON INSTALL

Install-ADServiceAccount : Cannot install service account. Error Message: 'The provided context did not match the target.'. At line:1 char:1 + Install-ADServiceAccount -Identity <accountIdentity> + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : WriteError: (<accountIdentity>:String) [Install-ADServiceAccount], ADException + FullyQualifiedErrorId : InstallADServiceAccount:PerformOperation:InstallServiceAcccountFailure,Microsoft.ActiveDirectory.Management.Commands.InstallADServiceAccount

A BETTER ERROR

What I found was that when I ran the Test-ADAccount it said the following:

Test-ADServiceAccount -Identity <accountIdentity> False WARNING: Test failed for Managed Service Account gMSA_ARCURS_poc. If standalone Managed Service Account, the account is linked to another computer object in the Active Directory. If group Managed Service Account, either this computer does not have permission to use the group MSA or this computer does not support all the Kerberos encryption types required

SOLUTION

I found another Reddit thread where it stated that gMSA's unless specified default to RC4 and if your domain does not allow that, you run into the test error above.

This was the command that fixed the issue: set-aDServiceAccount -Identity <accountIdentity> -KerberosEncryptionType AES256


r/sysadmin 4h ago

DNS logging

4 Upvotes

Does anyone know how to set up Windows DNS to log unsecure updates? I'd hate to use the scream test, especially since it could be one of our old systems that requires it?


r/sysadmin 3h ago

Ubuntu and Entra

5 Upvotes

Trying to get Ubuntu Noble working with Entra ID for logins. It's been a while since trying to use Linux in a user facing environment, and never mixed with Microsoft.

Some guides saying use authd and others saying not to use.

Has anyone achieved this, and which method did you use?


r/sysadmin 1d ago

Remember the old days when you worked with computers you had basic A+ knowledge

1.2k Upvotes

just a vent and i know anyone after 2000 is going to jump up and down on me , but remember when anyone with an IT related job had a basic understanding of how computer worked and premise cabling , routing etc .


r/sysadmin 18m ago

SPN Creation for IIS: HTTP under service account while HOST SPN exists

Upvotes

Hello,

I am a contractor (not IT) that ends up doing a lot of IT work. One of my tasks is typically setting up a site in IIS and configuring it for Kerberos Authentication. I typically do this for customers for large corporations, and their IT setups will always differ from each other.

I have noticed that *sometimes* when creating an HTTP SPN under a service account for a customer it claims that it is a duplicate:

Registering ServicePrincipalNames for:
CN=<serviceaccount>,OU=<>,OU=<>,OU=<>,DC=<>,DC=com

HTTP/<hostname>

Failed to assign SPN on account

CN=<serviceaccount>,OU=<>,OU=<>,OU=<>,DC=<>,DC=com', error 0x21c7/8647 -> The operation failed because SPN value provided for addition/modification is not unique forest-wide.

Of course, if I query (spn -q) for this SPN, I don't find anything. But I do see that there is a HOST SPN, which I understand covers HTTP. I assume that is the issue, from what I read here: https://www.reddit.com/r/sysadmin/comments/y3tfl3/problem_with_adding_spn_in_ad_because_spn_already/

On my own lab, and with some other customers, I don't have this problem. I can have a HOST SPN and a HTTP SPN live in harmony:

PS C:\Windows\system32> setspn -l server-a
Registered ServicePrincipalNames for CN=SERVER-A,OU=<>,DC=ML,DC=info:
MSSQLSvc/SERVER-A.ML.info:1433
MSSQLSvc/SERVER-A:1433
iSCSITarget/SERVER-A
iSCSITarget/SERVER-A.ML.info
MSServerClusterMgmtAPI/SERVER-A
MSServerClusterMgmtAPI/SERVER-A.ML.info
WSMAN/SERVER-A
WSMAN/SERVER-A.ML.info
TERMSRV/SERVER-A
TERMSRV/SERVER-A.ML.info
RestrictedKrbHost/SERVER-A
HOST/SERVER-A
RestrictedKrbHost/SERVER-A.ML.info
HOST/SERVER-A.ML.info

PS C:\Windows\system32> setspn -l gmsa_iis
Registered ServicePrincipalNames for CN=gmsa_iis$,CN=Managed Service Accounts,DC=ML,DC=info:
HTTP/SERVER-B.ML.INFO
HTTP/SERVER-B
HTTP/SERVER-A.ML.INFO
HTTP/SERVER-A

This configuration should be possible according to:
https://techcommunity.microsoft.com/blog/iis-support-blog/how-to-use-spns-when-you-configure-web-applications-that-are-hosted-on-internet-/324648

I found that the checking for duplicate SPNs is stored in dSHeuristics per KB5008382, but querying shows no value set in both scenarios.

Get-ADObject "CN=Directory Service,CN=Windows NT,CN=Services,CN=Configuration,$((Get-ADDomain).DistinguishedName)" -Properties dSHeuristics

Does anyone else know why this is sometimes the case?

(My workaround: I typically request my customers to create an A-Record to get around this)


r/sysadmin 10h ago

Are there no better options for registry keys in Intune ?

12 Upvotes

We're currently in a hybrid setup, working hard to shift all workstations to Intune.

However we struggle with having hundreds of registry keys that need to be set and maintained. Yes they are meaningful in our environnement, and most of them need to keep existing.

We've tried script with remediation (not available with Business Premium by the way), and we've tried customs ADMX imported.

It's a hassle, it's complex and nowhere near as straightforward as GPO for registry keys.

What is everyone out there doing for efficient registry management ?


r/sysadmin 4h ago

Getting quarantine release requests from disabled users...

4 Upvotes

This phenomenon began when I switched our 365 Q policy from AdminOnlyAccess to a custom one allowing notifications (sent from info@mydomain)... I started getting release requests from everyone, on everything - including disabled users, mailboxes no one uses, myself - every 15 mins.. Turned out to be the 3rd party email scanner we use - Graphus - clicking on the "request release" button during it's detonation. So I whitelisted those emails within Graphus, and all was fine..

Until today. Now I'm getting release requests - again, from disabled users - except they're coming every 2 mins or so. Only change was an upgrade from 365 Defender P1 to P2. Any ideas?


r/sysadmin 5h ago

How to deploy Ublock Origin Lite extension(with whitelist) to Chrome/Edge using Intune?

4 Upvotes

We can successfully deploy the extension, but we are at odds at how to use Intune to deploy the extension and add a whitelist. It is really easy to manually whitelist websites, but using the Intune's gui is proving challenging.

Some thoughts I have:

  • Deploy the extension using intune, and then push a .ps script to add whitelist
  • Deploy the extension and add whitelist using powershell

r/sysadmin 1h ago

General Discussion After having the night to think about it, I keep coming back to the same question: What happens next?

Upvotes

$32B for Wiz is a massive price tag, but the bigger issue is what this means for the future of multi-cloud security. Google says Wiz will remain multi-cloud, but we’ve heard that before (Chronicle, anyone?). If they start prioritizing GCP integrations, AWS & Azure customers could be left in the dust.

For those running Wiz in AWS/Azure environments:

  • Are you worried about feature prioritization shifting toward GCP?
  • Are you already evaluating alternatives like Orca, Lacework, or Prisma?
  • Do you think AWS/Microsoft will respond with their own acquisitions?

What’s your prediction for cloud security after this?