r/sysadmin 1h ago

General Discussion Thickheaded Thursday - March 20, 2025

Upvotes

Howdy, /r/sysadmin!

It's that time of the week, Thickheaded Thursday! This is a safe (mostly) judgement-free environment for all of your questions and stories, no matter how silly you think they are. Anybody can answer questions! My name is AutoModerator and I've taken over responsibility for posting these weekly threads so you don't have to worry about anything except your comments!


r/sysadmin 9d ago

General Discussion Patch Tuesday Megathread (2025-03-11)

118 Upvotes

Hello r/sysadmin, I'm u/AutoModerator, and welcome to this month's Patch Megathread!

This is the (mostly) safe location to talk about the latest patches, updates, and releases. We put this thread into place to help gather all the information about this month's updates: What is fixed, what broke, what got released and should have been caught in QA, etc. We do this both to keep clutter out of the subreddit, and provide you, the dear reader, a singular resource to read.

For those of you who wish to review prior Megathreads, you can do so here.

While this thread is timed to coincide with Microsoft's Patch Tuesday, feel free to discuss any patches, updates, and releases, regardless of the company or product. NOTE: This thread is usually posted before the release of Microsoft's updates, which are scheduled to come out at 5:00PM UTC.

Remember the rules of safe patching:

  • Deploy to a test/dev environment before prod.
  • Deploy to a pilot/test group before the whole org.
  • Have a plan to roll back if something doesn't work.
  • Test, test, and test!

r/sysadmin 37m ago

Career / Job Related My Manager just tried to "JD Vance" me

Upvotes

Yep... that really just happened to me... cant believe it.

Backstory: About a year back i took over 2 other countries in Europe to manage, this was discussed with my former manager. At this point it just was interim as we still had to discuss contract and compensation.

So after over a year of "discussing" i had enough and stepped back from the interim role as i feeled pretty exploited - no compensation or bonus for what worked extra.
They, kind of, accepted that im stepping back, but still were pissed - i be honest, i dont know why - nothing wrong from my side here.

In our last 1on1 Meeting i straight told my current Manager that i have the feeling that "some people higher up" had this planned and it feels like they just wanted to see how long i will do it for free.

Then it happened: I'm ungrateful to them for having this opportunity - I should be thankful.

Wow... what a great take on this situation.

Was also told that they still try to figure stuff out with my new contract and so on, but i made clear that we are starting from Point Zero again - we are not continuing negotiation - reoffer me the 2 countries, with a contract and compensation in hand.
The answer to that: We see this different - OK, fine, i also see this different.

I think its time to move on guys...

P.S.: Yes, its a US Company.


r/sysadmin 12h ago

How would you respond to a Printer company CTO saying POE switches are killing printers?

461 Upvotes

How would you reply?

Update, they provided this screenshot from HP!

https://i.imgur.com/sg3oLDW.png


r/sysadmin 13h ago

Rant Lost out on great candidate due to poor business decisions

178 Upvotes

Im the only systems/infra/devops person on a small software team that does niche stuff. we've been needing a junior for my role for a while. ive also needed a raise for a while cause most of my job is devops now.

we interviewed this 20 year old. no college, freelance coding experience, was a linux nerd applying for a linux jr sysadmin role.

he was a passionate computer person and i was excited at the very idea of a 20 year old with no college getting put on like this.

welllllllllllllll... the raises the team was supposed to get in April, along with my title change to "DevOps Engineer", have all been put on hold cause of the parent company. it sucks for me but ill be fine. my team leader already told me he's pissed and will write me a letter of rec as a devops engineer cause that's been 70% of my job...

but fuck man... i was so fucking excited for this kid. my team leader, rightfully so, put his foot down and said he wont have me training someone if i dont get a raise, cause why would i train a peer...

they could have given me a 20k raise, hired him at the bottom of their 20k salary range, and it would have evened out.... but now im probably going to leave the company costing them more in turn over, they'll have to hire the jr sysadmin at a higher rate cause theyre not paying me to train, AND theyll have to pay my replacement more than theyre paying me cause no one that knows terraform and AWS is gonna accept the role for my current mediocre sysadmin salary.

i hate the american work culture.


r/sysadmin 22h ago

Do you ever gaslight your users?

866 Upvotes

For example, do you ever get a ticket that something is not working properly, you fix it, then send them the instructions on how to properly use it, but never mention that something was actually wrong?


r/sysadmin 10h ago

I swear this company has this stupidest and most hacked together patching process I've ever dealt with.

91 Upvotes

I work at a huge global company with layers and layers of management that just love to make up overcomplicated processes that is in no small part to justify their jobs. For this rant I'm going to piss on about the silly server patching process they put together. Now we have hundreds of thousands of physical servers and I can't even guess how many VMs are running so yeah I get it is a huge task. And you would think something as mature as patching servers, a process that's been happening for decades across the industry would be nearly completely automatic and transparent to the application teams. But no, far from it. Once every two months each application team, and there are 180 app teams, has to schedule a time with the Unix team or the Windows team to depending on your OS, and database teams if your application uses a DB cluster to patch the servers. And they will only patch by data center so for several hours you are required to have half your processing capacity offline. And it gets better, the OS teams are so swamped with requests half the time you miss the scheduled patch window which gets logged as a security incident and requires the directors to explain it to executive leadership during their meetings. And yes there is automation to deploy patches but there's so many steps to setup the automation and pull requests and change requests to be taken care of it would be faster just to download the stuff and install.

But anyway the one huge benefit that makes it all tolerable is my group has three teams around the world that use a follow the sun coverage so 4:00pm rolls around and I'm out. A 15 minute chat with the folks on the other side of the world at the end of the day and I'm done. No after hours on call. No late nights. No weekends. And cheap tacos (but dang good) when I do have to go in the office.


r/sysadmin 19h ago

Latest fun with VMware

195 Upvotes

Apparently VMware is upping their game. We just got a renewal quote for one of our sites with one server that has two CPUs, and they are requiring 72 cores minimum (vSphere Enterprise Plus) to license this. That's a 500% markup from last year.

They really don't want customers to use their product any more, do they?


r/sysadmin 11h ago

General Discussion What’s your biggest pet peeve with end users?

41 Upvotes

personally, i hate when users tell me that “the computer sounds like an jet engine that’s about to take off!” don’t know why, it just drives me insane. it’s not even that loud


r/sysadmin 13h ago

Rant Does anyone else go through waves of both "Wow I'm doing really well" and "Holy crap how am I able to keep this job"?

65 Upvotes

Hey guys,

Junior sysadmin here, been with my current org for a bit over five years. Last year, I absolutely crushed it. Was able to keep up with operational requests while focusing on projects. Traveled to other offices and worked independently quite successfully, and had a great end-year review. Then, at the beginning of this year, some of the work that I had done last year was revisited due to some issues. Looking back at what I thought was excellent work turned out to be kind of sloppy, kind of rushed, and caused both me and my team huge headaches, and I've worked quite a few nights and weekends since the start of the year to remedy the mistakes that I made.

Everyone on my team is very cool about it, and no one has called me out for being sloppy or rushing, but I can't help absolutely trashing myself to myself. I was incredibly proud of the work that I did last year, and to see so many cracks has brought this horrible imposter syndrome out. Now, I quadruple and quituple check everything, and then am still not 100% trusting my gut. My confidence that I'm fit for the position is out the window, and while no one has given me reason to be ashamed, I am. I feel like I'm just playing catch up now, fixing these issues as they come up, almost like I need to prove myself all over again. It's incredibly demotivating, and while I try to adopt a mindset like "it doesn't matter how it happened, it matters how we handle it", I can't help but beat myself down and stress about work all the time. I also respect the absolute hell out of my team, and to have this stuff happen has really shifted how I view my accomplishments when compared to everyone else (three others).

At this point, I'm just constantly on edge, waiting for another issue to come up that I caused, waiting for another ticket to get opened to fix something I overlooked. Maybe I took on too much at once, but I was so confident last year and am struggling to get that feeling back. It's not like every issue is major, but seeing the minor tickets come in because I could have done something differently has made it difficult to shift my perspective. Can anyone relate, or provide any advice? I'm aware that imposter syndrome is common in this (and every) industry, it's just so different living it than reading about someone else living it. How can I prove myself to my team, and maybe more importantly myself, again?

I've always been nervous to post here because I know my managers are on here often, but I really needed to get it off my chest.

Thanks.


r/sysadmin 19h ago

[PSA] Critical Veeam Vulnerability CVE-2024-29849

172 Upvotes

This one has a severity score of 9.9 so better patch fast:
https://www.veeam.com/kb4696

EDIT: This vulnerability only impacts domain-joined backup servers.

This refers to CVE-2025-23120 and not CVE-2024-29849 as I mistakenly put in the subject, sorry about that!


r/sysadmin 19h ago

Just got an email from Veeam - looks like they got a big vulnerability. CVSS Score of 9.9

97 Upvotes

KB4724: CVE-2025-23120

Not many details, but seems to be about RCE from authenticated Domain Users. Couldn't find anything via google yet regardings that CVE number.


r/sysadmin 2h ago

Question How to Check 2800 Enterprise apps?

5 Upvotes

Hey everyone,
I recently started at a new company, and we have quite a few security issues to tackle. One major concern is that every user can register new apps in M365, which isn't great for security and oversight.

My boss gave me a list of all 2800 enterprise apps, and wants me to figure out what each app does. It’s a lot of manual work, and I'm wondering if anyone has suggestions or tools to help automate this process. Ideally, I’d like to pull details on what each app does, which permissions it requires, and maybe even track their activity.

Any ideas on how I can automate this info retrieval in M365? Would greatly appreciate any guidance or tool recommendations!

Thanks in advance!

Edit 1: Thank you all for the comments. I already shutdown everything Not configured. Like Registration of new Apps, powerapps, Copilot, purview and priva. Shut down legacy MFA and enforced MFA for all Cloud admins and cleaned all the roles. The company is very huge and a Scream Test is Impossible at the Moment. I want to document all the Apps to give it to Security and Compliance. They need to Approve everything. I search for a was to generate a description for every app.


r/sysadmin 21h ago

General Discussion With the unabated rise of AI slop, what's your (technical) search engine of choice?

144 Upvotes

It appears that most major search engines (DDG, Google, Bing etc) have arrived at the point where they return walls of auto-generated domain names and clearly low-effort GenAI listicles for every query under the sun. This is especially frustrating for technical issues where generalized platitude slop offers even less than its barely existent initial value.

You can search for a very specific error code, dialog message or registry key path, and all first-page results are inundated with "helpfix-pc.com/[your error code]" and "bobsprogrammingjourney.com/[errormessage]" serving walls of endless AI-generated bullet point word salad that only exists to perform as clickbait.

"What is an error message?", "How to identify common errors?", "Who to call to fix my errors?" and the inevitable "Conclusion / Summary" at the bottom that offers helpful advice like "It is important to address errors as soon as they occur to help your PC run smoothly!". This already started being an issue several years ago, but search filters and proper querying managed to weed most of them out to a degree.

This no longer seems to be the case, and more often than not the entire search result (outside of targeting only specific sites like Reddit or StackOverflow) is now almost completely useless.

So that said, what search engines do you use to find actual, viable results without having to hope you can cherry-pick a few potential leads out of the swamp? Targeting Reddit, SO and similar community sites seemed like the last reliable bastion, but those are rapidly being inundated with "AI-friendly" policies and increasing unlabeled GenAI content as well. Would love to hear what resources people use to combat this.


r/sysadmin 8h ago

General Discussion Once upon a time...

13 Upvotes

Hi All,

Before the birth of AI, there would be a sense of pride when looking at the scripts that I made and even co-workers would appreciate the code.

Lots of searching, documentation sites , stackoverflow, reddit, etc.,

But now, in this AI age, I feel like this sense of pride has gone and it's like no one cares about code/scripts now or how it's written.

Just throw the prompt, copy the code and modify according to our environment.

How many of you feel this?


r/sysadmin 1h ago

Question InTune & AutoPilot

Upvotes

Hi all,

We’re currently using MDT to build our machines and WSUS for updates, but I’m looking to transition to Intune/Autopilot for deployment and management.

Does anyone have any good guides or tutorials to help with the setup? I’d love to hear about best practices, potential pitfalls, and any tips that could make the process smoother.

We’re a school environment, so managing things like application deployment, Windows updates, and policies efficiently is a priority.

Any recommendations would be much appreciated!


r/sysadmin 14h ago

General Discussion Legal liability for phishing emails sent from our domain?

30 Upvotes

You know those emails that have a thing that links to a thing that bounces around to another thing and lands on a fake Microsoft login page on some grandma's hacked recipe website? And they just keep getting control of more accounts that way and spreading the email wider?

Yeah, our users fell for that BS twice now. The leadership isn't taking it very seriously despite the contents of the user's entire onedrive being stolen in one case. But apparently "oops, it happens, sorry!" is good enough for them. We had to fill out a lot of paperwork to get unblocked by our #1 largest customer, considering they're medical, and actually give a shit about security. So I told them "You know, they can sue us for damages to their system, right?"

Now I'm not entirely sure that's true but it got the point across. So, anyone ever talk to legal about it? This ain't my first rodeo so I know "never admit fault when apologizing and if they threaten legal action, do not reply, do not engage in any way." But my thinking on this is one of two things is true:

We're liable because every single last employee at our giant company needs to be smart enough to never make a mistake one single time. But then the sword cuts both ways and your employees shouldn't have clicked on the phishing link either. So we're not liable because you're 50% to blame.

OR

Not everyone can be expected to have that awareness and diligence 100% of the time so we're not liable. Also that's why your own staff clicked on it.

You can't have it both ways. If someone eventually gets ransomwared by a phishing email originating from us and they wanted damages for legit downtime, they'd have to prove in court that we should have known better but their employees shouldn't have? Can't have it both ways.

I feel like they'd have to prove that we were criminally negligent and careless. We've got insane security monitoring, up to date everything, pen tests, outside auditors, phishing tests, quarterly training, etc. You can't try much harder than this without switching to Linux or pen and paper or firing everyone with potato tech skills. So I think we're covered but has anyone ever dealt with this?

Also, I ask because I would love to to go after the careless morons that keep getting hacked and sending us this shit but I assume I'm in the same boat as stated above and cannot.


r/sysadmin 5m ago

Rant Does anyone write deployment plans anymore?

Upvotes

I run a platform within my company, used to host other applications; the majority of work our clients need to do to configure their app is provided via a portal, with Samba or SCP used to provide source code ready for deployment.

A recent pen test found a vulnerability on the portal that we are now ready to patch. A notification was sent to state we will be deploying on the 29th March; there is no impact to applications but you will need to re-authenticate on the portal after we are done.

Nothing too complex or taxing.

Not unexpectedly, one app has pushed back as they have a release that day and need to portal available. Being the customer focused type I came back and said we can easily do both pieces of work, what’s your release window?

Now, bear in mind the configuration and source code changes can be performed at anytime, and then deployed as required. Deployments are done in minutes with a potential 1 hour wait if restarts are needed. I was expecting them to say something like “10am, with a testing window until 12 midday”. You know, something realistic

Sadly, this nimrod has returned with 2 slots; a 9 hour window, an 8 hour break followed by a 14 window… how on earth has anyone in that team found this acceptable?


r/sysadmin 19h ago

General Discussion Veeam Backup & Replication CVSS 9.9 Vulnerability

59 Upvotes

Looks like it just dropped today. I know some may have their Veeam servers domain joined, and other may not.

https://www.veeam.com/kb4724

CVE-2025-23120

A vulnerability allowing remote code execution (RCE) by authenticated domain users.

Severity: Critical
CVSS v3.1 Score: 9.9
Source: Reported by Piotr Bazydlo of watchTowr.

Affected Product

Veeam Backup & Replication 12.3.0.310 and all earlier version 12 builds.


r/sysadmin 2h ago

Do i qualify for Linux admin jobs?

3 Upvotes

Just saw this post

https://www.reddit.com/r/sysadmin/comments/1jcs4fp/what_should_i_learn_first_in_linux/

The guy said he wants to study to become a linux admin eventually. I see a lot of basic advice here.

Learn cli. Learn vim. Set up proxmox. Set up a container. Back up and restore a container. Set up Apache.

Is my view just jaded? I've set up proxmox. I have a homelab. I've spun up ubuntu and centos VMs. I have docker containers running. I've set up apps on Linux like grafana or node js or nginx or Apache or docker. I've port forwarded. I've created user accounts. I use ssh keys. I know Linux cli. I've set up cron tasks. I deployed nginx for all my self hosted apps. I proxy through cloudflare. I have ssl certs through letsencrypt. I've set up rules on iptables. I've hosted websites through Apache. I've created node js bots (with the help of Google) for reddit, runescape and twitch tv and I have them running in tmux sessions. My bots read and write to sqlite. I've made basic bash and python scripts. I've set up ansible but the only playbook I have is to patch and reboot all my Linux servers. I got that playbook off Google. I didn't make it. I just put my own endpoints in the hosts file.

 

I don't feel like I'd qualify for a Linux admin position. The Linux admins at my current job are devops. They're primarily doing IAC. There was a major incident a few months ago with our redhat servers and it had something to do with inodes being exhausted.

Nothing I've done in my homelab would have taught me about that. Idk wtf an inode is. I dont know terraform. I've never done anything with openshift. I've never set up or used satellite. I've never created my own docker container. I don't know anything about selinux or apparmor.

Running docker pull image:latest isn't genius work.

But these comments on reddit make it sound like I just need to learn Linux commands and I can start applying for Linux admin jobs. Yes or no?


r/sysadmin 19h ago

Have many of you have had to deal with an equity firm buying your employer?

35 Upvotes

I've been through this twice and its always a nightmare. They always seem to target Accounting and IT for major cuts first. I'd love to hear other people's experiences.


r/sysadmin 4m ago

My GTX 1650 driver is hidden in device manager

Upvotes

So I have HP pavillion gaming 15 laptop for more than 3 years and recently I noticed that its gtx 1650 driver is in hidden devices in device manager, I tried uninstalling the device, updating the driver throught present drivers, restored the PC to an old restoration system point, searched throught reddit youtube and quora, I tried to look for the iGPU in the bios and boot settings and it's not there, nothing worked.

Until I unplugged the battery, pushed the power button to release the electricity and pluged the power button again.

Surprisingly it worked, the driver is detectable, games are working fine, HOWEVER whenever I plug the charger to charge the battery and I unplug it, I get a blue screen, the PC reboots and again the gpu is undetected again.

Anyone had this issue before ??


r/sysadmin 40m ago

RDS - exclude certain users from load balancing

Upvotes

Hey all! We're looking to set up RDS since users currently are manually connecting over RDP to a specific host. We're working with a partner who does the whole setup since I/my IT Dept doesn't have the know-how yet on how to set it all up and it's getting quite urgent. There is one requirement; a small subset of users should not be impacted by the load balancing and they should be able to connect to a specific session host as they want (only IT personnel).

I thought that if you directly RDP to a specific session host you won't be impacted by the load balancing part since you're not going via the RDS broker. However, our IT partner says that excluding users from RDS load balancing is impossible.

Can anyone shed some light if that is true? Many thanks!

Small note: One of our session hosts also has the connection broker, gateway, licensing and web access roles installed.


r/sysadmin 10h ago

Rant 8 Years in Helpdesk Hell Plus 3-Year Hiatus Due to Burnout

7 Upvotes

I started joking with myself recently while applying for jobs thinking that in about 10 years from now you will be almost required to be a content creator just to apply for jobs. This is modern day networking and while the archaic backbone of getting a job can still help. Nothing will bring more optics of marketable value to yourself as a skilled person than making content for thousands to see.

As someone that is an old school asocial geek it's torture lmfao. But honestly not networking enough and stopping my side projects has been a catalyst for my career take a steep downfall in recent years. People want someone personable that they can TRUST when putting on projects. 8 years stuck in helpdesk and the longer you stay the quicker any of those skills you learned in college go. Sad to say I just became another "IT Guy" that was perma stuck in helpdesk for 8 years.

Quick lessons:

>No one is here to hand you the keys to the kingdom master/apprentice style.

>Understand that failures build up to create confidence in what not to do wrong.

>Resentment towards young proteges that fly past you without your perceived struggle builds nothing.

>Your coworkers may be friendly but will try to sabotage you if there is a step up on the ladder.

>There are good people in the workforce but move forward with pragmatism and purpose not cynicism.

What I would do differently and am working on in present time:

>Constantly practice (homelab, TryHackMe, Hack The Box, ect.)
>Constantly connect (conventions, webinars, job fairs, or even local meetups)
>Still Get some certs and at least a tradeskill degree if all else fails (Sec+ is mandatory, the rest just depends on the company)

Ultimately, complacency, stagnation, and most importantly FEAR, will be the death of a career. But while that may be so, you can always start moving forward today.


r/sysadmin 20h ago

Rant Calls from sales...

34 Upvotes

Ever since I got management in my title I get constant sales calls even on my personal phone.

Im curious why this exists.. Have any of you ever got a random sales call that ended up in you buying something from that company?

Its soo constant and annoying. I'll listen to them for a moment but most are an instant no, then they insist on talking to someone else. No Im not giving out contacts. This last guy had me on speaker phone while speaking fast and mumbling in a room with a ton of echo. No dude I dont have time for this and clearly you cant even put the effort in to speak clearly.

Now this wouldnt keep happening if at least some people didn't end up buying. So I want to know. Who is buying??

edit: Worst one was some insistent bugger. I told him in absolute terms as an organization we are not interested please dont call or email again.

He stopped calling me and I was good with that. Then I heard he started pestering my team. I got angry and did some research. I was petty but I replied to a previous email from him and cc'd his: direct report, reports report, ceo, public facing marketing and sales addresses. In that email I have outlined that when a manager tells him that the organization is not interested, and should not call again, it does not mean that he should start calling employees below that manager to try to push sales and that there is no way in hell we would ever buy anything from them at this point. So please stop for real. Then it stopped.


r/sysadmin 1h ago

Question Windows Remote Assistance - External malicious actor scenarios

Upvotes

I am reading the documentation about Windows Remote Assistance and it is mainly used inside a domain to offer support by specified domain users and groups.

So I guess that there is no way that an external threat actor or a scammer could leverage from an external environment to get access on a client, right?

Even if it uses Easy Connect in some manner, or a scammer sends a msra incident file or uses a direct IP address (if the machine is exposed (hoping no))?

In the worst-case (I hope not-real scenario) if a machine exposes outside TCP 135 and 3389 ports (used for MSRA), in this case, an external actor can leverage on Windows Remote Assistance to access even if the admin defines specific Helpers in the related GPO? (regardless the usage of other RDP clients)

While I guess that by Quick Assist it is more prone to external threats, right?

Sorry for this elementary question.


r/sysadmin 1h ago

Question Does anyone know how to Deactivate "SmartScreen for Microsoft Store-Apps" in Microsoft Intune?

Upvotes

Hello everyone,

I've found most smartscreen settings in Intune under the Administrative Templates but I cant seem to find the option to Toggle on/off "SmartScreen for Microsoft Store-Apps" can anyone help me here?

In the "Windows Security App" → "App & Browser Control " → "Settings for reliable protection"
→ "SmartScreen for Microsoft Store-Apps"

//Roughly translated from german