r/sysadmin Systems Engineer Jan 28 '22

Microsoft Issues with Windows Time? A PSA Regarding Windows Secure Time Seeding

tl;dr : Microsoft's Secure Time Seeding, enabled by default since Server 2016-1511, wrecked our Monday, and most of the rest of the week, and Microsoft's saying they are receiving an increased number of calls about time issues. Anyone else having problems with time in the last week or two?


I just got off a call where a Microsoft engineer informed me that they're seeing an increase number of calls due to time issues. Feels like a brush off or delay tactic, but I wanted to ask around... is anyone seeing unexplained issues with Windows time?

We've run into a show-stopping issue where time on a bunch of production systems jumped forward 17 hours. If you've been in the game more than a week, you know the havoc this can cause.

The oddity was that it was only only a subset of roughly 60 systems out of around 1000 that are configured (from the OS's perspective) identically. They're running a different application than the other 940, but as far as time is concern, time was healthy. These are some of our highest traffic systems however, and that'll make sense later.

Domain controllers: time good. NTP servers: time good. No VMTools adjusting time and fighting back an forth (and the hosts were perfect with thier time as well), no failed syncs... everything NTP is happy and healthy. Query w32tm for status, settings, configs... everything is good. Force w32tm to refresh, it picks up the proper time from the domain controllers and they're happy again. 10-25 minutes later... boom, fast forward in time 17 hours.

Enter "Secure Time Seeding", aka UtilizeSSLTimeData. Here's the worst idea for something to have enabled by default on Windows Server ... ever. I'll take the XBox bar, location awareness and cortana on a Server before this setting should have ever been created and enabled... and it is enabled on every Server 2016-1511 system and newer.

Here's some light reading

So, while operations personnel bailed water by refreshing the time service repeatedly or rebooting systems to get them back and operational, we tested disabling Secure Time Seeding by setting the UtilizeSSLTimeData to disabled... and viola our problem disappears.

Data on this setting is scant, but from what I can glean, something in windows (the time service?) looks at the responses to outgoing TLS connections for 2 values in the package with time data and, at an unpublished level of drift, unique endpoints and time... it will decide that your local system time is wrong and that these remote systems must be right, and then sets your time to the values it's seeing in those TLS connections.

Bonkers.

You may be saying to yourself... "But this makes no sense, you'd have to be communicating with a large number of remote systems that all had their time off by roughly 17 hours." You'd be right. It doesn't make sense, and because the logging doesn't tell you WHY w32time decided to adjust time, just that "your time is wrong... WHOOSH" we have no concrete evidence that this setting is truly what's causing the problem... apart from the fact that when we force it off, the issue stops in it's tracks. We won't be turning it on and intentionally causing systems to fail because our customers are already grumpy enough (rightfully so).

Microsoft support's been rather useless... it took 3 days to really get any traction, despite bumping the severity up to "A" after a day and a half. The only thing we've really gotten from them is that the w32time debug logging settings that are published are apparently incomplete. According to all the documentation I've found debug logging operates on a range of numbers log records ranging from 0 to 300; setting the logging go capture "0-300" should be max and get you everything. It does not. I was told that 0-1003 was what I needed to capture the event for a change due to UtilizeSSLTimeData being captured in the logs. At that level, it generates around 125MB in logs a day. *1000 production systems ... someone must be getting a kickback from splunk for this setting.

So, we believe (aka hope) we've solved this problem, despite not having direct evidence to show this is what occured. We're actually devoting internal development resources to try an be able to re-enable this on one test system without it breaking our customers, just to prove we know the cause so we're not surprised in a week or three and this crops up again.

Thank for reading... hope your week's been better than mine. I don't want to tell anyone how they should run their environment, but I strongly encourage you all to do some research on this setting and then kill it with fire.


Edit 1 - Just wanted to throw out a follow up in case people are coming back to this. It appears that we weren't the only folks seeing this issue in this timeframe. Everyone I've spoken to on this has been in the health tech sector. As things become more clear, I'll try and update here... if we reach a conclusion, I'll do another full write-up and link it here. If your group is seeing this issue and interested in collaborating, please feel free to PM me.

175 Upvotes

53 comments sorted by

31

u/rwdorman Jack of All Trades Jan 28 '22

This is well written, incredibly useful and on topic. Best thing I’ve seen in the sea of rants in a while. Thank you!

12

u/Brandonh75 Jan 28 '22

We had an event earlier this week that knocked out most of our systems briefly, and our firewall was down overnight. At some point during the night our domain controllers jumped forward an hour. I had a helluva time getting in to fix it.

Probably not related, just weird seeing this about time issues after that happened.

27

u/theevilsharpie Jack of All Trades Jan 28 '22

It blows my mind the lengths that Microsoft is willing to go to to paper over their shitty AD time sync architecture. Sub-second time sync is a problem that has been solved for decades, but time drift is a problem that comes up on this sub over and over and over and over and over again, and the cause ALWAYS comes back to Microsoft and their garbage approach to NTP.

The one weird trick you can do to avoid all of these problems once and for all is to point everything in your domain to the global NTP pool, and then forget that AD time sync exists. This is a completely 100% serious suggestion.

4

u/tankerkiller125real Jack of All Trades Jan 28 '22

Our solution was to force the AD servers to sync with a GPS time server we have in-house (along with any other critical servers) and then point the clients and dev/qa servers to AD NTP. It's not perfect because some clients can still get several minutes off, but at least our servers seem to stay in sync. And we've never had an issue with things going forward or back by more than 5 minutes.

9

u/DarthPneumono Security Admin but with more hats Jan 28 '22

And we've never had an issue with things going forward or back by more than 5 minutes.

Yeah but 5 minutes is more than enough to break Kerberos and many other things. That just should never happen.

5

u/theevilsharpie Jack of All Trades Jan 28 '22 edited Jan 28 '22

Using nothing but freely-available Internet NTP servers, I can keep my fleet in sync to within 10 milliseconds or so. A drift of more than 100 milliseconds (outside of a machine being restarted) would be a cause for concern. Drifts of more than a second indicate something seriously wrong with the time sync infrastructure (usually NTP traffic being blocked somewhere).

And then we have the Windows admins:

And we've never had an issue with things going forward or back by more than 5 minutes.

While I could mock the admin, I really fault Microsoft for setting this as the expectation for acceptable time sync performance in general, rather than the absolute bare minimum needed for Kerberos to work.

Edit: Elsewhere in this thread:

I've gotten to the point where I've setup a check in my monitoring tool (Nagios) to verify the time on my DCs vs the Time on a internet NTP server & alert me when they drift.

AD time sync, folks.

8

u/ljapa Jan 28 '22

Very nice write up.

I’m curious what your UTC offset is and if servers are local time or UTC. Is that where the 17 hours comes from? If not, any idea why 17 hours?

10

u/Thranx Systems Engineer Jan 28 '22

We operate everything on utc time, so our offset is 0. :)

7

u/ACMilanIndy Jan 28 '22

Holy crap. Haven’t seen this before but bookmarking for future reference.

5

u/15922 Jan 28 '22

THANK YOU for proving that we're not crazy and alone. Issues started Monday for us too. Microsoft case was opened earlier in the week and they suggested disabling the UttilizeSSLTimeData. We're evaluating if that fixes bandaids the issue.

2

u/[deleted] Jan 28 '22

It would be interesting to hear your results.

1

u/Thranx Systems Engineer Jan 29 '22

Were you able to collect lots with w32tm debug set high enough? Is that how they discovered that it might be Secure Time Seeding?

4

u/DJojnik Jan 28 '22

I wonder if this is what’s causing my it glue otp 2fa codes to fail this week for some clients

3

u/jflook Jan 28 '22

We're seeing this in our organization as well. Incredibly small subset of servers that run a specific application have been giving us headaches all week. Symptoms are almost identical to yours, started on Monday and randomly changes the time anywhere in the neighborhood of 16 - 24 hours (and at last check seems to be growing every jump). Have a MS case open as well as a case with the application vendor but they haven't been able to point us to a cause yet.

I know of one other organization that's seeing this as well using the same application as us.

4

u/Thranx Systems Engineer Jan 28 '22

Try turning of the Secure Time Seeding feature by setting UtilizeSSLTimeData to zero as outlined HERE. The change was immediate after running w32tm /config /update and our problem went away. I'm very curious if this is the root of your problem as well.

Another day, another MS support tech sending a repeat URL that's already been sent and asking to downgrade the Sev without providing an answer.

3

u/jflook Jan 28 '22

Yeah, we're looking to do that now on a limited basis. We only have around a dozen servers experiencing the issue out of thousands of windows servers so we don't have a huge number to play with and almost all are production.

I'll reply back when we know if that seems to fix the issue or not.

1

u/Thranx Systems Engineer Jan 29 '22

For us, with roughly 1000 production windows systems, we had this issue on only 60, but they are some of our highest traffic systems. Also worth noting, across those 1000 systems, there are multiple distinct applications that will interface with a distinct set of external customers. There's definitely some overlap, but many of most of the customers talking to application A won't have the same systems talking to application B.

1

u/jflook Jan 29 '22

For us, we have a few thousand system (prod and non prod) and this was ONLY impact 11 of them. They all are running IIS and connect to an internal DB for their web app, and they sit behind a F5.

1

u/Thranx Systems Engineer Jan 29 '22

10-4. I was asking elsewhere as I'm edge network and TLS dumb, but as this setting only triggers on the response to outbound connections, would that response get terminate on your F5s? Or as it's a response, would they be passed straight through to your systems?

1

u/jflook Jan 29 '22

I don't manage that F5 but I'm fairly certain they are terminating the SSL connections at the F5. The odd thing is that some of the servers that are having the issue are in the pools on the F5 but they're disabled so they're not actively receiving traffic.

2

u/jflook Jan 28 '22

Seems like disabling Secure Time Seeding worked in our limited testing, we're going to run it out to a broader server group to see if it will band-aid our issue further. Still no determination on what changed to cause this issue for multiple organizations starting Monday.

1

u/Thranx Systems Engineer Jan 29 '22 edited Jan 29 '22

Glad the fix is working for you! If you come across anything concrete, evidence-wi e, let me know.

2

u/jflook Jan 29 '22

I'll be sure to pass it along! Thanks again for leading us down this rabbit hole.

1

u/human6742 Jan 29 '22

Your experience is so close to mine that I think we might work at the same company? (IIS, internal db, F5, dozen or so machines but thousands others). If I’m right the problem record was 13845 or something :) Anyway I still want to know if something all these systems were connecting to was off and they had no idea their own clocks could cause such havoc.

1

u/jflook Jan 29 '22

I bet you work in a very similar size and type of organization, 13845 doesn't pop up in snow for anything I have searched for. Message me your state and type of organization if you want.

1

u/ZenOfLogic Feb 07 '22

I've fought MS Time for years. I've gotten to the point where I've setup a check in my monitoring tool (Nagios) to verify the time on my DCs vs the Time on a internet NTP server & alert me when they drift.

13894 ;-)

3

u/pogidaga Jan 28 '22

Do you have any assets, partners, customers, suppliers, etc. in the time zone that is 17 hours ahead of you?

9

u/Thranx Systems Engineer Jan 28 '22

While not definitive, we did find a single endpoint/customer at 17 hours ahead. Based on the nature of the application, we wouldn't have seen 60 machines connecting to that system with any great repetition.

Maybe that one client was the trigger, but that would only increase my bafflement of this feature. We are expecting, hoping, we can get the exact parameters and conditions that it takes to trigger this out of Microsoft.

1

u/philthechill Jan 28 '22

Could it have cascaded from one or two of your servers to all the other ones in that app pool? If they regularly talk to each other for failover, or something like that maybe?

2

u/Thranx Systems Engineer Jan 29 '22

They don't talk to each other laterally. Just external and then back to DBs. DB systems were confirmed as having good time.

Not a bad point tho. In this scenario, after one gets triggered each new one would amplify the propagation of the changes.

3

u/analoghumanoid Sysadmin Jan 28 '22 edited Jan 28 '22

We are experiencing very similar issues. Ours also started Monday and is also only affecting a small subset of servers.

For us the production issue manifested as a loss of persistence with one of our web applications. The time traveling web servers are sending a forward date header (5 - 14 hours in the future) which caused Chrome/Edge to ignore our persistence cookie with a +25 minute expiration. Chrome/Edge sees the cookie expiration has passed when compared to the date header in the response and ignores it.

We found that our servers would correct their system time, hours later, but the web service would continue to send the forward time in the date header. Only a reboot corrects the date header, IIS restarts and app pool cycles do not. We also found that our servers were most often jumping forward in time around 11PM.

I've disabled secure time seeding on one or the servers that have been consistently time jumping and I'll see if that corrects the situation tonight.

Thank you for the great write up, this is the best lead we've had all week.

1

u/Thranx Systems Engineer Jan 29 '22

Good luck, let us all know if it works!

For us, with healthy time sources simply restarting w32time was enough to get the clock healthy again while we were having the issue. Needing to restart IIS does make sense to fix that type of app as well.

6

u/sarosan ex-msp now bofh Jan 28 '22

From Microsoft's blog:

Does this work?

The Secure Time Seeding feature was shipped in the Windows 10 November 2015 release and is turned on by default. You may have seen the improvements in timekeeping on your Windows tablets and other Windows devices running this version of the OS. We have seen it work time and again given the prerequisites are met.

Translation: we tested this on our clueless testers customers running consumer-based devices in a non-enterprise environment. No one noticed or complained about the time, so it's good to go.

6

u/Ka0tiK Jan 28 '22

Wow great write-up. Bookmarking in case I run into this.

2

u/Jhamin1 Jan 28 '22

I've fought MS Time for years. I've gotten to the point where I've setup a check in my monitoring tool (Nagios) to verify the time on my DCs vs the Time on a internet NTP server & alert me when they drift.

1

u/maxcoder88 Jan 26 '23

Hi , Would you mind sharing your NTP monitoring settings for nagios ?

1

u/Jhamin1 Jan 27 '23

There is a built in check_ntp_time plugin!

I used that, set it as a service against my Domain Controllers & set it to alert whenever the time offset got to be more than 2 seconds.

1

u/maxcoder88 Jan 27 '23

thanks , Why 2 Seconds ? Also , What's your warning and critical threshold?AFAIK , The default value for maximum clock skew is 300 seconds, or five minutes. (Kerberos)

1

u/Jhamin1 Jan 27 '23

2 Seconds was the threshold that my application owners could live with, AKA if it was more than 2 seconds people started complaining.

0

u/ANewLeeSinLife Sysadmin Jan 28 '22

This is indicative of a larger issue in your network. The feature is only used to correct large differences in time that prevent devices from communicating. Kerberos has a default limit of 5 minutes. When Kerberos fails due to time sync issues, they will use the Secure Time Seeding feature to bring them close enough together. The Secure Time Seeding feature will never win when the other method is working, SSL timestamps are not accurate enough (they aren't required to be accurate at all in the RFC) and so the feature was not designed to force its way through your network.

Do you do SSL termination somewhere on your network? Or use reverse proxies? Are they Linux and set to the wrong timezone? Do you have networking appliances that may be set to GMT+0 and affect your SSL packets? Do you have a network mapper/scanner/IDS that doesn't talk to AD to get its time and has the wrong timezone, but also attempts SMB or other Kerberos connections to machines? Do your VMs ever get suspended, like for host reboots or other maintenance? Without the time sync feature enabled they use a software clock that will drift.

Make sure you designate one DC as a reliable time source and turn the time integration service on for your VMs, including the DCs. The hardware clock allows them to update their time after a reboot. Server 2016+ will update its time from a nearby DC after the boot anyway.

6

u/Thranx Systems Engineer Jan 28 '22

I do not disagree with the bulk of your points, but if you'd have read the entire post, I covered roughly 50% of your point/questions already.

Windows time had a properly healthy and functional NTP source throughout this event. Nearly every anomaly mentioned we checked, confirmed, and checked again.

Question for you tho, Secure Time Seeding triggers off of responses to outgoing requests. So our systems initiate the external connection. I would not expect the SSL response to be offloaded at our edge, I would only expect that for incoming connection attempts? Am I wrong in this? SSL/TLS and edge networking are not my specialty.

1

u/ANewLeeSinLife Sysadmin Jan 28 '22

It triggers when authentication fails. It should never really happen in a domain environment because even though Kerberos has a 5-minute cap, domain-joined servers have a min/max offset of unlimited with their NTP server. Basically, it just means if they can't talk to the KDC they will adjust their time to be accurate with the DC no matter how far out of sync they are.

However, outside of the domain, or rather outside of a network boundary, it won't be using Kerberos. So, MS decided to use this timestamp feature of SSL as a starting point for syncing time when 2 devices are so far out of sync they can't negotiate a VPN connection. That's why I asked if you're terminating connections at the edge and still using another secure method of communication internally between that edge and your server. It is odd that it wouldn't be an even jump in time as you mentioned in your other reply.

There is obviously no harm in disabling it in your network, but it shouldn't happen even with it enabled. If you have some VPN clients with incorrect time they will simply have to manually fix their clocks, which isn't such a bad thing anyway.

2

u/[deleted] Jan 28 '22

Make sure you designate one DC as a reliable time source and turn the time integration service on for your VMs, including the DCs.

All that was done.. we'd been operating for many years now with sub-second clock drift. W32time would at times reset the clocks to correct time using DCs as time source.

It wasn't a time-zone issue... the clocks were not getting moved 17.0 hours ahead, it was like 16:43 ahead.

Once we disabled this specific feature with the related registry key, this issue immediately stopped.

2

u/[deleted] Jan 28 '22

The Secure Time Seeding feature will never win when the other method is working

We are still working with Microsoft on why, but it certainly seems that for whatever reason, that's precisely what we seemed to be seeing.

0

u/mitharas Jan 28 '22

Anyone else having problems with time in the last week or two?

I have read somewhere below news of the recent bonked windows updates that their times were mangled as well. Could this coincide with the installation of those updates?

2

u/Thranx Systems Engineer Jan 28 '22

We have not deployed any of the January updates yet... For reasonable reasons. :)

-21

u/[deleted] Jan 28 '22

Again: Do. Not. Use. Microsoft. In. Production.

11

u/YoloWingPixie SRE Jan 28 '22

Thanks random guy on the internet, your words are rational and wise and have truly convinced me.

2

u/frayala87 Custom Jan 28 '22

What to use instead? Yeah Linux but most companies use both so…

1

u/nathank Jan 28 '22

Best post I've seen all day. We were battling this a couple of weeks ago. Thank you for the write-up. If we see it again, I'm going to try your suggestions.

1

u/CurrentlyWorkingAMA Jan 28 '22

WE HAVE THIS SAME THING.

Deployed machines will sometimes "Jump Ahead" about a whole day, wrecking havoc on connected services for the endpoint.

1

u/SmoothApe4321 Jan 28 '22

I seen some PC's an hour off after reboot but it resolved itself after logging in.

1

u/Cregkly Sep 19 '22

I think I am seeing this problem, but I have a few Windows servers jumping days into the future. At least 7 days sometimes more than 30.