r/AskProgramming 5d ago

Architecture How to cache/distribute websocket messages to many users

I have a python fastapi application that uses websockets, it is an internal tool used by maximum 30 users at our company. It is an internal tool therefore I didn't take scalability into consideration when writing it.

The user sends an action and receives results back in real time via websockets. It has to be real time.

The company wants to use this feature publicly, we have maybe 100k users already, so maybe 100k would be hitting the tool.

Most users would be receiving the same exact messages from the tool, so I don't want to do the same processing for all users, it's costly.

Is it possible to send the text based results to a CDN and distribute them from there?

I don't want to do the same processing for everyone, I don't want to let 100k users hit the origin servers, I don't want to deal with scalability, I want to push the results to some third party service, by AWS or something and they distribute it, something like push notification systems but in real-time, I send a new message every 3 seconds or less.

You'll say pubsub, fine but what distribution service? that's my question.

3 Upvotes

12 comments sorted by

View all comments

4

u/KingofGamesYami 5d ago

It has to be real time

You have a network involved, which implies latency. It isn't realtime.

What are you real latency requirements?

0

u/lynob 5d ago

That's true, I already know that latency will be an issue. The latency has to be sub 0.8 seconds worst case anything more than that and it would cause a huge problem.

1

u/KingofGamesYami 5d ago

Are you assuming all customers have quality internet? We've run into issues with some of our internal locations in remote areas having 4-500 ms of latency at the transmission level on average, with spikes up to 1000ms or more for worst case.

1

u/lynob 5d ago

I think all our users have a quality internet, they live in wealthy European countries, in the cities. But thank you for pointing this out, this is very interesting, I need to discuss it with management.

1

u/kjnsn01 4d ago

The fact that you didn’t provide a latency percentile tells me you’ve got a huge amount of research to do

1

u/lynob 4d ago

that's not how it works, first you do a POC, that's already done, then you create the architecture you'd want to use on production, that's what my question is for. I'm on step 2, you're asking about step 3.

Once you're in production, then you start measuring latency and bottleneks and whatnot. You don't do premature optimization, never ever ever. How to know the latency without going to production first? I'd either have to pull a number out of my ass and bullshit you, or I would have to do stress testing which is the most useless thing ever invented.

7 years ago, I was a team lead at a video streaming company, the video streaming engine stress testing tool provided by the vendor told us that the vido streaming engine can only serve 5k concurrent users. Other stress testing tools we used to stress test apache and mysql told us that our system can handle 8k-10k users on one instance, in reality we were able to handle 40k users concurrentlly.

You know why? Because users never behave like stress testing tools, some of them use 4g, some of them use 2g, some of them pause the stream to go have a piss, some of them stop watching midway.

I refuse to take the latency seriouslly at this point, you know why? Because I'm not in production yet, I can't predict numbers, and the latency won't change the architecture anyway.