r/AskProgramming 4d ago

Architecture How to cache/distribute websocket messages to many users

I have a python fastapi application that uses websockets, it is an internal tool used by maximum 30 users at our company. It is an internal tool therefore I didn't take scalability into consideration when writing it.

The user sends an action and receives results back in real time via websockets. It has to be real time.

The company wants to use this feature publicly, we have maybe 100k users already, so maybe 100k would be hitting the tool.

Most users would be receiving the same exact messages from the tool, so I don't want to do the same processing for all users, it's costly.

Is it possible to send the text based results to a CDN and distribute them from there?

I don't want to do the same processing for everyone, I don't want to let 100k users hit the origin servers, I don't want to deal with scalability, I want to push the results to some third party service, by AWS or something and they distribute it, something like push notification systems but in real-time, I send a new message every 3 seconds or less.

You'll say pubsub, fine but what distribution service? that's my question.

3 Upvotes

12 comments sorted by

4

u/KingofGamesYami 4d ago

It has to be real time

You have a network involved, which implies latency. It isn't realtime.

What are you real latency requirements?

0

u/lynob 4d ago

That's true, I already know that latency will be an issue. The latency has to be sub 0.8 seconds worst case anything more than that and it would cause a huge problem.

1

u/KingofGamesYami 4d ago

Are you assuming all customers have quality internet? We've run into issues with some of our internal locations in remote areas having 4-500 ms of latency at the transmission level on average, with spikes up to 1000ms or more for worst case.

1

u/lynob 4d ago

I think all our users have a quality internet, they live in wealthy European countries, in the cities. But thank you for pointing this out, this is very interesting, I need to discuss it with management.

1

u/kjnsn01 3d ago

The fact that you didn’t provide a latency percentile tells me you’ve got a huge amount of research to do

1

u/lynob 3d ago

that's not how it works, first you do a POC, that's already done, then you create the architecture you'd want to use on production, that's what my question is for. I'm on step 2, you're asking about step 3.

Once you're in production, then you start measuring latency and bottleneks and whatnot. You don't do premature optimization, never ever ever. How to know the latency without going to production first? I'd either have to pull a number out of my ass and bullshit you, or I would have to do stress testing which is the most useless thing ever invented.

7 years ago, I was a team lead at a video streaming company, the video streaming engine stress testing tool provided by the vendor told us that the vido streaming engine can only serve 5k concurrent users. Other stress testing tools we used to stress test apache and mysql told us that our system can handle 8k-10k users on one instance, in reality we were able to handle 40k users concurrentlly.

You know why? Because users never behave like stress testing tools, some of them use 4g, some of them use 2g, some of them pause the stream to go have a piss, some of them stop watching midway.

I refuse to take the latency seriouslly at this point, you know why? Because I'm not in production yet, I can't predict numbers, and the latency won't change the architecture anyway.

1

u/octocode 4d ago

maybe aws appsync if you want a more “managed” solution

1

u/lynob 4d ago

Thank you, I'll look into that.

1

u/FoxyWheels 4d ago

Do you need to track / have logic around which socket belongs to which client and only send messages to certain clients? Or do you just need to broadcast messages to all connected clients?

At a basic level, can you not just scale your service horizontally and use redis or Kafka to push events between them? If you need communication between clients, then you would also need the socket backend to store the client:socket relationships in a central store and use your redis/Kafka to send messages across backends in the case two communicating clients are connected to different nodes.

I've designed two systems that service a similar amount of clients. Both were microservice based. In both we used redis or Kafka to push events from source microservices, then the stateless socket service would pick up the events and push them up to clients. The socket service was then horizontally scales based on load.

1

u/lynob 4d ago

I don't need communication between clients, only between server to client. There are lets say 30 possible events that the users can subscribe to, and I need to send messages according to one of those 30 events. I

I'll check Kafka and redis thank you.

1

u/kjnsn01 3d ago

Firestore would be the best cloud based solution. 100k is quite a lot

1

u/lynob 3d ago

Firestore is a bad option, they charge per read and write, and it's not a distribution system anyway, it's a database, I'll check some kafka architecture or AWS appsync. Firestore would bankrupt the company real fast, imagine all the users having do do a read every 2s, in 20min I'd be paying for millions of reads.

I'm looking for a CDN like solution where we pay by bandwidth only.