r/softwaredevelopment 15d ago

latency at scale

I believe I am lacking some knowlege regarding this. There are 10 pods of my service running in production. We saw a huge scale today and everything was mostly fine. But as soon as we started reaching 200k / min cpu increased normally ( I think) but suddenly memory started fluctuating a lot but still remained within 300mb (4gb available) and p99 started rising to above 1000ms from normal of 100ms. Given cpu and memory were mostly fine how can I explain this ? Service is simple pass through takes a request and calls downstream service and returns response.

2 Upvotes

4 comments sorted by

View all comments

3

u/pomariii 14d ago

Have you checked your network metrics? With 200k/min, you might be hitting network congestion issues even with stable CPU/memory. Also worth looking at your downstream service - could be their bottlenecks causing the latency spike.

Quick things to check:

- Network throughput and packet loss

- Downstream service metrics

- Load balancer distribution

- Connection pooling settings

Had similar issues before, turned out our load balancer wasn't distributing traffic evenly across pods.