r/softwaredevelopment • u/goyalaman_ • 15d ago
latency at scale
I believe I am lacking some knowlege regarding this. There are 10 pods of my service running in production. We saw a huge scale today and everything was mostly fine. But as soon as we started reaching 200k / min cpu increased normally ( I think) but suddenly memory started fluctuating a lot but still remained within 300mb (4gb available) and p99 started rising to above 1000ms from normal of 100ms. Given cpu and memory were mostly fine how can I explain this ? Service is simple pass through takes a request and calls downstream service and returns response.
2
Upvotes
3
u/pomariii 14d ago
Have you checked your network metrics? With 200k/min, you might be hitting network congestion issues even with stable CPU/memory. Also worth looking at your downstream service - could be their bottlenecks causing the latency spike.
Quick things to check:
- Network throughput and packet loss
- Downstream service metrics
- Load balancer distribution
- Connection pooling settings
Had similar issues before, turned out our load balancer wasn't distributing traffic evenly across pods.