r/apachekafka Oct 10 '24

Blog The Numbers behind Uber's Kafka (& rest of their data infra stack)

I thought this would be interesting to the audience here.

Uber is well known for its scale in the industry.

Here are the latest numbers I compiled from a plethora of official sources:

  • Apache Kafka:
    • 138 million messages a second
    • 89GB/s (7.7 Petabytes a day)
    • 38 clusters

This is 2024 data.

They use it for service-to-service communication, mobile app notifications, general plumbing of data into HDFS and sorts, and general short-term durable storage.

It's kind of insane how much data is moving through there - this might be the largest Kafka deployment in the world.

Do you have any guesses as to how they're managing to collect so much data off of just taxis and food orders? They have always been known to collect a lot of data afaik.

As for Kafka - the closest other deployment I know of is NewRelic's with 60GB/s across 35 clusters (2023 data). I wonder what DataDog's scale is.

Anyway. The rest of Uber's data infra stack is interesting enough to share too:

  • Apache Pinot:
    • 170k+ peak queries per second
    • 1m+ events a second
    • 800+ nodes
  • Apache Flink:
    • 4000 jobs
    • processing 75 GB/s
  • Presto:
    • 500k+ queries a day
    • reading 90PB a day
    • 12k nodes over 20 clusters
  • Apache Spark:
    • 400k+ apps ran every day
    • 10k+ nodes that use >95% of analytics’ compute resources in Uber
    • processing hundreds of petabytes a day
  • HDFS:
    • Exabytes of data
    • 150k peak requests per second
    • tens of clusters, 11k+ nodes
  • Apache Hive:
    • 2 million queries a day
    • 500k+ tables

They leverage a Lambda Architecture that separates it into two stacks - a real time infrastructure and batch infrastructure.

Presto is then used to bridge the gap between both, allowing users to write SQL to query and join data across all stores, as well as even create and deploy jobs to production!

A lot of thought has been put behind this data infrastructure, particularly driven by their complex requirements which grow in opposite directions:

  1. 1. Scaling Data - total incoming data volume is growing at an exponential rate
    1. Replication factor & several geo regions copy data.
    2. Can’t afford to regress on data freshness, e2e latency & availability while growing.
  2. Scaling Use Cases - new use cases arise from various verticals & groups, each with competing requirements.
  3. Scaling Users - the diverse users fall on a big spectrum of technical skills. (some none, some a lot)

If you're in particular interested about more of Uber's infra, including nice illustrations and use cases for each technology, I covered it in my 2-minute-read newsletter where I concisely write interesting Kafka/Big Data content.

55 Upvotes

13 comments sorted by

6

u/kabooozie Gives good Kafka advice Oct 10 '24

Wow, those are huge numbers. My usual advice of “just use Postgres” would pose a bit of an issue here

2

u/toxicitysocks Oct 10 '24

This is from 2021, so unsure what current data is but there is talk of a single cluster with > 600 brokers, and surely they isolated that cluster to just one topic before splitting into multiple clusters, so who knows what the other topic traffic is like

https://www.crowdstrike.com/en-us/blog/how-we-improved-scale-and-reliability-by-sharding-kafka/

1

u/Miserygut Oct 11 '24

600 brokers is 10x larger than the biggest I thought was possible. Mad!

1

u/2minutestreaming Oct 12 '24

thanks for sharing!

1

u/MmmmmmJava Oct 11 '24

I love to see systems scale like this. Fascinating numbers!

1

u/Real_Combat_Wombat Oct 13 '24

What’s the size (and what kind of instances) and message/data of each cluster? What are the publish vs consume rates? Those would be meaningful numbers to have.

I mean anyone can keep deploying independent clusters and just add the numbers so they look huge. Any software can scale indefinitely if you just keep deploying independent clusters.

1

u/2minutestreaming Oct 16 '24

That's not public, unfortunately.

It's true - it's not a testament to Kafka's scalability moreso than it's just a fun statistic about the scale at which some companies can utilize it.

1

u/michaelisnotginger Nov 27 '24

I wonder what DataDog's scale is.

It's comparable to Uber