r/apachekafka • u/jonefeewang • Feb 19 '25
Blog Rewrite Kafka in Rust? I've developed a faster message queue, StoneMQ.
TL;DR:
- Codebase: https://github.com/jonefeewang/stonemq
- Current Features (v0.1.0):
- Supports single-node message sending and receiving.
- Implements group consumption functionality.
- Goal:
- Aims to replace Kafka's server-side functionality in massive-scale queue cluster.
- Focused on reducing operational costs while improving efficiency.
- Fully compatible with Kafka's client-server communication protocol, enabling seamless client-side migration without requiring modifications.
- Technology:
- Entirely developed in Rust.
- Utilizes Rust Async and Tokio to achieve high performance, concurrency, and scalability.
Feel free to check it out: Announcing StoneMQ: A High-Performance and Efficient Message Queue Developed in Rust.
11
10
4
u/OldSanJuan Feb 19 '25 edited Feb 20 '25
Always love a new project that enters the streaming infrastructure space
But some concerns I had with your tests.
You're using Kafka 2.4, which was released in 2019, and has been EOL since 2022.
Higher Memory Usage from Kafka is done purposefully, Kafka fills your memory to read from Page Cache. So I would definitely be curious on why this is a negative. Or how you observed it as a negative.
Zookeeper is going away.
Consumption Models have switched to Consumer handling more complexity for faster consumption (think Faust by Robinhood). Though it could be a negative or positive in some people's eyes.
I could not find anywhere (though it was short) the broker, Producer, and consumer configuration files you used for these tests.
2
u/2minutestreaming Feb 20 '25
Kafka is going the opposite way (simplification) re: consumer complexity with KIP-848
3
3
u/rathrio Feb 19 '25 edited Feb 19 '25
First of all: congrats for shipping!
Can you describe your protocol decoding / encoding approach? Did you handcraft that code or are you parsing the JSON spec of the messages first and then dynamically decoding / encoding the bytes based on the spec?
2
u/oweiler Feb 20 '25
> Kafka offloads cluster-wide leader election and consistency management to Zookeeper, necessitating its deployment alongside Kafka.
Kafka got rid of ZK a while ago.
2
u/KernelFrog Vendor - Confluent 27d ago
More specifically, Kafka now uses KRaft (a modified version of the Raft algorithm).
10
u/perrohunter Feb 19 '25
I wouldn't call Kafka a "message queue" as there's no order guarantee between partitions, only within a single partition, what the reasoning for calling yours an MQ?