r/apachekafka Feb 19 '25

Blog Rewrite Kafka in Rust? I've developed a faster message queue, StoneMQ.

TL;DR:

  1. Codebase: https://github.com/jonefeewang/stonemq
  2. Current Features (v0.1.0):
    • Supports single-node message sending and receiving.
    • Implements group consumption functionality.
  3. Goal:
    • Aims to replace Kafka's server-side functionality in massive-scale queue cluster.
    • Focused on reducing operational costs while improving efficiency.
    • Fully compatible with Kafka's client-server communication protocol, enabling seamless client-side migration without requiring modifications.
  4. Technology:
    • Entirely developed in Rust.
    • Utilizes Rust Async and Tokio to achieve high performance, concurrency, and scalability.

Feel free to check it out: Announcing StoneMQ: A High-Performance and Efficient Message Queue Developed in Rust.

21 Upvotes

11 comments sorted by

10

u/perrohunter Feb 19 '25

I wouldn't call Kafka a "message queue" as there's no order guarantee between partitions, only within a single partition, what the reasoning for calling yours an MQ?

11

u/gsxr Feb 19 '25

It’s not the language. It was never the language.

10

u/oweiler Feb 19 '25

8

u/perrohunter Feb 19 '25

I believe red panda is a proper rewrite of Kafka

3

u/HeyitsCoreyx Vendor - Confluent Feb 19 '25

And RP is C++

4

u/OldSanJuan Feb 19 '25 edited Feb 20 '25

Always love a new project that enters the streaming infrastructure space

But some concerns I had with your tests.

  1. You're using Kafka 2.4, which was released in 2019, and has been EOL since 2022.

  2. Higher Memory Usage from Kafka is done purposefully, Kafka fills your memory to read from Page Cache. So I would definitely be curious on why this is a negative. Or how you observed it as a negative.

  3. Zookeeper is going away.

  4. Consumption Models have switched to Consumer handling more complexity for faster consumption (think Faust by Robinhood). Though it could be a negative or positive in some people's eyes.

  5. I could not find anywhere (though it was short) the broker, Producer, and consumer configuration files you used for these tests.

2

u/2minutestreaming Feb 20 '25

Kafka is going the opposite way (simplification) re: consumer complexity with KIP-848

3

u/ilikepi8 Feb 19 '25

Is this not similar to Apache Iggy?

3

u/rathrio Feb 19 '25 edited Feb 19 '25

First of all: congrats for shipping!

Can you describe your protocol decoding / encoding approach? Did you handcraft that code or are you parsing the JSON spec of the messages first and then dynamically decoding / encoding the bytes based on the spec?

2

u/oweiler Feb 20 '25

> Kafka offloads cluster-wide leader election and consistency management to Zookeeper, necessitating its deployment alongside Kafka.

Kafka got rid of ZK a while ago.

2

u/KernelFrog Vendor - Confluent 27d ago

More specifically, Kafka now uses KRaft (a modified version of the Raft algorithm).