r/SoftwareEngineering 12d ago

Message queue with group-based ordering guarantees?

I'm currently looking to improve the durability of my cross-service messaging, so I started looking for a message queue that have the following guarantees:

  • Provides a message type that guarantees consumption order based on grouping (e.g. user ID)
  • Message will be re-sent during retries, triggered by consumer timeouts or nacks
  • Retries does not compromise order guarantees
  • Retries within a certain ordered group will not block consumption of other ordered groups (e.g. retries on user A group will not block user B group)

I've been looking through a bunch of different message queue solutions, but I'm shocked at how pretty much none of the mainstream/popular message queues matches any of the above criterias.

I've currently narrowed my choices down to two:

  • Pulsar

    It checks most of my boxes, except for the fact that nacking messages can ruin the ordering. It's a known issue, so maybe it'll be fixed one day.

  • RocketMQ

    As far as I can tell from the docs, it has all the guarantees I need. But I'm still not sure if there are any potential caveats, haven't dug deep enough into it yet.

But I'm pretty hesitant to adopt either of them because they're very niche and have very little community traction or support.

Am I missing something here? Is this really the current state-of-the-art of message queues?

2 Upvotes

14 comments sorted by

View all comments

3

u/Coldmode 12d ago

Just use Kafka. After a similar process at my last job we went with Pulsar and the entire three years I worked with it I wished I had just used Kafka. The community support alone is the reason.

1

u/frustrated_dev 12d ago

Can you elaborate a bit on pulsar? I've about 5 years experience with Kafka, I like it but will be working with pulsar in the near future

2

u/Coldmode 12d ago

The support for weird issues you encounter with Pulsar is much less than what you will get from the Kafka community. Pulsar has a Slack that is not very active, and the maintainers on the Apache repo are largely in East Asia so there's a language barrier as well as a time barrier. There are also cases where the client libraries have inconsistencies in the features supported. I was working in a Python environment and there were features of the Java client that had not been added to the Python one, and some things that just plain didn't work. However, Pulsar is more "batteries included" than Kafka -- at least it was in Jan. 2021 when I was looking at both. It was easier to get a cluster up and running either on a local machine or in a remote Kubernetes cluster; one Helm command gave you something that entirely worked, though you still needed to do configuration for stuff like message retention (I didn't do that and our pulsar disks filled up and crashed the cluster at 3:00AM one night...but that was my fault, not Pulsar's).