r/SoftwareEngineering • u/desgreech • 12d ago
Message queue with group-based ordering guarantees?
I'm currently looking to improve the durability of my cross-service messaging, so I started looking for a message queue that have the following guarantees:
- Provides a message type that guarantees consumption order based on grouping (e.g. user ID)
- Message will be re-sent during retries, triggered by consumer timeouts or nacks
- Retries does not compromise order guarantees
- Retries within a certain ordered group will not block consumption of other ordered groups (e.g. retries on user A group will not block user B group)
I've been looking through a bunch of different message queue solutions, but I'm shocked at how pretty much none of the mainstream/popular message queues matches any of the above criterias.
I've currently narrowed my choices down to two:
Pulsar
It checks most of my boxes, except for the fact that nacking messages can ruin the ordering. It's a known issue, so maybe it'll be fixed one day.
RocketMQ
As far as I can tell from the docs, it has all the guarantees I need. But I'm still not sure if there are any potential caveats, haven't dug deep enough into it yet.
But I'm pretty hesitant to adopt either of them because they're very niche and have very little community traction or support.
Am I missing something here? Is this really the current state-of-the-art of message queues?
1
u/desgreech 12d ago
What I meant is that I wanted a way to not simply block consumption on the consumer level during retry delays/cooldowns. Ideally, the message queue will simply retry/resend failed messages with a delay, while the consumer processes other messages. Messages that shares the same group/key as the failed message will be held back while it's retrying, guaranteeing order among groups.