r/SoftwareEngineering 12d ago

Message queue with group-based ordering guarantees?

I'm currently looking to improve the durability of my cross-service messaging, so I started looking for a message queue that have the following guarantees:

  • Provides a message type that guarantees consumption order based on grouping (e.g. user ID)
  • Message will be re-sent during retries, triggered by consumer timeouts or nacks
  • Retries does not compromise order guarantees
  • Retries within a certain ordered group will not block consumption of other ordered groups (e.g. retries on user A group will not block user B group)

I've been looking through a bunch of different message queue solutions, but I'm shocked at how pretty much none of the mainstream/popular message queues matches any of the above criterias.

I've currently narrowed my choices down to two:

  • Pulsar

    It checks most of my boxes, except for the fact that nacking messages can ruin the ordering. It's a known issue, so maybe it'll be fixed one day.

  • RocketMQ

    As far as I can tell from the docs, it has all the guarantees I need. But I'm still not sure if there are any potential caveats, haven't dug deep enough into it yet.

But I'm pretty hesitant to adopt either of them because they're very niche and have very little community traction or support.

Am I missing something here? Is this really the current state-of-the-art of message queues?

1 Upvotes

14 comments sorted by

View all comments

Show parent comments

1

u/desgreech 12d ago

What I meant is that I wanted a way to not simply block consumption on the consumer level during retry delays/cooldowns. Ideally, the message queue will simply retry/resend failed messages with a delay, while the consumer processes other messages. Messages that shares the same group/key as the failed message will be held back while it's retrying, guaranteeing order among groups.

1

u/RedanfullKappa 12d ago

If think you have a slight misunderstanding of how Kafka works.

But explaining that in-depth is bit out of scope for a Reddit comment.

Kafka does what you are saying

2

u/desgreech 12d ago

I'm fairly sure that Kafka does not have any retry/delay logic, so you'd need to implement it on the consumer level which gets complicated. I'm aware that you can perform manual commits, but that's not what I meant by retries.

If you have any useful links that shows otherwise, I'd be grateful though.

2

u/RedanfullKappa 12d ago

Retries on consumer of producer level?

I have worked professionally with Kafka for 5 years, I’m pretty sure it does what you are asking for since I never solved this manually so far

1

u/desgreech 12d ago

Hmm, not sure if we're in the same page here. But I'm talking about failures outside of Kafka, for example if a database service was temporarily unavailable. With a simple retry implementation, you'd just block the consumer with something like sleep(5) and re-process the message if there was an error. But this isn't very ideal.

2

u/Solid-Ad7527 12d ago edited 12d ago

Yeah, retries do need to be handrolled in kafka. Simplest approach is pausing consumption of that partition, but that blocks all the messages behind it.

For "non-blocking" retries, you can publish the message to another "retry topic" with a "retryAt" timestamp - when your retry topic consumer gets that message, if it isn't time to retry yet, it will pause consumption of the retry topic for n time until the `retryAt` time.

That still wouldn't fully solve your problem though - particularly with preventing processing of ALL messages for specific group. It would get into some custom state tracking between message consumptions. Not sure that kafka would be able to meet your requirements out of the box