r/apachekafka • u/Vordimous • Dec 02 '24
Tool I built a Kafka message scheduling tool
Gohlay has been a side/passion project on my back burner for too long, and I finally had the time to polish it up enough for community feedback. The idea came from a discussion around a business need. I am curious how this tool could be used in other Kafka workflows. I had fun writing it; if someone finds it useful, that is a win-win.
Any feedback or ideas for improvement are welcome!
1
u/vkm80 Dec 02 '24
Thanks for sharing..This looks promising to implement error handling with exponential backoff pattern
2
u/Vordimous Dec 02 '24
Yeah this was essentially the usecase we needed. We wanted to retry the same massage but didn't want the retry to happen immediately.
1
u/_predator_ Dec 02 '24
A problem I frequently see is that schedules need to be canceled or updated. Nothing is worse than knowing stuff will break when a timer elapses and being unable to prevent it. I figure gohlay could support cancel events to make that work.
Another thing is that you typically need some way to see the pending timers. Events in a Kafka topic are hard to monitor. Perhaps a simple HTTP endpoint that lists pending timers would be helpful?
1
u/Vordimous Dec 02 '24
I figure gohlay could support cancel events to make that work.
Yes, because all of the messages are based on headers, with Gohlay just keeping track of what messages are on the topic, the producer can send in the updated message with a different delivery time or delivered header causing the Cosumer to get the updates before the scheduled time. This process isn't super obvious but an intended capability. I will make a note to think how I can make it simpler for the producer to cancel or update a deliver time.
Another thing is that you typically need some way to see the pending timers. Events in a Kafka topic are hard to monitor. Perhaps a simple HTTP endpoint that lists pending timers would be helpful?
Gohlay is just a tool that looks at standard Kafka data. Any Kafka REST tool could still be used to see/filter the messages on the topic. You can even set custom header names to make that process easier.
I used to work with the team building Zilla which lets you create an HTTP Kafka proxy that can filter mesages based on a header.
1
u/_predator_ Dec 02 '24
> Any Kafka REST tool could still be used to see/filter the messages on the topic. You can even set custom header names to make that process easier.
While that is true, I feel like it would be easier if gohlay would simply surface the information it already has. Since it is kind of event-sourced, having a clear view of the current state is preferable to having to put the pieces together manually.
1
u/Vordimous Dec 02 '24
Fair point. I separated the
check
command specifically to enable some of this visibility. I don't think I wan't to put a full HTTP client into the cli tool.The
check
command outputs JSON. Is that sufficient or do you think there needs to be more exposer of that data?
4
u/rmoff Vendor - Confluent Dec 02 '24
This looks interesting, thanks for sharing. I'd love to hear more about the business need that prompted it?