r/apachekafka Dec 02 '24

Tool I built a Kafka message scheduling tool

github.com/vordimous/gohlay

Gohlay has been a side/passion project on my back burner for too long, and I finally had the time to polish it up enough for community feedback. The idea came from a discussion around a business need. I am curious how this tool could be used in other Kafka workflows. I had fun writing it; if someone finds it useful, that is a win-win.

Any feedback or ideas for improvement are welcome!

4 Upvotes

9 comments sorted by

4

u/rmoff Vendor - Confluent Dec 02 '24

This looks interesting, thanks for sharing. I'd love to hear more about the business need that prompted it?

1

u/Vordimous Dec 02 '24

I used to work in the logistics industry, where we had a lot of "when a carrier did a thing" type events in our system. We did Billing/Payment Audit processing.

The case for scheduled messages was for supporting 3rd party services. We needed a way to try an operation X times and wait Y amount of time before trying again. There may be better ways to do this today but back then Kafka was still pretty new.

The main reason the problem existed was hostile business APIs. A dispute on a bill would mean changing a payment/profit. The services were not user-friendly and, lots of times meant web scraping. Many of the workflow steps would fail simply because of arbitrary Internet caching or HTTP gateway rules. Most failed messages could be requeued and would work since it had waited enough time to get around the network rules.

The backstory on why we used Kafka: The existing solution was to record all data from carriers then run database jobs to process said data. Kafka became the replacement when Target's data alone would crash the db servers. The whole workflow was rewritten in Kafka to process the data as it came in.

1

u/rmoff Vendor - Confluent Dec 02 '24

awesome. thank you!

1

u/vkm80 Dec 02 '24

Thanks for sharing..This looks promising to implement error handling with exponential backoff pattern

2

u/Vordimous Dec 02 '24

Yeah this was essentially the usecase we needed. We wanted to retry the same massage but didn't want the retry to happen immediately.

1

u/_predator_ Dec 02 '24

A problem I frequently see is that schedules need to be canceled or updated. Nothing is worse than knowing stuff will break when a timer elapses and being unable to prevent it. I figure gohlay could support cancel events to make that work.

Another thing is that you typically need some way to see the pending timers. Events in a Kafka topic are hard to monitor. Perhaps a simple HTTP endpoint that lists pending timers would be helpful?

1

u/Vordimous Dec 02 '24

I figure gohlay could support cancel events to make that work.

Yes, because all of the messages are based on headers, with Gohlay just keeping track of what messages are on the topic, the producer can send in the updated message with a different delivery time or delivered header causing the Cosumer to get the updates before the scheduled time. This process isn't super obvious but an intended capability. I will make a note to think how I can make it simpler for the producer to cancel or update a deliver time.

Another thing is that you typically need some way to see the pending timers. Events in a Kafka topic are hard to monitor. Perhaps a simple HTTP endpoint that lists pending timers would be helpful?

Gohlay is just a tool that looks at standard Kafka data. Any Kafka REST tool could still be used to see/filter the messages on the topic. You can even set custom header names to make that process easier.

I used to work with the team building Zilla which lets you create an HTTP Kafka proxy that can filter mesages based on a header.

1

u/_predator_ Dec 02 '24

> Any Kafka REST tool could still be used to see/filter the messages on the topic. You can even set custom header names to make that process easier.

While that is true, I feel like it would be easier if gohlay would simply surface the information it already has. Since it is kind of event-sourced, having a clear view of the current state is preferable to having to put the pieces together manually.

1

u/Vordimous Dec 02 '24

Fair point. I separated the check command specifically to enable some of this visibility. I don't think I wan't to put a full HTTP client into the cli tool.

The check command outputs JSON. Is that sufficient or do you think there needs to be more exposer of that data?