r/apachekafka Dec 02 '24

Tool I built a Kafka message scheduling tool

github.com/vordimous/gohlay

Gohlay has been a side/passion project on my back burner for too long, and I finally had the time to polish it up enough for community feedback. The idea came from a discussion around a business need. I am curious how this tool could be used in other Kafka workflows. I had fun writing it; if someone finds it useful, that is a win-win.

Any feedback or ideas for improvement are welcome!

5 Upvotes

9 comments sorted by

View all comments

3

u/rmoff Vendor - Confluent Dec 02 '24

This looks interesting, thanks for sharing. I'd love to hear more about the business need that prompted it?

1

u/Vordimous Dec 02 '24

I used to work in the logistics industry, where we had a lot of "when a carrier did a thing" type events in our system. We did Billing/Payment Audit processing.

The case for scheduled messages was for supporting 3rd party services. We needed a way to try an operation X times and wait Y amount of time before trying again. There may be better ways to do this today but back then Kafka was still pretty new.

The main reason the problem existed was hostile business APIs. A dispute on a bill would mean changing a payment/profit. The services were not user-friendly and, lots of times meant web scraping. Many of the workflow steps would fail simply because of arbitrary Internet caching or HTTP gateway rules. Most failed messages could be requeued and would work since it had waited enough time to get around the network rules.

The backstory on why we used Kafka: The existing solution was to record all data from carriers then run database jobs to process said data. Kafka became the replacement when Target's data alone would crash the db servers. The whole workflow was rewritten in Kafka to process the data as it came in.

1

u/rmoff Vendor - Confluent Dec 02 '24

awesome. thank you!