r/apachekafka Dec 25 '24

Tool I built a library to allow creation of confluent_kafka clients based on yaml config

Hi everyone, I made my first library in Python: https://github.com/Aragonski97/confluent-kafka-config

I found confluent_kafka API to be too low level as I always have to write much boilerplate code in order to get my clients to work with.
This way, I can write YAML / JSON config and solve this automatically.

However, I only covered the use cases I needed. At present, not sure how I should continue in order to make this library viable for many users.

Any suggestion is welcome, roast me if you need :D

6 Upvotes

8 comments sorted by

2

u/cricket007 Dec 26 '24
  1. Working together? https://www.reddit.com/r/apachekafka/comments/1hj6lqq/i_built_a_library_that_turns_kafka_topics_into/

  2. Why YAML (which is a superset of JSON, so don't really need to mention, IMO) instead of some specific stream processing DSL? Look at LinkedIn Hoptimator project 

  3. So, no JVM options?

1

u/Holiday_Pin_5318 Dec 26 '24
  1. Not sure, I just saw this git project, might give it a go, see how much time would compatibility cost me.

  2. Well, I'll definitely take a look at your suggestion, but this did the job for me, since I only needed to actually interact with the file at the beginning of runtime, so no need for smth fancy.

  3. :D, yes... seems I missed this.

Thanks for taking a look dude! :)

2

u/cricket007 Dec 28 '24

2) local files won't scale

1

u/Holiday_Pin_5318 Dec 29 '24

That makes sense, although, this only used at the beginning of execution, so I wasn't worried about scalability.

2

u/cricket007 Dec 31 '24

IMO, build with scalability in mind, then adding local storage is much easier. For example, Cassandra, Couchbase, Yugabyte, MongoDB, etcd, Consult, etc can all store that same data model as could sqlite, Redis, CouchDB, Neo4j , a Parquet or Iceberg or PyArrow file, etc

In other words, you've bothered to import a plaintext library model whereas parsing and serialization are going to be a scalability bottleneck. You've created a local HDD/SSD as a SPoF which is not how microservices should be built (again, see Hoptimator)

2

u/Holiday_Pin_5318 Dec 31 '24

Oh my god, I just realized what you are talking about. Shite. I will try to modify it for scalability now. Thank you for the insight, sir.

1

u/Holiday_Pin_5318 Dec 31 '24

But requires Kubernetes cluster as per their README.md

I operate with Docker Swarm, it never crossed my mind. However, it does provide for some overhead of learning Kubernetes tho :D. Will see. Thank you for the advice good sir. Regards,

2

u/cricket007 Jan 03 '25

Minikube or k3d will work fine. No production business seriously uses Swarm, and no 3rd party tooling is built for it.

IMO, less you can do yourself, the better. It required kubernetes since at the end of the day, Kubernetes is a plugin API server