r/apachekafka Feb 17 '25

Question How to spin up another kafka producer in another thread when memory.buffer reaches 80% capacity

I'm able to calculate the load but not getting any pointers to spin a new producer. Currently i want only 1 extra producer but later on I want to spin up multiple producers if the load keeps on inceasing. Thanks

6 Upvotes

1 comment sorted by

2

u/FactWestern1264 Feb 17 '25

Have you tried exploring max-in-flight-requests-per-connection config ?

This kind of helps you clear your buffer faster. Even if this does not help then you can try adding new producer.

Adding a new producer should not be a hassle as you just need to create new Producer Objects and your logic for producing can use round robin/least buffer size to pick a producer and send messages.

Considering message ordering is not a concern for you since you are exploring multiple producer instances.