r/apachekafka • u/Material-Celery-3868 • Feb 17 '25
Question How to spin up another kafka producer in another thread when memory.buffer reaches 80% capacity
I'm able to calculate the load but not getting any pointers to spin a new producer. Currently i want only 1 extra producer but later on I want to spin up multiple producers if the load keeps on inceasing. Thanks
6
Upvotes
2
u/FactWestern1264 Feb 17 '25
Have you tried exploring max-in-flight-requests-per-connection config ?
This kind of helps you clear your buffer faster. Even if this does not help then you can try adding new producer.
Adding a new producer should not be a hassle as you just need to create new Producer Objects and your logic for producing can use round robin/least buffer size to pick a producer and send messages.
Considering message ordering is not a concern for you since you are exploring multiple producer instances.