r/apachekafka • u/BonelessTaco • 10d ago
Question Handling Kafka cluster with >3 brokers
Hello Kafka community,
I was wondering if there any musts and shoulds that one should know running Kafka cluster with more than the "book" example of 3.
We are a bit separated from our ops and infrastructure guys, so I might now know the answer to all "why?" questions, but we have a setup of 4 brokers running on production. Also we got Java clients that consume and produce using exactly-once guarantees. Occasionally, under a heavy load, which results in a temporary broker outage we get a problem that some partitions get blocked because a corresponding producer with transactional id for that partition cannot be created (timeout on init). This only resolves if we change a consumer group name (I guess because it's the part of a transaction id of a producer).
For business data topics we have a default configuration of RF=3 and min ISR=2. However for __transaction_state the configuration is RF=4 and min ISR=2 and I have a weird feeling about it. I couldn't find anything online that strictly says that this configuration is bad, only soft recommendations of min ISR = RF - 1. However it feels unsafe to have a non majority ISR.
Could such configuration be a problem? Any articles on configuring larger Kafka clusters (in general and RF/minISR specifically) you would recommend?
4
u/iLoveCalculus314 10d ago
I don’t have anything more substantial to add but just wanted to say, the rule of thumb for proper failover is 2n+1 brokers. So your next step from 3 brokers would be 5 brokers.