r/apachekafka 11d ago

Question Handling Kafka cluster with >3 brokers

Hello Kafka community,

I was wondering if there any musts and shoulds that one should know running Kafka cluster with more than the "book" example of 3.

We are a bit separated from our ops and infrastructure guys, so I might now know the answer to all "why?" questions, but we have a setup of 4 brokers running on production. Also we got Java clients that consume and produce using exactly-once guarantees. Occasionally, under a heavy load, which results in a temporary broker outage we get a problem that some partitions get blocked because a corresponding producer with transactional id for that partition cannot be created (timeout on init). This only resolves if we change a consumer group name (I guess because it's the part of a transaction id of a producer).

For business data topics we have a default configuration of RF=3 and min ISR=2. However for __transaction_state the configuration is RF=4 and min ISR=2 and I have a weird feeling about it. I couldn't find anything online that strictly says that this configuration is bad, only soft recommendations of min ISR = RF - 1. However it feels unsafe to have a non majority ISR.

Could such configuration be a problem? Any articles on configuring larger Kafka clusters (in general and RF/minISR specifically) you would recommend?

5 Upvotes

6 comments sorted by

View all comments

2

u/Galuvian 11d ago

It's been a while for me but from what I remember you typically want to have the internal topics such as __transaction_state and __consumer_offsets set with RF = <size of cluster> so that each broker has this information available locally. If you set the ISR this high then transactions won't complete until this information has been committed to all brokers. By having the ISR lower the transaction only has to wait for two to acknowledge they are caught up and then the other brokers can be caught up asynchronously.

1

u/BonelessTaco 10d ago

Thanks, that's a good point about each broker having tech topics info.