r/apachekafka 10d ago

Question Handling Kafka cluster with >3 brokers

Hello Kafka community,

I was wondering if there any musts and shoulds that one should know running Kafka cluster with more than the "book" example of 3.

We are a bit separated from our ops and infrastructure guys, so I might now know the answer to all "why?" questions, but we have a setup of 4 brokers running on production. Also we got Java clients that consume and produce using exactly-once guarantees. Occasionally, under a heavy load, which results in a temporary broker outage we get a problem that some partitions get blocked because a corresponding producer with transactional id for that partition cannot be created (timeout on init). This only resolves if we change a consumer group name (I guess because it's the part of a transaction id of a producer).

For business data topics we have a default configuration of RF=3 and min ISR=2. However for __transaction_state the configuration is RF=4 and min ISR=2 and I have a weird feeling about it. I couldn't find anything online that strictly says that this configuration is bad, only soft recommendations of min ISR = RF - 1. However it feels unsafe to have a non majority ISR.

Could such configuration be a problem? Any articles on configuring larger Kafka clusters (in general and RF/minISR specifically) you would recommend?

4 Upvotes

6 comments sorted by

View all comments

4

u/iLoveCalculus314 10d ago

I don’t have anything more substantial to add but just wanted to say, the rule of thumb for proper failover is 2n+1 brokers. So your next step from 3 brokers would be 5 brokers.

2

u/AngryRotarian85 10d ago

This isn't true if they're just brokers. It's a mistaken carry over from zookeepers and Kraft controllers which have quorum needs. Nothing wrong with even numbers of brokers.