Kafka Configuring Clusters Broker Consideration

Broker Considerations while Configuring Clusters

Having multiple brokers allows us to scale the load across multiple servers. Secondly we can guard against data loss due to single system failures.

Replication will allow us to perform maintainance work without downtime to consumers.

How many brokers?

Disk Capacity

If cluster is require to maintain 10 TB data, and each broker can hold 5 Tb we need 5 brokers.

Replicas can again change this math, if we need to have two replicas per broker then we would need 20Tb.

CPU

Not a major cause of problem but if we have excessive client connections then we need to rev up CPU.

Networking

Broker Configurations

There are only two requirements for a broker to be part of the cluster:

1. They must have same configuration for zookeeper.connect
2. They must have unique broker.id