1

We want to install Kafka Kraft cluster on 35 machines ( machines are DELL physical machines - type R760 And OS is Linux RHEL 8.6 version )

Since we are dealing with Kafka production cluster , we want to understand what is the best Practice and the right approach to install the machines and configuration

The question about the cluster that we are preparing is about the controllers

Do we need to set and install the controllers on dedicated machines and not on Kafka machines itself?

for example I gives here PLAN1 and PLAN2

PLAN1

We can install 3 controllers on the first 3 Kafka machines ( means both PID's are running on Kafka machine as broker pid and controller pid [ pid - Linux process id ]

kafka1-3 will include pid for broker and pid for controller

kafka4+ will include only pid for broker

enter image description here

PLAN2

We can install 3 controllers on different dedicated machines

35 physical machines for Kafka Kraft service

3 dedicated VM machines for Kafka controllers ( VM machines since from our understanding controllers are not consume so much resources )

enter image description here

The second question , is about how many controllers services we need?

Is it always 3 controllers for 3 Kafka machines or for 15 Kafka machines or for 35 Kafka machines ?

Improve this question

1 Answer 1

Reset to default
0

The question is old but for completeness:

  1. Technically you can install controller as separate process on existing brokers (i.e. if you have brokers 1,2,3 you can run controller as separate process/jvm). In that case though there is chance your controller process might have to fight for resources with broker process unless you isolate it with cgroups or containers. Another simpler option is to have dual role. You can configure broker to be controller as well. Generally recommended in small or staging setups. For production I'd probably go with dedicated controllers (on own separate VMs).
  2. You need exactly 3 controllers. It doesn't matter how many brokers you have. 3 controllers with 3 brokers. Or 3 controllers with 100 brokers. Important is that the number of controllers is odd so they can form quorum. You could have i.e. 5 controllers if you need big resiliency but I've read somewhere in KIP it's not really supported as of Kafka ver 3.8 (late 2024).
Improve this answer

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.