You can configure partitions in Kafka to process messages in parallel and enable concurrent message consumption.
The number of partitions must be equal to the number of concurrent consumers of the client. If the number of partitions is less than the number of consumers, some consumer instances will not receive messages to process. If the number of partitions is more than the number of consumers, some consumer instances will process messages from several partitions.
The following image shows how a Kafka listener works with partitions in a Secure Agent group:
In a Secure Agent cluster, the Kafka consumer is active only on one node. However, if the same Kafka consumer runs on a Secure Agent group, it processes messages in parallel across all the nodes in the group. Kafka distributes messages among client instances without duplication.
The Kafka partitions and the Kafka consumers that run on a Secure Agent group distribute the load between the Secure Agent nodes. This configuration also scales to process more data when you add more partitions and Secure Agents to the group. This increases the Kafka message throughput.
You can configure multiple concurrent consumers by enabling the load balancing option in the Event Sources tab of the Kafka connection. You can use this option if your Process Server uses a Secure Agent cluster configuration so that it creates one consumer per Secure Agent.
Ensure that the messages are evenly distributed between partitions so that all the Secure Agents in a group are equally loaded. This will provide maximum throughput and evenly share the load on the Secure Agents.
For more information about Kafka partitioning, see the Kafka documentation.
