Kafka Partition Assignment Strategies

Partition Assignment Strategies in Kafka Consumers🔗

Range🔗

In Kafka, the Range assignment strategy determines how partitions are distributed among consumers in a consumer group.

When using Range assignment:

Each consumer is assigned a consecutive range of partitions from each topic it subscribes to.
The assignment is done separately for each topic.

For example:

Suppose there are two consumers: C1 and C2
And two topics: T1 and T2
Each topic has three partitions: 0, 1, and 2

Under Range assignment:

For topic T1:
C1 gets partitions 0 and 1
C2 gets partition 2
For topic T2:
C1 again gets partitions 0 and 1
C2 gets partition 2

So, C1 ends up with four partitions in total (two from each topic), while C2 gets only two partitions.

This imbalance happens because the Range assignor works topic by topic, and if the number of partitions is not evenly divisible by the number of consumers, the earlier consumers in the list get slightly more partitions.

In summary:

Range assignment divides partitions in order and independently per topic.
It can lead to uneven load distribution if partitions do not divide evenly among consumers.

Round Robin🔗

In Kafka, the RoundRobin assignment strategy distributes partitions more evenly across consumers in a consumer group.

Here’s how it works:

It takes all partitions from all subscribed topics and then assigns them to consumers one by one in order.
This process continues in a round-robin fashion until all partitions are assigned.

For example, suppose you have:

Two consumers: C1 and C2
Two topics: T1 and T2
Each topic has three partitions: 0, 1, and 2

Under RoundRobin assignment:

C1 gets:
Partitions T1-0, T1-2, and T2-1
C2 gets:
Partitions T1-1, T2-0, and T2-2

Because partitions are assigned one at a time across all topics, the distribution is more balanced.

In general:

If all consumers subscribe to the same set of topics (which is common), RoundRobin ensures that each consumer gets nearly the same number of partitions, with at most a one-partition difference.
This makes it a good choice for maintaining load balance among consumers.

Sticky🔗

The Sticky Assignor in Kafka is a partition assignment strategy designed to achieve two main objectives:

Balanced partition distribution: It tries to assign partitions across consumers as evenly as possible, ensuring that each consumer handles roughly the same amount of data and workload.
Minimal partition movement during rebalances: When a rebalance occurs (for example, when a consumer joins or leaves the group), the Sticky Assignor tries to keep existing partition assignments unchanged as much as possible. This minimizes the need to move partitions between consumers, which reduces processing interruptions and improves stability.

How It Works🔗

In the initial assignment, when consumers first join the group, the Sticky Assignor distributes partitions almost exactly like the RoundRobin assignor — evenly across consumers.
On subsequent rebalances, instead of completely reshuffling all partitions, it keeps as many existing assignments as possible and only moves the partitions that need to be reassigned to maintain balance.

Example🔗

If two consumers C1 and C2 are assigned partitions from a topic and C3 joins the group:

The Sticky Assignor will not reassign all partitions.
It will only move a few partitions from C1 and C2 to C3 so that the new distribution is still balanced but with minimal movement.

If consumers in the same group subscribe to different topics, the Sticky Assignor performs even better than the RoundRobin Assignor. It achieves a more balanced assignment across topics while still minimizing unnecessary partition shuffling.

Summary🔗

Property	Sticky Assignor Behavior
Partition balance	As balanced as RoundRobin
Rebalance behavior	Minimizes partition movement
Efficiency	Reduces overhead during rebalances
Handling of different topic subscriptions	More balanced than RoundRobin

In short, the Sticky Assignor provides both load balance and stability by keeping partition assignments steady across rebalances while ensuring even distribution.

Cooperative Sticky Rebalancer🔗

Good question — the Sticky Assignor and the Cooperative Sticky Assignor are closely related, but the main difference lies in how they handle rebalancing and the level of disruption caused when consumers join or leave a group.

Let’s break it down clearly.

1. Sticky Assignor (also known as Eager Sticky Assignor)🔗

Rebalance behavior: Eager (full) rebalance

When a rebalance occurs (for example, a consumer joins or leaves the group), all consumers in the group stop consuming.
All current partition assignments are revoked, and the group coordinator reassigns partitions to all consumers from scratch.
Even though the assignor tries to keep the new assignment as “sticky” as possible (minimizing partition movements), all consumers must pause until the rebalance completes.
This can cause short processing interruptions and temporary underutilization of resources.

Analogy: Think of it as reshuffling all cards in a deck every time a new player joins, even if most cards could have stayed where they were.

Advantages:

Simple to reason about.
Ensures fully consistent assignments.

Disadvantages:

Causes temporary downtime during rebalances.
Can lead to throughput drops when consumer group membership changes frequently.

2. Cooperative Sticky Assignor🔗

Rebalance behavior: Incremental (cooperative) rebalance

The Cooperative Sticky Assignor builds on the Sticky Assignor but adds incremental rebalancing.
Instead of revoking all assignments, it only revokes and reassigns the partitions that need to move.
Other consumers keep consuming from their current partitions while the reassignment takes place.
The rebalance process is incremental and non-disruptive — only affected consumers pause briefly.

Analogy: If a new player joins, you only take a few cards from others to give them, without stopping the whole game.