Skip to content

Are Kafka Consumers Thread Safe🔗

Kafka consumers are not thread-safe.

1. What “thread safety” means🔗

Imagine a library.

  • You have one librarian (the Kafka consumer).
  • You have multiple helpers (threads).
  • They all want to check out books (poll messages from Kafka).

If all the helpers try to use the same librarian’s checkout desk at the same time — things get confusing:

  • Two helpers reach for the same book.
  • The librarian loses track of who took what.
  • Records get mixed up.

That’s what happens when multiple threads use a non-thread-safe object at the same time.

2. Kafka’s consumer = that single librarian🔗

Kafka’s consumer object (the one you create from KafkaConsumer) is not thread-safe.

That means:

  • Only one thread should call its methods (poll(), commitSync(), close(), etc.) at a time.
  • If multiple threads touch it, you can get data corruption, missed messages, or weird crashes.

3. Why KafkaConsumer is not thread-safe🔗

Kafka’s consumer manages a lot of state internally — things like:

  • The list of partitions it owns.
  • The last offsets it read.
  • The last committed offsets.
  • Heartbeats (to the group coordinator).
  • The network connection to the broker.

All of that is stored inside the consumer object, not in Kafka itself.

If two threads try to change that state at once:

  • One might be polling new messages.
  • Another might be committing offsets.
  • Another might be closing the consumer.

Result:

  • Messages get processed twice or skipped.
  • Kafka thinks the consumer “died” (missed heartbeats).
  • You might even get a ConcurrentModificationException.

So Kafka’s design keeps things simple:

“One consumer instance = one thread.”

4. What happens if you ignore this rule🔗

If multiple threads share a single KafkaConsumer, you can see things like:

  • ConcurrentModificationException
  • IllegalStateException: Consumer is not subscribed to any topics
  • CommitFailedException
  • Random missing messages
  • Consumer group instability (constant rebalances)

All of these are signs that more than one thread is calling the consumer’s methods.

5. The correct way: one consumer per thread🔗

Back to our library analogy.

If you have 3 helpers who each want to check out books, the right way is:

  • Give each helper their own librarian (their own KafkaConsumer).
  • Each librarian works independently.
  • Each one has their own checkout desk and logbook.

That way, there’s no confusion.

In Kafka terms:🔗

If you want multiple threads to read from Kafka, do this:

One thread → One KafkaConsumer instance → Some partitions

Kafka will automatically balance partitions across consumers in the same group.

So if you have:

  • 6 partitions
  • 3 consumer threads (each with its own consumer) → Each thread gets 2 partitions.

6. How Kafka ensures each thread gets a fair share🔗

When you create multiple consumers in the same consumer group, Kafka’s group coordinator does the balancing for you.

Example:

Topic: orders
Partitions: P0, P1, P2, P3, P4, P5
Thread Consumer Assigned Partitions
Thread-1 Consumer-1 P0, P1
Thread-2 Consumer-2 P2, P3
Thread-3 Consumer-3 P4, P5

Each consumer reads only its partitions. If one thread stops, Kafka reassigns its partitions to others.

7. But what if you want multiple threads processing messages?🔗

Ah — this is the common tricky part.

Let’s say you want one thread to poll messages, but multiple worker threads to process the messages in parallel.

That’s fine — as long as only one thread is calling poll() and commit().

Here’s how you do it safely:

  1. One thread runs the consumer (polls from Kafka).
  2. It hands the fetched messages (records) to a thread pool (e.g., ExecutorService).
  3. Worker threads process those messages.
  4. Once the workers finish, the consumer thread commits offsets.

That’s thread-safe because:

  • Only one thread talks to Kafka.
  • Worker threads handle business logic separately.

8. Example flow (kid analogy)🔗

Imagine:

  • One librarian (KafkaConsumer)
  • Many book readers (worker threads)
  • The librarian checks out stacks of books to readers (polls records)
  • Readers read the books (process messages)
  • When everyone finishes, the librarian marks the books as “done” (commit offsets)

That’s how you keep order.

But if readers start trying to check out or return books directly — chaos!

9. Summary of best practices🔗

Goal Safe approach
Want to read messages faster? Use multiple consumer threads, each with its own KafkaConsumer (same group).
Want to process messages in parallel? Use one consumer thread + worker thread pool for processing.
Never do this Share one KafkaConsumer across threads.
OK to do this Share one KafkaProducer across threads (it is thread-safe).

10. Deep analogy summary🔗

Concept Analogy Rule
KafkaConsumer A librarian with a logbook Only one person (thread) should use it
Partitions Juice boxes or book piles Each consumer thread gets its own
Group rebalancing The teacher redistributing books Kafka automatically does it
Poll loop Librarian giving out new books Only librarian handles borrowing/returning
Worker threads Readers Can read (process), but don’t talk to the librarian

11. Key takeaway🔗

Kafka consumers are not thread-safe, because they keep a lot of internal state that would break if multiple threads accessed it at once.

So you must choose one of two patterns:

  1. Multi-consumer pattern:

  2. One thread per consumer.

  3. Each consumer has its own partitions.

  4. Single-consumer + worker pool pattern:

  5. One thread polls Kafka.

  6. Worker threads process data concurrently.
  7. The consumer thread alone commits offsets.

Both work well — just don’t mix them up.

In one line:🔗

Treat your KafkaConsumer like a delicate machine — only one person should operate it at a time. Others can help process the output, but no one else touches the controls.

From a single consumer group, you can have multiple consumers, each running in its own thread, but you cannot share one consumer instance across multiple threads.

Let’s restate and unpack this so it sticks.

🧱 1. Consumer group = the whole team🔗

Think of a consumer group as a team of workers (consumers) all reading from the same topic together. Each worker (consumer) gets assigned a unique subset of the topic’s partitions.

So if you have:

  • A topic with 6 partitions
  • And 3 consumers in the same group → Each one will read from 2 partitions.

🧍 2. Each consumer = one worker🔗

Each consumer instance (the KafkaConsumer object) manages:

  • Its own partitions
  • Its own offset tracking
  • Its own heartbeat with the group coordinator

That means every consumer needs to have:

  • Its own independent thread of control (poll loop)
  • Its own internal state and connection to the broker

So:

1 consumer = 1 thread = 1 partition subset

That’s the safe pattern.

🚫 3. What you cannot do🔗

You cannot have multiple threads call methods (like poll(), commitSync(), etc.) on the same KafkaConsumer instance.

Because:

  • KafkaConsumer is not thread-safe
  • Its internal state will get corrupted
  • Kafka might throw exceptions like:
IllegalStateException: Consumer is not subscribed to any topics

So this is illegal:

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);

// WRONG — two threads sharing one consumer
new Thread(() -> pollLoop(consumer)).start();
new Thread(() -> commitLoop(consumer)).start();

→ both threads talk to the same consumer instance.

Kafka does not allow that.

✅ 4. What you can do safely🔗

You can have multiple consumers (each with their own instance) running in the same consumer group, each on a different thread.

Example:

for (int i = 0; i < numConsumers; i++) {
    new Thread(new ConsumerRunnable(groupId, topics)).start();
}

Where ConsumerRunnable creates its own KafkaConsumer:

public void run() {
    KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
    consumer.subscribe(topics);
    while (true) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
        // process records
    }
}

Here:

  • Each thread has its own consumer.
  • All consumers belong to the same group.
  • Kafka automatically distributes partitions between them.

This is perfectly safe and the recommended approach.

⚙️ 5. Another safe pattern: one consumer, multiple worker threads🔗

This is also safe — and often used when you have more partitions than threads, or you want tight control.

Pattern:

  • One thread runs the KafkaConsumer (polls records)
  • It puts records into a queue
  • Worker threads pick up the messages and process them concurrently

Example:

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
ExecutorService workers = Executors.newFixedThreadPool(5);

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> record : records) {
        workers.submit(() -> processRecord(record));
    }
}

Here:

  • Only one thread touches the consumer.
  • Worker threads handle processing.
  • Safe and efficient.

🧠 6. Summary — what’s safe and what’s not🔗

Scenario Safe? Explanation
One consumer, one thread ✅ The standard pattern.
One consumer, multiple threads ❌ Not thread-safe. Causes errors or corruption.
Multiple consumers, one thread ❌ Pointless — only one can call poll(). Others are idle.
Multiple consumers, multiple threads ✅ The correct multi-threaded model for one group.
One consumer thread + worker threads for processing ✅ Only one thread polls, others process.

🔍 7. Why “multiple consumers in the same thread” doesn’t make sense🔗

Technically, you can create multiple KafkaConsumer instances in one thread, but only one can be actively polling at a time.

Kafka’s design assumes each consumer has a dedicated polling loop, so having multiple consumers in one thread gives you no benefit — it just complicates your code.

That’s why we always say:

One thread → one consumer.

✅ Final rule of thumb🔗

If you remember just one sentence, make it this:

Each KafkaConsumer must be used by exactly one thread — but a consumer group can contain many consumers (each in its own thread).

Analogy🔗

Imagine:

  • A consumer group is a team of librarians working in one big library.
  • Each consumer is a librarian.
  • Each thread is the librarian’s brain that controls what they do.

If two librarians share one brain (same thread) — one of them does nothing. If one librarian has two brains (multiple threads) — chaos. Each librarian needs exactly one brain.