Skip to content

Replication In Kafka🔗

1. Replication as the Foundation of Reliability🔗

Kafka’s reliability — particularly its durability and fault tolerance — depends primarily on its replication mechanism.

  • Each Kafka topic is divided into partitions, which are the smallest unit of data storage and parallelism.
  • Each partition can have multiple replicas stored across different brokers (physical servers in the Kafka cluster).

Replication ensures that even if one broker crashes, another broker can take over with no data loss. This mechanism is what gives Kafka its strong guarantees about message survival and recoverability.

2. Partition Structure and Leadership🔗

a) Partition Basics🔗

A partition:

  • Is an ordered, append-only log of messages.
  • Is stored on a single disk of a broker.
  • Preserves the order of messages within itself.
  • Can be either online (available) or offline (unavailable).

Kafka ensures that producers and consumers always interact with the leader replica of a partition.

b) Leader and Follower Replicas🔗

Each partition has:

  • One leader replica
  • Zero or more follower replicas

The leader handles:

  • All produce requests (writing messages)
  • All consume requests (reading messages)

Followers:

  • Do not serve client requests.
  • Simply replicate data from the leader continuously to stay up to date.

This replication model is known as leader-based replication.

3. What Happens During Normal Operation🔗

  1. The producer sends data to the leader replica of the target partition.
  2. The leader appends the data to its local log.
  3. The followers fetch this data from the leader and append it to their logs.
  4. Once all in-sync replicas confirm they have the data, the message is considered committed.
  5. Consumers are allowed to read only committed messages.

This process ensures that messages are safely stored in multiple locations before they are visible to consumers, protecting against single-broker failures.

4. Handling Failures: Leader Election🔗

If the leader of a partition fails (for example, due to a broker crash), Kafka automatically promotes one of the in-sync replicas (ISRs) to become the new leader.

The in-sync replicas are those that are confirmed to have fully replicated the leader’s data up to a recent point in time. This guarantees that the new leader will have all committed messages, preserving data integrity.

Kafka’s controller (a special broker role) handles this leader election process automatically.

5. How Kafka Defines “In-Sync Replica” (ISR)🔗

A replica is part of the ISR set (In-Sync Replica set) if it satisfies certain conditions. The ISR always includes the current leader and all followers that are up to date.

A follower is considered in sync if:

  1. Active ZooKeeper session

  2. The broker hosting the follower must have an active session with ZooKeeper (or the Kafka controller in newer versions).

  3. This means it has sent a heartbeat within a certain time limit (default 6 seconds, configurable).

  4. If the follower fails to send heartbeats, Kafka assumes it’s down and removes it from the ISR.

  5. Recent Fetch Activity

  6. The follower must have fetched data from the leader within a specific time window (default 10 seconds).

  7. If it hasn’t fetched new data recently, it’s marked as out of sync.

  8. No Lag for a Recent Period

  9. The follower must have been fully caught up with the leader at least once in the last 10 seconds (configurable).

  10. It’s not enough to merely fetch data slowly; the follower must prove it can keep up with the leader in real time at least occasionally.

If a follower falls behind (due to network delay, slow disk, or high load), Kafka removes it from the ISR. Only ISRs are eligible for leadership if the current leader fails.

6. Why These Rules Matter🔗

These conditions guarantee that:

  • Only up-to-date replicas can take over as leader.
  • Kafka never promotes a follower that is missing committed data.
  • Data consistency is maintained even during broker crashes.

This mechanism prevents data loss and inconsistent reads, which are critical for reliability.

However, it also means that if too many followers are slow or disconnected, Kafka might have fewer in-sync replicas, which can affect availability. This is one of Kafka’s core reliability–availability trade-offs.

7. Example Walkthrough🔗

Imagine a topic orders with:

  • Replication factor = 3
  • Partition P1 stored on brokers B1 (leader), B2 (follower), and B3 (follower)

During normal operation:

  1. The producer writes to B1 (leader).
  2. B2 and B3 fetch and replicate data from B1.
  3. All three are in the ISR.
  4. A message is considered committed when all three confirm replication.

If B1 (the leader) fails:

  • Kafka elects either B2 or B3 (whichever is in the ISR) as the new leader.
  • Producers and consumers are redirected to the new leader automatically.
  • Data remains consistent — no committed message is lost.

8. Configurable Parameters (Defaults May Vary)🔗

Parameter Description Default
zookeeper.session.timeout.ms Time for a broker to send a heartbeat to ZooKeeper 6000 ms
replica.lag.time.max.ms How long a follower can go without fetching data before being removed from ISR 10000 ms
replica.lag.max.messages Max number of messages a follower can lag before removal (deprecated in newer versions) N/A
min.insync.replicas Minimum number of in-sync replicas required for a message to be committed 1 or 2 (recommended 2 for safety)

These settings allow administrators to fine-tune how strict Kafka should be about replication and consistency.

9. Summary🔗

Concept Description
Replication Each partition is copied across multiple brokers for durability
Leader Replica Handles all read/write requests for a partition
Follower Replica Replicates data from the leader
ISR (In-Sync Replica) Set of replicas that are fully caught up and eligible to become leader
Leader Election When a leader fails, an ISR replica is promoted automatically
Durability Messages are safe as long as at least one ISR remains alive

In essence:🔗

Kafka’s replication mechanism is the backbone of its reliability model. It ensures that:

  • Messages are not lost even if brokers fail.
  • Only up-to-date replicas can become leaders.
  • Order and durability are preserved.

This design — leader-based replication with in-sync follower tracking — is what allows Kafka to maintain both high availability and data integrity in distributed, failure-prone environments.

If a replica loses connection to ZooKeeper, stops fetching new messages, or falls behind and can’t catch up within 10 seconds, the replica is considered out of sync. An out-ofsync replica gets back into sync when it connects to ZooKeeper again and catches up to the most recent message written to the leader.

This usually happens quickly after a temporary network glitch is healed but can take a while if the broker the replica is stored on was down for a longer period of time.

An in-sync replica that is slightly behind can slow down producers and consumers—since they wait for all the insync replicas to get the message before it is committed. Once a replica falls out of sync, we no longer wait for it to get messages.

It is still behind, but now there is no performance impact. The catch is that with fewer in-sync replicas, the effective replication factor of the partition is lower, and therefore there is a higher risk for downtime or data loss.