Kafka ISR, High Watermark & Leader Epoch - Deep Dive Guide

Posted on In

Introduction

Apache Kafka’s reliability and consistency guarantees are built on three fundamental mechanisms: In-Sync Replicas (ISR), High Watermark, and Leader Epoch. These mechanisms work together to ensure data durability, prevent data loss, and maintain consistency across distributed partitions.

🎯 Interview Insight: Interviewers often ask “How does Kafka ensure data consistency?” This document covers the core mechanisms that make Kafka’s distributed consensus possible.

In-Sync Replicas (ISR)

Theory and Core Concepts

The ISR is a dynamic list of replicas that are “caught up” with the partition leader. A replica is considered in-sync if:

  1. It has contacted the leader within the last replica.lag.time.max.ms (default: 30 seconds)
  2. It has fetched the leader’s latest messages within this time window

graph TD
A[Leader Replica] --> B[Follower 1 - In ISR]
A --> C[Follower 2 - In ISR]
A --> D[Follower 3 - Out of ISR]

B --> E[Last Fetch: 5s ago]
C --> F[Last Fetch: 10s ago]
D --> G[Last Fetch: 45s ago - LAGGING]

style A fill:#90EE90
style B fill:#87CEEB
style C fill:#87CEEB
style D fill:#FFB6C1

ISR Management Algorithm


flowchart TD
A[Follower Fetch Request] --> B{Within lag.time.max.ms?}
B -->|Yes| C[Update ISR timestamp]
B -->|No| D[Remove from ISR]

C --> E{Caught up to leader?}
E -->|Yes| F[Keep in ISR]
E -->|No| G[Monitor lag]

D --> H[Trigger ISR shrink]
H --> I[Update ZooKeeper/Controller]
I --> J[Notify all brokers]

style A fill:#E6F3FF
style H fill:#FFE6E6
style I fill:#FFF2E6

Key Configuration Parameters

Parameter Default Description Interview Focus
replica.lag.time.max.ms 30000 Maximum time a follower can be behind How to tune for network latency
min.insync.replicas 1 Minimum ISR size for writes Consistency vs availability tradeoff
unclean.leader.election.enable false Allow out-of-sync replicas to become leader Data loss implications

🎯 Interview Insight: “What happens when ISR shrinks to 1?” Answer: With min.insync.replicas=2, producers with acks=all will get exceptions, ensuring no data loss but affecting availability.

Best Practices for ISR Management

1. Monitoring ISR Health

1
2
3
4
5
6
7
# Check ISR status
kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --topic my-topic

# Monitor ISR shrink/expand events
kafka-log-dirs.sh --bootstrap-server localhost:9092 \
  --describe --json | jq '.brokers[].logDirs[].partitions[] | select(.isr | length < 3)'

2. Tuning ISR Parameters

1
2
3
4
5
6
7
8
9
# For high-throughput, low-latency environments
replica.lag.time.max.ms=10000

# For networks with higher latency
replica.lag.time.max.ms=60000

# Ensure strong consistency
min.insync.replicas=2
unclean.leader.election.enable=false

High Watermark Mechanism

Theory and Purpose

The High Watermark (HW) represents the highest offset that has been replicated to all ISR members. It serves as the committed offset - only messages below the HW are visible to consumers.


sequenceDiagram
participant P as Producer
participant L as Leader
participant F1 as Follower 1
participant F2 as Follower 2
participant C as Consumer

P->>L: Send message (offset 100)
L->>L: Append to log (LEO: 101)

L->>F1: Replicate message
L->>F2: Replicate message

F1->>F1: Append to log (LEO: 101)
F2->>F2: Append to log (LEO: 101)

F1->>L: Fetch response (LEO: 101)
F2->>L: Fetch response (LEO: 101)

L->>L: Update HW to 101

Note over L: HW = min(LEO of all ISR members)

C->>L: Fetch request
L->>C: Return messages up to HW (100)

High Watermark Update Algorithm


flowchart TD
A[Follower Fetch Request] --> B[Update Follower LEO]
B --> C[Calculate min LEO of all ISR]
C --> D{New HW > Current HW?}
D -->|Yes| E[Update High Watermark]
D -->|No| F[Keep Current HW]
E --> G[Include HW in Response]
F --> G
G --> H[Send Fetch Response]

style E fill:#90EE90
style G fill:#87CEEB

🎯 Interview Insight: “Why can’t consumers see messages beyond HW?” Answer: Ensures read consistency - consumers only see messages guaranteed to be replicated to all ISR members, preventing phantom reads during leader failures.

High Watermark Edge Cases

Case 1: ISR Shrinkage Impact

1
2
3
4
5
6
7
8
9
Before ISR shrink:
Leader LEO: 1000, HW: 950
Follower1 LEO: 960 (in ISR)
Follower2 LEO: 950 (in ISR)

After Follower1 removed from ISR:
Leader LEO: 1000, HW: 950 (unchanged)
Follower2 LEO: 950 (only ISR member)
New HW: min(1000, 950) = 950

Case 2: Leader Election


graph TD
A[Old Leader Fails] --> B[Controller Chooses New Leader]
B --> C{New Leader LEO vs Old HW}
C -->|LEO < Old HW| D[Truncate HW to New Leader LEO]
C -->|LEO >= Old HW| E[Keep HW, Wait for Replication]

D --> F[Potential Message Loss]
E --> G[No Message Loss]

style F fill:#FFB6C1
style G fill:#90EE90

Leader Epoch

Theory and Problem It Solves

Leader Epoch was introduced to solve the data inconsistency problem during leader elections. Before leader epochs, followers could diverge from the new leader’s log, causing data loss or duplication.

🎯 Interview Insight: “What’s the difference between Kafka with and without leader epochs?” Answer: Leader epochs prevent log divergence during leader failovers by providing a monotonic counter that helps followers detect stale data.

Leader Epoch Mechanism


graph TD
A[Epoch 0: Leader A] --> B[Epoch 1: Leader B]
B --> C[Epoch 2: Leader A]
C --> D[Epoch 3: Leader C]

A1[Messages 0-100] --> A
B1[Messages 101-200] --> B
C1[Messages 201-300] --> C
D1[Messages 301+] --> D

style A fill:#FFE6E6
style B fill:#E6F3FF
style C fill:#FFE6E6
style D fill:#E6FFE6

Data Structure and Storage

Each partition maintains an epoch file with entries:

1
2
3
4
5
6
Epoch | Start Offset
------|-------------
0     | 0
1     | 101
2     | 201
3     | 301

Leader Election with Epochs


sequenceDiagram
participant C as Controller
participant L1 as Old Leader
participant L2 as New Leader
participant F as Follower

Note over L1: Becomes unavailable

C->>L2: Become leader (Epoch N+1)
L2->>L2: Increment epoch to N+1
L2->>L2: Record epoch change in log

F->>L2: Fetch request (with last known epoch N)
L2->>F: Epoch validation response

Note over F: Detects epoch change
F->>L2: Request epoch history
L2->>F: Send epoch N+1 start offset

F->>F: Truncate log if necessary
F->>L2: Resume normal fetching

Preventing Data Divergence

Scenario: Split-Brain Prevention


graph TD
A[Network Partition] --> B[Two Leaders Emerge]
B --> C[Leader A: Epoch 5]
B --> D[Leader B: Epoch 6]

E[Partition Heals] --> F[Controller Detects Conflict]
F --> G[Higher Epoch Wins]
G --> H[Leader A Steps Down]
G --> I[Leader B Remains Active]

H --> J[Followers Truncate Conflicting Data]

style C fill:#FFB6C1
style D fill:#90EE90
style J fill:#FFF2E6

🎯 Interview Insight: “How does Kafka handle split-brain scenarios?” Answer: Leader epochs ensure only one leader per epoch can be active. When network partitions heal, the leader with the higher epoch wins, and followers truncate any conflicting data.

Best Practices for Leader Epochs

1. Monitoring Epoch Changes

1
2
3
4
5
6
# Monitor frequent leader elections
kafka-log-dirs.sh --bootstrap-server localhost:9092 \
  --describe --json | jq '.brokers[].logDirs[].partitions[] | select(.leaderEpoch > 10)'

# Check epoch files
ls -la /var/lib/kafka/logs/my-topic-0/leader-epoch-checkpoint

2. Configuration for Stability

1
2
3
4
5
6
7
# Reduce unnecessary leader elections
replica.socket.timeout.ms=30000
replica.socket.receive.buffer.bytes=65536

# Controller stability
controller.socket.timeout.ms=30000
controller.message.queue.size=10

Integration and Best Practices

The Complete Flow: ISR + HW + Epochs


sequenceDiagram
participant P as Producer
participant L as Leader (Epoch N)
participant F1 as Follower 1
participant F2 as Follower 2

Note over L: ISR = [Leader, F1, F2]

P->>L: Produce (acks=all)
L->>L: Append to log (LEO: 101)

par Replication
    L->>F1: Replicate message
    L->>F2: Replicate message
end

par Acknowledgments
    F1->>L: Ack (LEO: 101)
    F2->>L: Ack (LEO: 101)
end

L->>L: Update HW = min(101, 101, 101) = 101
L->>P: Produce response (success)

Note over L,F2: All ISR members have message
Note over L: HW advanced, message visible to consumers

Production Configuration Template

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# ISR Management
replica.lag.time.max.ms=30000
min.insync.replicas=2
unclean.leader.election.enable=false

# High Watermark Optimization
replica.fetch.wait.max.ms=500
replica.fetch.min.bytes=1024

# Leader Epoch Stability
controller.socket.timeout.ms=30000
replica.socket.timeout.ms=30000

# Monitoring
jmx.port=9999

🎯 Interview Insight: “How do you ensure exactly-once delivery in Kafka?” Answer: Combine ISR with min.insync.replicas=2, acks=all, idempotent producers (enable.idempotence=true), and proper transaction management.

Advanced Scenarios and Edge Cases

Scenario 1: Cascading Failures


graph TD
A[3 Replicas in ISR] --> B[1 Replica Fails]
B --> C[ISR = 2, Still Accepting Writes]
C --> D[2nd Replica Fails]
D --> E{min.insync.replicas=2?}
E -->|Yes| F[Reject Writes - Availability Impact]
E -->|No| G[Continue with 1 Replica - Consistency Risk]

style F fill:#FFE6E6
style G fill:#FFF2E6

Scenario 2: Network Partitions


flowchart LR
subgraph "Before Partition"
    A1[Leader: Broker 1]
    B1[Follower: Broker 2]
    C1[Follower: Broker 3]
end

subgraph "During Partition"
    A2[Isolated: Broker 1]
    B2[New Leader: Broker 2]
    C2[Follower: Broker 3]
end

subgraph "After Partition Heals"
    A3[Demoted: Broker 1]
    B3[Leader: Broker 2]
    C3[Follower: Broker 3]
end

A1 --> A2
B1 --> B2
C1 --> C2

A2 --> A3
B2 --> B3
C2 --> C3

style A2 fill:#FFB6C1
style B2 fill:#90EE90
style A3 fill:#87CEEB

Troubleshooting Common Issues

Issue 1: ISR Constantly Shrinking/Expanding

Symptoms:

  • Frequent ISR change notifications
  • Performance degradation
  • Producer timeout errors

Root Causes & Solutions:


graph TD
A[ISR Instability] --> B[Network Issues]
A --> C[GC Pauses]
A --> D[Disk I/O Bottleneck]
A --> E[Configuration Issues]

B --> B1[Check network latency]
B --> B2[Increase socket timeouts]

C --> C1[Tune JVM heap]
C --> C2[Use G1/ZGC garbage collector]

D --> D1[Monitor disk utilization]
D --> D2[Use faster storage]

E --> E1[Adjust replica.lag.time.max.ms]
E --> E2[Review fetch settings]

Diagnostic Commands:

1
2
3
4
5
6
7
# Check ISR metrics
kafka-run-class.sh kafka.tools.JmxTool \
  --object-name kafka.server:type=ReplicaManager,name=IsrShrinksPerSec

# Monitor network and disk
iostat -x 1
ss -tuln | grep 9092

Issue 2: High Watermark Not Advancing

Investigation Steps:

  1. Check ISR Status:
1
2
kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --topic problematic-topic
  1. Verify Follower Lag:
1
2
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group __consumer_offsets
  1. Monitor Replica Metrics:
1
2
3
# Check replica lag
kafka-run-class.sh kafka.tools.JmxTool \
  --object-name kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=*

🎯 Interview Insight: “How would you troubleshoot slow consumer lag?” Answer: Check ISR health, monitor replica fetch metrics, verify network connectivity between brokers, and ensure followers aren’t experiencing GC pauses or disk I/O issues.

Issue 3: Frequent Leader Elections

Analysis Framework:


graph TD
A[Frequent Leader Elections] --> B{Check Controller Logs}
B --> C[ZooKeeper Session Timeouts]
B --> D[Broker Failures]
B --> E[Network Partitions]

C --> C1[Tune zookeeper.session.timeout.ms]
D --> D1[Investigate broker health]
E --> E1[Check network stability]

D1 --> D2[GC tuning]
D1 --> D3[Resource monitoring]
D1 --> D4[Hardware issues]

Performance Tuning

ISR Performance Optimization

1
2
3
4
5
6
7
8
9
# Reduce ISR churn
replica.lag.time.max.ms=30000  # Increase if network is slow
replica.socket.timeout.ms=30000
replica.socket.receive.buffer.bytes=65536

# Optimize fetch behavior
replica.fetch.wait.max.ms=500
replica.fetch.min.bytes=1024
replica.fetch.max.bytes=1048576

High Watermark Optimization

1
2
3
4
5
6
# Faster HW advancement
replica.fetch.backoff.ms=1000
replica.high.watermark.checkpoint.interval.ms=5000

# Batch processing
replica.fetch.response.max.bytes=10485760

Monitoring and Alerting

Key Metrics to Monitor:

Metric Threshold Action
ISR Shrink Rate > 1/hour Investigate network/GC
Under Replicated Partitions > 0 Check broker health
Leader Election Rate > 1/hour Check controller stability
Replica Lag > 10000 messages Scale or optimize

JMX Monitoring Script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
#!/bin/bash
# Key Kafka ISR/HW metrics monitoring

# ISR shrinks per second
echo "ISR Shrinks:"
kafka-run-class.sh kafka.tools.JmxTool \
  --object-name kafka.server:type=ReplicaManager,name=IsrShrinksPerSec \
  --one-time

# Under-replicated partitions
echo "Under-replicated Partitions:"
kafka-run-class.sh kafka.tools.JmxTool \
  --object-name kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions \
  --one-time

# Leader election rate
echo "Leader Elections:"
kafka-run-class.sh kafka.tools.JmxTool \
  --object-name kafka.controller:type=ControllerStats,name=LeaderElectionRateAndTimeMs \
  --one-time

🎯 Final Interview Insight: “What’s the relationship between ISR, HW, and Leader Epochs?” Answer: They form Kafka’s consistency triangle - ISR ensures adequate replication, HW provides read consistency, and Leader Epochs prevent split-brain scenarios. Together, they enable Kafka’s strong durability guarantees while maintaining high availability.

This guide provides a comprehensive understanding of Kafka’s core consistency mechanisms. Use it as a reference for both system design and troubleshooting scenarios.