Fix Kafka ReplicaNotAvailableException Errors (2026)

The Kafka broker failed to get a leader for a partition, meaning it couldn’t serve requests for that partition because no replica was available to act as the leader.

Cause 1: ZooKeeper Connection Issues

Diagnosis: Check Kafka broker logs for ZooKeeperConnectionException or NoNodeException errors. Also, check ZooKeeper server logs for client connection errors or errors related to session expiration. On the Kafka broker, run echo stat | nc <zookeeper_host> <zookeeper_port> and look for stat output showing the number of connected clients. If it’s less than expected, or if you see avg_latency or max_latency that are too high, ZooKeeper might be struggling.

Fix:

1. Restart Kafka Brokers: Often, a simple restart of the affected Kafka brokers resolves transient ZooKeeper connection issues.
2. Check ZooKeeper Ensemble Health: Ensure all ZooKeeper servers in the ensemble are running and healthy. If ZooKeeper is not in a quorum, Kafka will lose its cluster state.
3. Increase ZooKeeper Session Timeout: If ZooKeeper is under heavy load, Kafka brokers might be timing out. In zoo.cfg on ZooKeeper servers, increase tickTime (e.g., from 2000 to 6000 ms) and initLimit/syncLimit accordingly. Then restart ZooKeeper and Kafka brokers.
4. Network Connectivity: Verify network connectivity and firewall rules between Kafka brokers and ZooKeeper servers. Ensure the ports are open.

Why it works: Kafka relies heavily on ZooKeeper for cluster coordination, including leader election and partition metadata. If a broker cannot maintain a stable connection to ZooKeeper, it loses track of the cluster state, including which replicas are available to be leaders, leading to ReplicaNotAvailableException.

Cause 2: Under-Replicated Partitions Due to Broker Failures

Diagnosis: On a Kafka broker, run kafka-topics.sh --bootstrap-server <broker_host>:9092 --describe --topic <topic_name>. Look for partitions where the "Leader" is -1 and the "Replicas" list contains brokers that are currently down or unresponsive. You can also use kafka-min-ISR-shrunk-to-zero-check (a custom script or logic you might have) or monitor metrics like UnderReplicatedPartitions.

Fix:

1. Identify and Restart Failed Brokers: Determine which brokers are down and attempt to restart them. Once they rejoin the cluster, Kafka’s controller will attempt to re-elect leaders for partitions that were previously unavailable.
2. Increase min.insync.replicas (if applicable and appropriate): If min.insync.replicas is set higher than the number of available replicas for a partition, no leader can be elected. Lowering this value temporarily (e.g., from 3 to 2) can allow a leader election, but understand the implications for data durability. Caution: This should be a temporary measure. The ideal fix is to bring brokers back online.
3. Manual Leader Election (as a last resort): If brokers are permanently gone and cannot be replaced quickly, you might need to manually reassign partitions using kafka-reassign-partitions.sh. This forces Kafka to pick a new leader from the available replicas.

Why it works: When a broker hosting the leader replica for a partition goes down, Kafka’s controller initiates a leader election among the remaining in-sync replicas (ISRs). If no ISRs are available or if min.insync.replicas cannot be satisfied, the partition becomes unavailable. Restarting brokers or reassigning partitions ensures a healthy replica can assume the leader role.

Cause 3: Network Partitioning Between Brokers

Diagnosis: Check broker logs for repeated connection attempts to other brokers that fail, or messages indicating a loss of quorum. Network diagnostics like ping, traceroute, and tcpdump between affected brokers can reveal packet loss, high latency, or blocked ports. Look for metrics like NetworkProcessorAvgIdlePercent on brokers; consistently low values might indicate network saturation.

Fix:

1. Resolve Network Issues: Identify and fix the underlying network problems: faulty cables, misconfigured switches, firewall rules blocking inter-broker traffic (ports 9092, 2181, etc.), or IP address conflicts.
2. Adjust Broker replica.lag.time.max.ms: If network latency is high but stable, increasing replica.lag.time.max.ms (e.g., from 10000 to 30000 ms) can give replicas more time to catch up before they are considered out of sync, potentially preventing leader loss if the network is temporarily slow.
3. Rebalance Partitions: After network stability is restored, use kafka-reassign-partitions.sh to rebalance partitions to ensure replicas are distributed across healthy network segments.

Why it works: Network partitions can cause brokers to appear unavailable to each other, even if they are running. This prevents the controller from detecting a healthy replica or achieving quorum for leader election, especially if the partition is spread across the network boundary.

Cause 4: Controller Overload or Failure

Diagnosis: Monitor Kafka broker logs for messages related to the controller, such as "Controller became leader," "Controller failed to send metadata update," or "Epoch mismatch." High CPU or memory usage on the broker designated as the controller can be an indicator. Kafka metrics like ControllerQueueSize and ControllerNumActiveTopics can also signal overload.

Fix:

1. Restart the Controller Broker: If a specific broker is acting as the controller and is overloaded, restarting it can allow a new controller to be elected.
2. Scale ZooKeeper: A struggling ZooKeeper ensemble can bottleneck the controller, as it relies on ZooKeeper for state management. Ensure ZooKeeper is adequately provisioned and healthy.
3. Tune Controller Settings: While less common, parameters like controller.listener.names and controller.quorum.voters (in newer Kafka versions) can be reviewed if controller issues are persistent.

Why it works: The Kafka controller is responsible for managing cluster state, including partition leadership and replica synchronization. If the controller is overloaded or fails, it cannot perform these tasks, leading to partitions becoming unavailable.

Cause 5: Insufficient Disk Space or I/O Bottlenecks

Diagnosis: Check Kafka broker logs for IOException: No space left on device or messages indicating slow disk writes. Monitor disk usage on the brokers using df -h and disk I/O performance using tools like iostat. High RequestQueueTimeMs or LocalLogAppendTimeMs metrics can point to I/O issues.

Fix:

1. Free Up Disk Space: Delete old logs, move data to different disks, or add more storage to the affected brokers.
2. Upgrade Disk Hardware: If I/O is the bottleneck, consider migrating to faster SSDs or improving the storage subsystem.
3. Tune message.log.segment.bytes and log.retention.hours: Adjusting these parameters can influence how quickly log segments are rolled and deleted, potentially freeing up space faster.

Why it works: Kafka brokers need free disk space and sufficient I/O throughput to write messages to disk and serve replica requests. If these resources are exhausted, brokers cannot function correctly, leading to partitions becoming unavailable.

Cause 6: Incorrect unclean.leader.election.enable Setting

Diagnosis: While not a direct cause of the error, if unclean.leader.election.enable=true is set and a broker with stale data becomes leader, it can mask underlying issues. This setting is configured per topic or broker. Check server.properties or topic configurations.

Fix: Set unclean.leader.election.enable=false (recommended for most production use cases). If this was enabled and caused an issue, you might need to reassign partitions to ensure you have a leader with the most up-to-date data after recovering from outages.

Why it works: When unclean.leader.election.enable is true, a leader can be elected even if it’s not in the in-sync replica set (ISR). This can lead to data loss and subsequent availability issues if that stale leader fails. Setting it to false enforces that only ISRs can become leaders, ensuring data consistency but requiring more replicas to be available.

The next error you’ll likely encounter if you fix ReplicaNotAvailableException without addressing underlying issues is a LeaderNotAvailableException or a flood of NotEnoughReplicasAfterAppendException if you’ve enabled idempotence or transactional producers.