The Kafka broker is rejecting connection attempts from your consumer because it believes the consumer is not authorized to access any topics, and this is causing the consumer to fail its subscription setup.
Common Causes and Fixes:
-
Incorrect
bootstrap.serversConfiguration: The consumer is connecting to the wrong Kafka cluster, or a broker that’s not properly configured to serve metadata.- Diagnosis: Check your consumer’s configuration file (e.g.,
consumer.propertiesor your application’s config) for thebootstrap.serverssetting. Ping the listed brokers from where your consumer is running. - Fix: Ensure
bootstrap.serverspoints to a comma-separated list of reachable Kafka brokers in your target cluster. For example:bootstrap.servers=kafka-broker-1.example.com:9092,kafka-broker-2.example.com:9092. - Why it works: This directly tells the consumer where to find the Kafka cluster’s metadata and other brokers.
- Diagnosis: Check your consumer’s configuration file (e.g.,
-
Missing or Incorrect Topic Subscription: The consumer code explicitly isn’t telling Kafka which topics it wants to subscribe to, or it’s trying to subscribe to non-existent topics.
- Diagnosis: In your consumer application code, locate the
consumer.subscribe()method call. Verify the topic names passed to it. - Fix: Ensure the
subscribe()method is called with the correct topic names. For example, in Java:consumer.subscribe(Arrays.asList("my-topic", "another-topic"));. If subscribing via configuration, checktopicsortopic.listin your consumer properties. - Why it works: This is the fundamental instruction to Kafka, telling it which data streams the consumer is interested in processing.
- Diagnosis: In your consumer application code, locate the
-
ZooKeeper Connectivity Issues (for older Kafka versions or specific configurations): If your Kafka cluster relies on ZooKeeper for metadata and coordination, and the consumer’s broker cannot reach ZooKeeper, it can lead to stale or inaccessible topic lists.
- Diagnosis: From a Kafka broker’s command line, check ZooKeeper connectivity:
zookeeper-shell <zookeeper-host>:2181 ls /brokers/ids. If this fails or returns errors, ZooKeeper is the issue. - Fix: Ensure your Kafka brokers have correct
zookeeper.connectsettings in theirserver.propertiesand that network firewalls allow access. Restart the affected Kafka brokers. - Why it works: ZooKeeper is the source of truth for Kafka’s internal state, including which topics exist and which brokers are active.
- Diagnosis: From a Kafka broker’s command line, check ZooKeeper connectivity:
-
ACLs Denying Topic Access: Access Control Lists (ACLs) are configured on the Kafka broker, and the consumer’s principal (user/identity) is not granted
READorDESCRIBEpermissions on the subscribed topics.- Diagnosis: On a Kafka broker, use the Kafka security tools to list ACLs for your consumer’s principal and the target topics. For example:
kafka-acls.sh --bootstrap-server <broker-host>:9092 --list --principal User:my-consumer-user --topic my-topic. - Fix: Add the necessary ACLs. For instance, to grant a user read access to a topic:
kafka-acls.sh --bootstrap-server <broker-host>:9092 --add --allow-principal User:my-consumer-user --operations READ --topic my-topic. - Why it works: Kafka enforces security policies; without explicit permission, it will prevent the consumer from even seeing or reading from a topic.
- Diagnosis: On a Kafka broker, use the Kafka security tools to list ACLs for your consumer’s principal and the target topics. For example:
-
Topic Does Not Exist: The topic name provided in the
subscribe()call or configuration simply doesn’t exist in the Kafka cluster.- Diagnosis: Use the Kafka command-line tool to list all topics on the broker:
kafka-topics.sh --bootstrap-server <broker-host>:9092 --list. - Fix: Create the topic if it’s supposed to exist:
kafka-topics.sh --bootstrap-server <broker-host>:9092 --create --topic my-topic --partitions 3 --replication-factor 1. Alternatively, correct the topic name in your consumer’ssubscribe()call or configuration. - Why it works: You can’t subscribe to something that isn’t there. This ensures the topic is available for subscription.
- Diagnosis: Use the Kafka command-line tool to list all topics on the broker:
-
Broker Not Advertising Topics Correctly (Less Common): In complex multi-broker setups, a broker might not be properly advertising its available topics due to configuration issues or network partitioning.
- Diagnosis: Inspect the broker logs for errors related to topic metadata, partition leader election, or inter-broker communication. Check
advertised.listenersandlistenersinserver.properties. - Fix: Ensure
advertised.listenersis set to the correct network interface and port that clients can reach. Restart the affected broker. - Why it works: The consumer relies on brokers to provide an accurate, up-to-date list of available topics. If a broker’s metadata is faulty, it can mislead consumers.
- Diagnosis: Inspect the broker logs for errors related to topic metadata, partition leader election, or inter-broker communication. Check
After fixing these issues, you might encounter FETCHING_COORDINATOR_NOT_AVAILABLE if your consumer group is trying to join a partition that’s currently undergoing leader election.