Kafka brokers are rejecting client connections due to failed SASL authentication, preventing any client from producing or consuming messages.
Common Causes and Fixes for Kafka SASL Authentication Failures
1. Incorrect sasl.enabled Configuration
- Diagnosis: Check
server.propertieson your Kafka brokers. Ensuresasl.enabledis set totrue. If it’sfalse, SASL is disabled, and clients configured for SASL will fail. - Fix: Set
sasl.enabled=trueinserver.propertieson all brokers. - Why it works: This explicitly tells the Kafka broker to expect and process SASL authentication mechanisms for incoming connections.
2. Mismatched sasl.mechanism
- Diagnosis: Verify the
sasl.mechanismsetting in both your client configuration (e.g.,producer.properties,consumer.properties) and your broker configuration (server.properties). Common mechanisms arePLAIN,GSSAPI(Kerberos), andSCRAM-SHA-256/SCRAM-SHA-512. - Fix: Ensure the
sasl.mechanismvalue is identical on both the client and broker. For example, if your client hassasl.mechanism=PLAIN, your broker must also havesasl.mechanism=PLAIN(or be configured to support it viasasl.mechanism.list). - Why it works: The client and broker must agree on the authentication protocol to use. A mismatch means they can’t understand each other’s authentication handshake.
3. Incorrect sasl.jaas.config on Brokers
- Diagnosis: Examine the
sasl.jaas.configproperty inserver.propertieson your Kafka brokers. This file specifies the JAAS (Java Authentication and Authorization Service) configuration for Kafka. Common issues include typos, incorrect file paths for keytabs/principals, or wrong realm names. - Fix: For Kerberos (GSSAPI), ensure the
sasl.jaas.configpoints to the correctkeytabfile andprincipal. For example:
For PLAIN or SCRAM, ensure thesasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \ useKeyTab=true \ storeKey=true \ keyTab="/etc/kafka/kafka.service.keytab" \ principal="kafka/kafka-broker.example.com@EXAMPLE.COM";login.controller.factoryand relevantusersconfigurations are correct if you’re using Kafka’s internal user store. - Why it works: JAAS is the underlying framework Kafka uses for authentication. Correctly configured JAAS entries tell Kafka how to find and validate credentials (like Kerberos tickets or username/passwords).
4. Incorrect sasl.jaas.config on Clients
- Diagnosis: Inspect the
sasl.jaas.configin your client properties files. This is particularly relevant for Kerberos. Ensure the client’s JAAS configuration correctly identifies its principal and keytab, and that the Kerberos realm (.realm) and KDC (.kdc) are correctly specified if not using a defaultkrb5.conf. - Fix: For Kerberos clients, a typical
sasl.jaas.configlooks like this:
If using PLAIN or SCRAM, this might contain username/password information, but it’s generally better to use environment variables or external credential providers for security.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \ useKeyTab=true \ storeKey=true \ keyTab="/path/to/client.keytab" \ principal="client-principal@EXAMPLE.COM"; - Why it works: Clients need their own valid credentials and configuration to authenticate to the brokers. The client’s JAAS config dictates how it obtains and presents its identity.
5. Time Skew (Kerberos Specific)
- Diagnosis: If using Kerberos (
GSSAPI), significant time differences between the Kafka brokers, the client machines, and the Kerberos Key Distribution Center (KDC) can cause authentication failures. Check the system time on all relevant machines. - Fix: Ensure all servers (brokers, clients, KDC) are synchronized using Network Time Protocol (NTP). The maximum allowable skew is typically 5 minutes.
- Why it works: Kerberos tickets have a limited lifespan and are time-sensitive. If the clocks are too far apart, the ticket presented by the client will appear expired or invalid to the KDC or broker.
6. Missing or Incorrect Principal Mapping
- Diagnosis: For mechanisms like GSSAPI (Kerberos) or SCRAM, the broker needs to know how to map the authenticated principal (e.g.,
kafka/broker.host@REALM) to a Kafka user. This is often configured viaprincipal.mapper.rulesinserver.properties. Check if the rule correctly extracts the username from the authenticated principal. - Fix: Update
principal.mapper.rulesto correctly map the incoming principal to a Kafka user. For example, to map a Kerberos principaluser/host@REALMto a Kafka useruser:
Ensure the Kafka user (principal.mapper.rules=RULE:^User:([a-zA-Z0-9._/-]+)@.*$/$1/$1in this example) exists in your authorization configuration if authorization is also enabled. - Why it works: Authentication confirms who the client is, but mapping translates that identity into a Kafka-internal user that Kafka’s authorization layer can understand and use.
7. Incorrect File Permissions for Keytabs
- Diagnosis: For Kerberos, if the
keyTabfile specified insasl.jaas.configis not readable by the Kafka process user (e.g.,kafka), authentication will fail. - Fix: Ensure the Kafka user has read permissions on the keytab file. For example, if your keytab is at
/etc/kafka/kafka.service.keytaband Kafka runs as userkafkaand groupkafka:sudo chown kafka:kafka /etc/kafka/kafka.service.keytab sudo chmod 600 /etc/kafka/kafka.service.keytab - Why it works: The Kafka process needs to access the keytab file to load its Kerberos credentials and authenticate itself as a service principal to the KDC.
The next error you’ll likely encounter, after fixing authentication, is an authorization error if your ACLs are not correctly configured.