Fix Kafka Duplicate Sequence Number Errors (2026)

The Kafka producer is failing to commit messages because it’s detecting a duplicate sequence number, indicating that a message has already been acknowledged by the broker.

• Producer enable.idempotence not set to true: This is the most common oversight. Idempotence is the feature that prevents duplicates by tracking sequence numbers, and it’s off by default.

  • Diagnosis: Check your producer configuration file or the ProducerConfig object in your code. Look for enable.idempotence.
  • Fix: Set enable.idempotence=true in your producer configuration.
  • Why it works: When enable.idempotence is true, the producer assigns a unique sequence number to each message within a partition for a given producer session. The broker uses this sequence number to detect and discard duplicate messages that arrive due to retries.

• Producer acks setting is too lenient: If acks is set to 0 or 1, the producer might not wait for sufficient confirmation from the broker, leading to retries and potential duplicate detection.

  • Diagnosis: Examine the acks setting in your producer configuration.
  • Fix: Set acks=all (or -1) in your producer configuration.
  • Why it works: acks=all ensures that the producer waits for all in-sync replicas (ISRs) to acknowledge the message. This significantly reduces the chance of the producer thinking a message was lost when it was merely in transit or not yet replicated widely enough, thus preventing premature retries that trigger the duplicate sequence number error.

• max.in.flight.requests.per.connection greater than 1 with enable.idempotence=true: This is a tricky configuration interaction. If idempotence is enabled, the producer must guarantee that requests are sent in order within a connection to prevent duplicates.

  • Diagnosis: Check both enable.idempotence and max.in.flight.requests.per.connection in your producer configuration.
  • Fix: Set max.in.flight.requests.per.connection=1 when enable.idempotence=true.
  • Why it works: When enable.idempotence=true, the producer must ensure that messages are delivered exactly once. If max.in.flight.requests.per.connection is greater than 1, multiple requests can be in flight simultaneously. If a request fails and is retried, but an earlier request also succeeded and was just delayed, the retried request might be processed out of order, leading to a duplicate sequence number error if the broker expects sequential delivery. Setting it to 1 enforces strict ordering.

• Broker message.downconversion.enable set to false with older clients: While not directly a producer setting, this broker configuration can cause issues if clients are not sending messages in the expected format and idempotence is involved.

  • Diagnosis: Check the message.downconversion.enable setting on your Kafka brokers.
  • Fix: Set message.downconversion.enable=true on your brokers, or ensure all clients are using a Kafka protocol version that supports the message format the broker expects.
  • Why it works: If this is false, brokers will reject messages if their format doesn’t match the broker’s internal representation. Older clients might use a format that requires downconversion on the broker. If idempotence is enabled, the broker’s strict rejection of a message due to format mismatch, even if it’s not a duplicate content-wise, can be interpreted by the producer as a failure, leading to retries and the duplicate sequence number error.

• Network instability causing delayed ACKs: Intermittent network issues can cause acknowledgments from the broker to be delayed, making the producer believe a message hasn’t been delivered yet and triggering a retry.

  • Diagnosis: Monitor network latency and packet loss between your producers and Kafka brokers. Check producer logs for repeated "timeouts" or "connection refused" errors preceding the duplicate sequence number error.
  • Fix: Improve network stability between producers and brokers. Consider increasing delivery.timeout.ms in the producer configuration to 120000 (2 minutes) or higher if network latency is consistently high but acceptable.
  • Why it works: A higher delivery.timeout.ms gives the producer more time to receive an acknowledgment from the broker before giving up and retrying, thus tolerating temporary network glitches without falsely concluding a message is lost and needs to be resent.

• Producer request.timeout.ms too low: Similar to network issues, if the producer’s request timeout is too short, it might give up on a request that is still in progress on the broker, leading to retries and the duplicate sequence error.

  • Diagnosis: Check the request.timeout.ms setting in your producer configuration.
  • Fix: Increase request.timeout.ms to a value like 60000 (1 minute).
  • Why it works: This setting dictates how long the producer will wait for a response to a specific request (like producing a message) before considering it a failure. Increasing it allows more time for the broker to process the request and send back an acknowledgment, especially under load or with higher network latency.

The next error you’ll likely encounter, if you’ve only fixed the duplicate sequence number issue, is Out of order sequence number errors, which indicate that while idempotence is enabled, messages are arriving at the broker in a different order than they were sent.