Fix Kafka RecordTooLargeException Errors (2026)

The Kafka broker rejected a message because its size exceeded the configured maximum for a topic or for the broker itself.

Common Causes and Fixes

1. Producer max_request_size too small

Diagnosis: Check your producer configuration. If you’re using librdkafka, it’s max.request.size. If you’re using the Java client, it’s max.request.size.
Fix: Increase max.request.size on the producer to at least the size of the largest message you intend to send. For example, in librdkafka configuration:
```
max.request.size=10485760 # 10MB
```
This allows the producer to buffer and send larger individual requests to the broker.
Why it works: The producer won’t attempt to send a request larger than this value.

2. Broker message.max.bytes too small

Diagnosis: This is a per-broker setting. Check server.properties on your Kafka brokers.
```
grep message.max.bytes /path/to/kafka/config/server.properties
```
Fix: Increase message.max.bytes on all brokers to accommodate your largest messages. A common value is 10485760 (10MB).
```
message.max.bytes=10485760
```
Remember to restart your Kafka brokers after changing this setting.
Why it works: This is the maximum size of a single message that a broker will accept.

3. Broker replica.fetch.max.bytes too small

Diagnosis: This setting controls how much data a replica broker can fetch from a leader in a single request. It’s also in server.properties.
```
grep replica.fetch.max.bytes /path/to/kafka/config/server.properties
```
Fix: Increase replica.fetch.max.bytes to be at least as large as message.max.bytes. A good practice is to set it equal to or slightly larger than message.max.bytes.
```
replica.fetch.max.bytes=10485760
```
Restart your Kafka brokers.
Why it works: If a replica tries to fetch a message batch that exceeds this limit, it can also lead to RecordTooLargeException indirectly during replication.

4. Topic max.message.bytes too small

Diagnosis: This is a topic-level configuration. You can check it using the Kafka command-line tools.
```
kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic your_topic_name
```
Look for maxMessageBytes in the output.

Fix: Alter the topic configuration to increase max.message.bytes.

kafka-configs.sh --bootstrap-server localhost:9092 --alter --entity-type topics --entity-name your_topic_name --add-config max.message.bytes=10485760

This setting is often set to match message.max.bytes on the brokers.

Why it works: This is the maximum size of a single message allowed for this specific topic, overriding the broker’s message.max.bytes if it’s smaller.

5. Producer batch.size and linger.ms interaction

Diagnosis: While max.request.size dictates the maximum size of a single request, the batch.size (producer config) determines when a batch of records is sent, and linger.ms determines how long the producer waits for more records before sending. If batch.size is very large and linger.ms is short, you might try to send a batch that, even if it contains fewer records than max.request.size, ends up exceeding message.max.bytes after serialization and overhead. This is less common for RecordTooLargeException itself, but can contribute to larger messages.
Fix: Ensure your batch.size is reasonable and that linger.ms is not set to 0 if you want to benefit from batching without hitting size limits prematurely. If you are indeed sending very large individual messages, these settings are less relevant than max.request.size and the broker/topic limits.
Why it works: Proper batching can reduce overhead, but if individual messages are inherently large, this is secondary to the fundamental size limits.

6. Large compressed messages

Diagnosis: If you are using compression on the producer (compression.type), the compressed message size is what Kafka sees. However, the uncompressed size of the messages within a batch can still be a factor. The message.max.bytes and topic max.message.bytes settings apply to the serialized size, which includes compression overhead. The issue arises if the uncompressed size of individual messages, when batched, exceeds what the broker can handle, even after compression.
Fix: While you generally want to ensure message.max.bytes and topic max.message.bytes are large enough, consider if your compression strategy is optimal. If you’re compressing very small messages, overhead can increase. If you’re sending extremely large messages, compression might not reduce them enough to fit within limits. Sometimes, a different compression codec (e.g., lz4 for speed vs. snappy or gzip for higher compression) or disabling compression for exceptionally large messages (if feasible) can help.
Why it works: Kafka brokers enforce size limits on the serialized message payload. Compression reduces this payload size, but the underlying uncompressed data still contributes to the total.

7. Message serialization overhead

Diagnosis: The reported size of a message includes not just the raw data, but also Kafka’s internal framing, headers, and any metadata. This overhead can add a few hundred bytes per message.
Fix: Account for this overhead when setting your max.request.size, message.max.bytes, and topic max.message.bytes. If your messages are consistently just under a limit (e.g., 990KB and your limit is 1MB), adding serialization overhead can push them over. Increase the limits by a small buffer (e.g., to 1048576 or 1050000).
Why it works: Kafka’s protocol and internal structures add bytes to the raw payload that must be accounted for within the size limits.

After fixing these, you might encounter ProducerFencedException if your consumer group has been reassigned and an older producer is still trying to commit offsets.