How to Send Large Messages (Over 15MB) Using Kafka in Java?

What steps can I take to successfully send large messages (over 15MB) with Kafka using the Java Producer API?

private ProducerConfig kafkaConfig() {
    Properties props = new Properties();
    props.put("metadata.broker.list", BROKERS);
    props.put("serializer.class", "kafka.serializer.StringEncoder");
    props.put("request.required.acks", "1");
    props.put("message.max.bytes", "" + 1024 * 1024 * 40);
    return new ProducerConfig(props);
}

To send large messages using Kafka, particularly with versions earlier than 2.0, you need to adjust both the broker and producer settings to handle larger payloads. The default maximum message size is 1MB, which can lead to a MessageSizeTooLargeException when attempting to send larger messages. Follow these steps to configure your setup appropriately.

// Example of modified ProducerConfig
private ProducerConfig kafkaConfig() {
    Properties props = new Properties();
    props.put("metadata.broker.list", BROKERS);
    props.put("serializer.class", "kafka.serializer.StringEncoder");
    props.put("request.required.acks", "1");
    props.put("max.request.size", "" + 1024 * 1024 * 40); // for producer
    return new ProducerConfig(props);
}

Causes

• The default configuration for maximum message size in Kafka brokers is usually set to 1MB.
• Your producer settings may not be correctly configured to match the broker settings for maximum message size.
• Kafka version differences may have affecting configurations that handle larger messages.

Solutions

• Increase the 'max.message.bytes' configuration on the Kafka broker to accommodate larger payloads. This can be done in the server.properties file: 'max.message.bytes=40000000' (for 40MB).
• Ensure that the producer configuration also has an appropriate limit set by modifying the ProducerConfig settings in your Java code: 'props.put("max.request.size", "40000000");' to match the broker settings.
• Check for any other limits related to buffer sizes in your Kafka setup that might also need to be adjusted.

Mistake: Not updating both broker and producer configuration to the same message size limit.

Solution: Ensure that you adjust both server.properties for the broker and the producer properties in your Java code to have matching values.

Mistake: Overlooking other configuration settings related to batch sizes and buffer sizes.

Solution: Check the Kafka documentation for additional configurations like 'buffer.memory' and 'batch.size' that may also need adjustments.