From the course: Complete Guide to Apache Kafka for Beginners

Unlock the full course today

Join today to access over 23,200 courses taught by industry experts.

Large messages in Kafka

Large messages in Kafka - Kafka Tutorial

From the course: Complete Guide to Apache Kafka for Beginners

Start my 1-month free trial Buy for my team

Large messages in Kafka

“

Hi, this is Stephane Marek from Conduktor and in this lecture, we're going to have a look at sending large messages in Apache Kafka. So Kafka has a default of one megabytes per message in topics, which is honestly a very, very good default. And because large messages so bigger than one megabytes, they're considered inefficient and somewhat of an anti-pattern. But in case you really do need to send large messages in Apache Kafka, you have two approaches. Number one is to use an external store. It could be sending your message into HDFS, Amazon S3, Google Cloud Storage or whatever FTP you want, and then you send the reference of that message to Apache Kafka. I will show you this in the next slide. Or you can modify Kafka parameters. You must change the broker, producer and consumer settings to make it work, and I will show you that in the following slides as well. So option number one is to send the large messages using an external store. So for example, say you want to send a video, an…

Contents