From the course: Complete Guide to Apache Kafka for Beginners

Unlock the full course today

Join today to access over 23,200 courses taught by industry experts.

Wikimedia producer high-throughput implementation

Wikimedia producer high-throughput implementation - Kafka Tutorial

From the course: Complete Guide to Apache Kafka for Beginners

Start my 1-month free trial Buy for my team

Wikimedia producer high-throughput implementation

“

Hi, this is Stephane from Conduktor and in this lecture we,re to implement a high throughput producer. So we'll add snappy message compression in our producer and snappy is going to be very helpful if your messages are text based and they are for our use case. For example, if you have log lines or JSON documents and we have JSON documents. Snappy I like it because it has a good balance of CPU to compression ratio, but test whatever algorithm is good for you and make your own decisions. We'll also increase the batch size to 32kb and we'll introduce a small delay with linger millisecond to 20 millisecond. And we'll also check which partitioner is being used for our code. At the end our code is going to look like this. So let's get started. Okay. so let's launch our producer. while it is like this and we are going to have a look at the default value set for the settings. So I'm going to stop this. Okay, stop, stop, stop. Okay. So if we have a look at it, the batch size is 16kB. And then…

Contents