From the course: Complete Guide to Apache Kafka for Beginners

Unlock the full course today

Join today to access over 23,200 courses taught by industry experts.

Log compaction theory

Log compaction theory - Kafka Tutorial

From the course: Complete Guide to Apache Kafka for Beginners

Start my 1-month free trial Buy for my team

Log compaction theory

“

Hi, this is Stephane from Conduktor and now we're going to talk about the log cleanup policy called compact. So this is log compassion and log compassion means that your log contains at least the last known value for a specific key within a partition. I will show you a diagram that will make a lot of sense for you. So log compaction is very helpful when you just require a snapshot of the current state of a topic with the latest value of each key instead of the full history, which is, for example, log compaction is similar to data table and database where you just have well, the state of the table right now, but you don't know all the changes that happen to your table over time. So the idea is that we want to only keep the latest update for a key within our log. So here's an example that we'll make it concrete. Let's take an employee salary topic and you want to keep the only, only the most recent salary for employees. And what happens? Well, here is the order of rights from top to…

Contents