From the course: Complete Guide to Apache Kafka for Beginners

Unlock the full course today

Join today to access over 23,200 courses taught by industry experts.

Case study: Big data ingestion

Case study: Big data ingestion - Kafka Tutorial

From the course: Complete Guide to Apache Kafka for Beginners

Start my 1-month free trial Buy for my team

Case study: Big data ingestion

“

So Kafka historically was created to do big data ingestion. So it's a very common in the old days to have generic connectors that will take data and put it into Kafka and then from Kafka to offload it into HDFS, Amazon S3 or ElasticSearch, for example. So Kafka can serve a double purpose. In that case, it can be a speed layer for your real time applications while having a slow layer that will have applications extract in a batch manner in data stores that are going to be helpful, for example, HDFS and S3 when you want to do analytics. So Kafka as a front to big data is a very common pattern in the big data world, and it's also used as an ingestion buffers in front of other stores if you need some kind of buffer. So this is the architecture you would strive for. So you have your data producers could be any kind of data within your company that sends data into Kafka. There you would have a speed layer, so it could be your Kafka consumer, but also big data frameworks such as Spark…

Contents