Stream processing using Kafka

Editor's Notes

1. continuously, concurrently, and in a record-by-record fashion. But as a continuous infinite stream of data integrated from both live and historical sources.
2. A big data architecture contains several parts. Often, masses of structured and semi-structured historical data are stored in Hadoop (Volume + Variety). On the other side, stream processing is used for fast data requirements (Velocity + Variety). Both complement each other very well. This meetup focuses on real-time and stream processing.
3. IMAGE SOURCE https://image.slidesharecdn.com/demystifyingstreamprocessingwithapachekafka-161118053223/95/demystifying-stream-processing-with-apache-kafka-4-638.jpg?cb=1479447621 Synchronous and tightly coupled Scaling is possible by adding more instances to this service Latency sensitive and due to tight coupling its sensitive to failures.
4. you send all your inputs in and wait for your system to crunch all that data before it send all the output back.
5. in between request/response and batch systems. here you send some inputs in and you get some outputs back. this definition of SOME is left to the program. the o/p is available at variable times too. the BIG shift is that, stream processing knows that the data is unbounded and it shall never be complete. BENEFIT: It gives complete control to the program over the tradeoffs involved. (latency, correctness and cost )
6. DIY → you take your kafka libraries and you decide to decide to do everything yourself. If you have decided to do this then you should be aware of these hard problems.
7. producers publish data to Kafka brokers, and consumers read published data from Kafka brokers. Producers and consumers are totally decoupled, and both run outside the Kafka brokers in the perimeter of a Kafka cluster. A Kafka cluster consists of one or more brokers.
8. Kafka topics are divided into a number of partitions. Partitions allow you to parallelize a topic by splitting the data in a particular topic across multiple brokers — each partition can be placed on a separate machine to allow for multiple consumers to read from a topic in parallel. Consumers can also be parallelized so that multiple consumers can read from multiple partitions in a topic allowing for very high message processing throughput.
9. Kafka Connect is a tool for scalably and reliably streaming data between Apache Kafka and other data systems. It makes it simple to quickly define connectors that move large data sets into and out of Kafka. Kafka Connect’s scope is narrow: it focuses only on copying streaming data to and from Kafka and does not handle other tasks, such as stream processing,
10. Standalone: bin/connect-standalone worker.properties connector1.properties [connector2.properties connector3.properties ...] Standalone mode is the simplest mode, where a single process is responsible for executing all connectors and tasks. Since it is a single process, it requires minimal configuration. Distributed mode provides scalability and automatic fault tolerance for Kafka Connect. In distributed mode, you start many worker processes using the same group.id and they automatically coordinate to schedule execution of connectors and tasks across all available workers. curl -X POST -H "Content-Type: application/json" --data '{"name": "local-console-source", "config": {"connector.class":"org.apache.kafka.connect.file.FileStreamSourceConnector", "tasks.max":"1", "topic":"connect-test" }}' http://localhost:8083/connectors # Or, to use a file containing the JSON-formatted configuration # curl -X POST -H "Content-Type: application/json" --data @config.json http://localhost:8083/connectors
11. Sink connectors also have one additional option to control their input, topics - A list of topics to use as input for this connector