Apache Con 2021 Structured Data Streaming
•Download as PPTX, PDF•
1 like•332 views
Type safety is extremely important in any application built around a stream / queue. Type definition and evolution can either be built in the application or relied upon the data layer to support it out of the box allowing the application to only concentrate on business logic, not how of data store and evolution. It is this property of the good old relational databases (among others) that make them a favourite among all the modern NoSQL databases. Modern software architectures requires asynchronous communication (via stream / queue). While the data store and query design changes with asynchronous communication, type safety is still equally important. In this slide deck, used for Apache Con 2021 talk, we will go over ways in which one can force structure (schema) over the streaming data. As an example, we will talk about Apache Pulsar. Apache pulsar offers server as well as client side support for the structured streaming. We have been using pulsar for asynchronous communication among microservices in our nutanix beam and flow security central apps for over 1.5 years in production. This deck presents the technical details on what is schema, how to represent schema, what is available in the apache pulsar server and client side, how we have used pulsar’s schema support to build our use cases and our learnings from them.
Report
Share
More Related Content
What's hot
What's hot (20)
Similar to Apache Con 2021 Structured Data Streaming
Similar to Apache Con 2021 Structured Data Streaming (20)
More from Shivji Kumar Jha
More from Shivji Kumar Jha (18)
Recently uploaded
Recently uploaded (20)
Apache Con 2021 Structured Data Streaming
- 1. STRUCTURED DATA STREAMING SHIVJI KUMAR JHA @ShivjiJha TRACK: STREAMING in/shivjijha
- 2. About Me • Senior MTS at Nutanix • Platform Engineer – DBs, SOA, Infra, Streams • Love – Distributed data systems – Open-source software (OSS) • OSS Contributions – Apache Pulsar – MySQL
- 5. A Brief History… Of Databases • 1960: Flat Files • 1960s: Hierarchical Databases – Need for structure • 1980: SQL / Relational Databases – High-level language – Some more structure! • 2004: NoSQL – Scale & Availability above all – No relational model, less structure • 2010s: Distributed SQL – Well, no, we need structure* Image source: https://commons.wikimedia.org/wiki/File:Human_evolution.svg
- 6. A Brief History… Of Data Streams • Apache Kafka: – Built inside LinkedIn – 2011: Kafka becomes open source – 2012: Graduated from Apache incubator • Apache Pulsar – Built at Yahoo – 2016: Contributed to Open source – 2018: Top-level Apache project Image source: https://commons.wikimedia.org/wiki/File:Human_evolution.svg
- 7. History tell us… Evolution • SQL -> NoSQL -> Distributed SQL – Relation database has strict schema • Streaming bytes -> Schema Registry – Both Kafka and Pulsar support schema registry now! – Its not ideal to stream bytes • Use schema wherever possible – Err on the side of having schema OBJECTS PRIMITIVE TYPES BYTES BITS
- 11. Computers only know bits… Encoding data Example: Write an employee record • Sending data to a computer – Local computer – Over network • Can't send as is. • Encode to bits – Also, serialization • Send https://www.raywenderlich.com/books/swift-apprentice/v6.0/chapters/22-encoding-decoding-types
- 12. Computers only know bits… Decoding data Example: Read an employee record • Read data from a computer – Local computer – Over network • Turn bytes to employee rocrd • Decode to bits – Also, de-serialization • Use in program. https://www.raywenderlich.com/books/swift-apprentice/v6.0/chapters/22-encoding-decoding-types
- 13. Encoder / decoder placements Produce Data Consume Data • Choice 1 : App
- 14. Encoder / decoder placements Produce Data Consume Data • Choice 2 : Producer/Consumer
- 15. Encoder / decoder placements Produce Data Consume Data • Choice 3 : Stream Platform
- 16. Abstraction vs Flexibility Abstraction • Abstract out encode / decode to the stream platform • Lighter Individual apps – Single Responsibility Principle • Easy Evolution of schema – Versioning • Less bugs! Flexibility • Keep encode / decode in (each) app • Flexibility of choice – Schema formats • Schema evolution is hard – Versioning – Upgrade Path? • More bugs?
- 17. Flexibility? : Choose Wisely…
- 18. Flexibility? : Choose Wisely… • Flexibility is a good choice when: – Non-uniform data • Custom Fields • Non-uniform Types – Frequent schema migrations
- 20. Schema : Choice 1 Use Native serialization of programming language • Examples: – Java serialization – Python’s pickle – Ruby’s Marshall • Good – Easy implementation • Bad – Locked with same programming language for producer and consumer – Difficult to Evolve schema versions • Upgrade Path?
- 21. Schema : Choice 2 Use same format as web APIs (REST?) • Examples: – JSON – XML • Good – Familiar implementation, share code! – Text, readable, easy to debug • Bad – Key Name in every message, too much data – Auto detected type, may go wrong… – New types? Nested types? Ship POJO library? • Document? Synchronize? Ignore new data?
- 22. Schema : Choice 3 Struct Schema : Avro, Thrift, Protocol Buffers • Good – Binary formats, less space – Matured over the years – Well documented – Libraries in multiple languages – Good support in stream ecosystem – Evolution with versioning • Bad – Extra learning curve
- 23. Case Study: Apache Pulsar Schema
- 24. Apache Pulsar 101 PRODUCER CONSUMER • Cloud-native, • Distributed messaging and • Distributed streaming platform Apache Pulsar • Modular Design • Horizontally scalable • Low latency & high throughput • Multi-tenancy • Geo Replication Highlights
- 25. Pulsar schema : Byte Schema Domain Object Pulsar Producer : Sample Code
- 26. Pulsar schema : String Schema Producer Consumer
- 27. Pulsar schema : Avro Schema
- 28. Pulsar schema : Schema Registry • Topic to schemas mapping. • Stores accepted schemas for a topic. • Manages evolution with versioning. • Producer adds schema, if compatible. • Consumer fetches schema, given topic & message. • Schema => [name, payload, type, properties]
- 29. Schema Evolution • Manual – Check every schema before upgrade • Auto – updates – If new schema passes compatibility tests, producer uploads new version of schema.
- 30. Schema : Compatibility modes
- 31. Schema : Compatibility modes
- 34. Pulsar schema : Schema Registry AUTO_PRODUCE • Validates whether the bytes sent is compatible • If not, rejects. Produce<byte[]> pulsarProducer = client.newProducer (Schema.AUTO_PRODUCE()) … .create(); AUTO_CONSUME • Validate whether bytes sent from topic is compatible with schemas on topic. • If not, rejects. Consumer<GenericRecord> pulsarConsumer = client.newConsumer(Schema.AUTO_CONSUME ()) … .subscribe();
- 35. Topic Schema Mapping • Topic : schema = 1:1 ? • What about relative ordering? • Opinion: – Model domain to topic – Domain may have multiple schema. • Example : User, accounts, subscription – Prefer relative ordering – Work with parallel evolution • User v1 -> User V2 -> User V3 • Account v1 -> Account V2 -> Account V3 • Subscription v1 -> Subscription V2 -> Subscription V3
- 36. Schema across pipeline • Pulsar IO – Source (Examples : Flink, Spark, Elasticsearch) – Pulsar – Sink (Examples : Flink, Spark, Elasticsearch) • Same schema across pipeline – Unless you decorate.. – Unless different format for optimization • Type of query
- 37. Learnings
- 38. Learnings over the years • Struct schemas model domain objects well. • Binary representation is space efficient. • Use schemas management on apps, only if you need that extra flexibility. • Use schema registry by default. • Recommend Avro – Json schema – a bit too verbose – Proto awesome, not adopted well among sources / sinks. – Avro is adopted really well. • Decide and set compatibility / evolution rules. Worth it!
- 39. References 1. Pulsar docs: https://pulsar.apache.org/docs/en/schema-get- started/ 2. Schema auto update strategy: https://pulsar.apache.org/docs/en/pulsar-admin/#set-schema- autoupdate-strategy 3. Schema Evolution in Avro, Thrift, Protobuff: https://martin.kleppmann.com/2012/12/05/schema- evolution-in-avro-protocol-buffers-thrift.html 4. Topic design per domain: https://www.confluent.io/blog/put- several-event-types-kafka-topic/ 5. Schema Compatibility Design: https://docs.confluent.io/platform/current/schema- registry/avro.html#compatibility-types