Stream Processing with Apache Kafka, Samza, and Flink

• Eric Sun brings his 10 yrs experience on platform teams to the Data Pipelines team at GitHub. Our group oversees the messaging platforms that connect GitHub’s applications and data stores for millions of developers world-wide. He enjoys working on performance problems and specializes in incident analysis and remediation.

6:40 - 7:15: Revenue alerting at scale
Arsh Khandelwal & Kshitij Grover, Orb
In this talk, engineers from Orb will talk about the stream processing and query challenges of implementing a usage and spend alerting product features at the scale of millions of events a second. Orb powers billing for some of the world's fastest growing infrastructure companies, including Vercel, Replit, and Pinecone. In Orb, you can set up metrics (effectively queries) that determine the basis of consumption charges for your customers, and set up alerts that trigger as a result of customer spend. This translates to an invalidation architecture problem with some serious challenges to maintain correctness performantly, especially when allowing extremely flexible metric definitions.

• Arsh Khandelwal is an infrastructure engineer at Orb, and is responsible for a broad range of infrastructure efforts, including scaling our real-time cost alerting infrastructure, improving the reliability of our datastores, and tackling the ingestion system at Orb that handles billions of events a day. Before Orb, Arsh worked at Nuro on the platform team, responsible for storing logs across the system. Having seen a peek of how hard it was to build a system to track costs, Arsh wanted to bring business-aware data infrastructure to larger enterprises.
• Kshitij Grover is co-founder and CTO of Orb, the billing platform for the fastest growing infrastructure companies. Prior to founding Orb, Kshitij was an engineering leader at Asana. He received degrees in Computer Science and Philosophy at Caltech.

7:15 - 7:50: Streaming Data Enrichment and Analysis with Redpanda, CDC, and Python
Bryan Wood, Redpanda
Learn how to utilize a streaming data platform to facilitate real-time data enrichment from monolithic databases into modern, streaming architectures. We’ll leverage Debezium for Change Data Capture (CDC) and explore the development of a Python-based service that subscribes to this data for real-time processing and applies machine learning to perform sentiment analysis.
To show a practical use of how data streaming and processing pipelines work in real life, the processed sentiment data is then published on another topic, highlighting how simple this is to do now with new advancements in streaming and ML.
Attendees will gain insights into the use of Debezium with Redpanda, developing Python services for ML applications on streaming data, and strategies for data aggregation and visualization. This talk is ideal for product leaders, managers, architects, and technical leads interested in adopting streaming technologies to enhance data-driven decision-making in their organizations.

• Bryan Wood is a creative technology leader passionate about driving positive change and leading teams on the journey of continuous improvement. Bryan is inspired to deliver better product, quicker through low friction interactions, automation, and seamless integration with product organizations and stakeholders.
  Ideals & Values
  -- Assume good intent
  -- Innovation, Efficiency, and Simplicity lead the market
  -- Use the right tool for the right job
  -- Be passionate about your work
  -- Have as much fun as you can