Redpanda Data

Big news!!! 🎊 Redpanda has acquired the popular stream processing and connectivity framework Benthos. That means with our new Redpanda Connect we're now the most comprehensive end-to-end streaming data platform! 🔗 Gone are the days of suffering through multiple projects that manually stitch data from your streams to your data warehouse or operational database. Read what our founder Alexander Gallego has to say about all that's new at Redpanda. 👇 https://lnkd.in/gpUfGYgs

🗣️ Don't miss this short interview with Michał Maślanka at #JOTB24. Michał is responsible for Redpanda Data's Raft implementation and cluster orchestration bits. #softwaredevelopment #distributedsystems #Raft 👉 https://lnkd.in/dTvbzKdw

🎵 Don't stop me now I'm having such a good time, I'm having a ball Don't stop me now 🎵 Once you get started with Redpanda, you won't want to stop at all! Join our free online session on "Getting started with Redpanda" to learn about Redpanda's architecture, deployment options, practical use cases, and best practices. 🗓 Wednesday, July 10th 🌎 Americas: 9 am PT | 12 pm ET 🌍 EMEA: 8 am ET | 1 pm BST Sign up to join👇 https://lnkd.in/gjd75ZgJ

let's go docs team! so cool to see

I will be speaking at ServerlessDays Bengaluru about Building Streaming ETL Pipelines with the Redpanda Ecosystem, including Redpanda Connect. Remember to mark August 31 on your calendars if you're going to be in Bangalore. #serverless #redpadna #streamingetl

Mission-critical #streamingdata apps, like for stock trading or real-time #analytics, can't afford to wait for synchronous replies and risk hindering performance and the user experience. Happily enough, the asynchronous request-reply pattern enables a system to handle multiple requests concurrently, maintain high throughput, and ensure responsiveness 🙌 In this tutorial, we show you how this handy pattern works, popular use cases, and how to implement it in an async #Python app. https://lnkd.in/gjj5uqn4

How do you build 𝗿𝗲𝗮𝗹-𝘁𝗶𝗺𝗲 ML systems, at 𝘀𝗰𝗮𝗹𝗲, 𝘄𝗶𝘁𝗵𝗼𝘂𝘁 𝗯𝘂𝗿𝗻𝗶𝗻𝗴 𝗰𝗮𝘀𝗵? 🧠 ↓↓↓ 𝗧𝗵𝗲 𝗽𝗿𝗼𝗯𝗹𝗲𝗺 🤔 Let’s say you work as an ML engineer at a fintech startup, whose flagship product is a mobile app for online payments. The company is still small, but nonetheless a critical problem you need to tackle from day 0 is the automatic detection of fraudulent transactions. The system needs to be > 𝗙𝗮𝘀𝘁, otherwise, you will detect fraud too late, and the predictions, no matter how precise they are, will be useless. > 𝗦𝗰𝗮𝗹𝗮𝗯𝗹𝗲 based on the volume of transactions your users generate per second. You aim to become a serious contender to VISA or MasterCard, so you want a design that can scale to over 65,000 transactions per second. > 𝗖𝗼𝘀𝘁-𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁. You don’t want high upfront costs 💸 and complex infrastructure to mange (well, who does? 😛) So the question is > How can you build a 𝘀𝗰𝗮𝗹𝗮𝗯𝗹𝗲 and 𝗰𝗼𝘀𝘁-𝗲𝗳𝗳𝗲𝗰𝘁𝗶𝘃𝗲 𝗿𝗲𝗮𝗹-𝘁𝗶𝗺𝗲 system to detect fraud❓ 𝗧𝗵𝗲 𝘀𝗼𝗹𝘂𝘁𝗶𝗼𝗻 🧠 To make your system capable of → ingesting transactions, and → producing fraud notifications to end-users at scale and fast you can use a streaming data platform (aka message bus), like Apache Kafka or, even better, Redpanda Data. 𝗪𝗵𝗮𝘁 𝗮𝗯𝗼𝘂𝘁 𝘀𝗰𝗮𝗹𝗮𝗯𝗶𝗹𝗶𝘁𝘆? 🎛️ The scalability of the system depends on 2 things: > Number of brokers in your message bus and the resources you allocate to them (CPU, memory…). > Number of Docker instances you run of your Fraud Detector Service. Horizontal scaling 𝗪𝗵𝗮𝘁 𝗮𝗯𝗼𝘂𝘁 𝗰𝗼𝘀𝘁𝘀? 💸 To run this system you need at least 2 things: > A 𝗰𝗼𝗺𝗽𝘂𝘁𝗲 𝗽𝗹𝗮𝘁𝗳𝗼𝗿𝗺, where your Fraud Detector Service runs. For example, Amazon EKS, Google Cloud GKE or Quix Cloud, and > A 𝘀𝘁𝗿𝗲𝗮𝗺𝗶𝗻𝗴 𝗽𝗹𝗮𝘁𝗳𝗼𝗿𝗺, like Apache Kafka or Redpanda Data. 𝗔𝗽𝗮𝗰𝗵𝗲 𝗞𝗮𝗳𝗸𝗮 𝗼𝗿 𝗥𝗲𝗱𝗽𝗮𝗻𝗱𝗮 ❓ Apache Kafka is built in Java and requires a complex mesh of services to run, while Redpanda Data is designed from the ground up in C++ and shipped as a single binary. Which means it is easier and cheaper to run. 🤑 𝗠𝘆 𝗿𝗲𝗰𝗼𝗺𝗺𝗲𝗻𝗱𝗮𝘁𝗶𝗼𝗻 💡 If you want to start building real-time ML systems, without investing a lot in upfront infrastructure costs, I recommend you try 𝗥𝗲𝗱𝗽𝗮𝗻𝗱𝗮 𝗦𝗲𝗿𝘃𝗲𝗿𝗹𝗲𝘀𝘀. → With a 𝗳𝗲𝘄 𝘀𝗲𝗰𝗼𝗻𝗱𝘀 you get your production cluster up and running 🚀 → You 𝗱𝗼𝗻’𝘁 𝗻𝗲𝗲𝗱 𝘁𝗼 𝗺𝗮𝗻𝗮𝗴𝗲 𝗮𝗻𝘆 𝗶𝗻𝗳𝗿𝗮𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲 🎉, and → You 𝗼𝗻𝗹𝘆 𝗽𝗮𝘆 𝗳𝗼𝗿 𝘄𝗵𝗮𝘁 𝘆𝗼𝘂 𝘂𝘀𝗲, so you go from 0 USD and scale based on your growth 🤑 Click here to try it for FREE ↓↓↓ https://lnkd.in/eSd-P4Es ---- Hi there! It's Pau Labarta Bajo 👋 Every day I share free, hands-on content, on production-grade ML, to help you build real-world ML products. 𝗙𝗼𝗹𝗹𝗼𝘄 𝗺𝗲 and 𝗰𝗹𝗶𝗰𝗸 𝗼𝗻 𝘁𝗵𝗲 🔔 so you don't miss what's coming next #machinelearning #realtimeml #streaming

📣 JOIN US ONLINE: Emergence of the Live Data Stack The #data world is quickly moving to real time. We call the cool #technologies enabling this shift the “live data stack,” where real-time processing replaces batch, real-time storage replaces the data warehouse, and #AI takes the place of analytics. To get the scoop on these next-gen technologies and the best approaches for migrating from batch to real time, join us for a live virtual event hosted by Joe Reis, co-author of The Fundamentals of Data Engineering, and our distinguished founder and CEO Alexander Gallego! 🗓 Wednesday, July 17th 🕘 9 am PDT | 12 pm EDT | 5 pm BST Sign up to join👇 https://lnkd.in/gEE-N6jK

Wanna learn how to do 𝗳𝗲𝗮𝘁𝘂𝗿𝗲 ����𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 in 𝗿𝗲𝗮𝗹-𝘁𝗶𝗺𝗲 ⚡ using 𝗞𝗮𝗳𝗸𝗮, 𝗗𝗼𝗰𝗸𝗲𝗿 and 𝗣𝘆𝘁𝗵𝗼𝗻 🐍 ? ↓↓↓ 𝗧𝗵𝗲 𝗽𝗿𝗼𝗯𝗹𝗲𝗺 🏔️ Feature engineering is about transforming raw data into predictive signals for your Machine Learning model. And for many real-word problems, you need to run this transformation in 𝗿𝗲𝗮𝗹-𝘁𝗶𝗺𝗲. Otherwise, your features will be irrelevant. So, as a professional ML engineer, you need to -> go beyond static CSV files and local Jupyter notebooks, and -> learn the tools that help you build production-ready real-time systems ⚡. 𝗘𝘅𝗮𝗺𝗽𝗹𝗲 💁 In this video I show you step-by-step how to build a real-time feautre-engineering service that - 𝗿𝗲𝗮𝗱𝘀 raw crypto-trade data from a Kafka topic (input topic), - 𝘁𝗿𝗮𝗻𝘀𝗳𝗼𝗿𝗺𝘀 this raw data into Open-High-Low-Close features using stateful window aggregations, and - 𝘀𝗮𝘃𝗲𝘀 the final features into another Kafka topic (output topic), so the data is accessible to downstream services. 𝗧𝗼𝗼𝗹𝘀 🛠️ We will use are > 𝗣𝘆𝘁𝗵𝗼𝗻 𝗣𝗼𝗲𝘁𝗿𝘆 to package our Python code professionally > Quix Streams to do window aggregations in real-time ⚡ > Docker, Inc and Docker Compose to containerize our service. > Redpanda Data to enable communication between our service and the rest of the infrastructure. ---- Hi there! It's Pau Labarta Bajo 👋 Every day I share free, hands-on content, on production-grade ML, to help you build real-world ML products. 𝗙𝗼𝗹𝗹𝗼𝘄 𝗺𝗲 and 𝗰𝗹𝗶𝗰𝗸 𝗼𝗻 𝘁𝗵𝗲 🔔 so you don't miss what's coming next #machinelearning #realtimedata #mlops #realworldml

Overly complex architectures, temperamental dependencies, and fluctuating #cloud costs are enough to make anyone sick of #streamingdata 🤒 Our prescription? Try Redpanda Serverless! With instantly available clusters, no infrastructure to manage, and pay-as-you-go pricing — it's proven to help you stream #data faster and easier 🌴 Try Serverless for free👇 (You'll feel better after the first dose. Promise.) https://lnkd.in/ga4j-Y6S #streamingdatadoodles