Jonathan Ellis

In case you missed it, we have now released the second issue of our Confluent Chronicles – a fun and entertaining take on the fundamentals of stream processing, Apache Flink, and Confluent! We had such great feedback on the first issue that we couldn’t just stop there – you can check out and download the latest edition here: https://lnkd.in/gEFxA7af Would love to hear what you think of it!   #ApacheFlink #ApacheKafka #Confluent

39

1 Comment

Check out Jan Wieck's presentation, "Goodbye Postgres Sequences, Hello Snowflake Sequences" and discover different ways to avoid replicating sequences and how Snowflake IDs can help solve the problem, especially as we move to multi-master systems. Watch it today! https://hubs.la/Q02BJl-30 #pgedge #postgresql #postgresprofessionals #multimaster

20

1 Comment

I wrote a new post on the Sympathetic Ink blog to sum up The Deconstructed Database and what makes it composable. Learn more about the role of #Parquet, #Arrow, #DataFusion, Apache Iceberg, #Calcite and OpenLineage

63

4 Comments

Imagine seamlessly ingesting incremental data from *any* API right into #DeltaLake via Structured #streaming. This article by Irfan Elahi goes deep into the new Custom Data Source API in #apachespark, showing how you can create your own data source for streaming - with checkpointing capabilities to achieve exactly-once guarantees! https://lnkd.in/gEvr5kQJ

17

Have you tried running the DBRX models from Databricks? In our latest blog post we provide a tutorial to get you started with the model on Ori’s public cloud. We were very impressed with DBRX’s performance on the Hugging Face Open LLM Leaderboard before the updates last week. While DBRX Instruct did not score as highly under the new scoring system, it's still one of the best open models for Language Understanding (MMLU), Programming (HumanEval), and Math (GSM8K). We also used BeFOri, the benchmarking framework from Ori to determine DBRX Base performs inference most efficiently between 40 and 256 concurrent requests on one of Ori's 8 x H100 PCIe bare metal servers. Check out the blog to learn how you can use BeFOri to optimise self-hosted LLM performance on any infrastructure! https://lnkd.in/gi55WEks

43

Vishnu Vasanth will be discussing a fresh approach to building high-performance lakehouse compute engines for open table formats like Delta Lake, Apache Iceberg, and Apache Hudi at The Fifth Elephant 2024 Annual Conference. 🔍 Dive deep into: 1. Overcoming challenges with distributed compute engines like Apache Spark, Trino Software Foundation/Presto Foundation, and Apache Flink. 2. Implementing a Kubernetes-native, decentralized architecture for better performance. 3. Achieving high concurrency, complex queries, and handling massive data volumes with ease. Why attend? 1. Learn about a hyperefficient compute engine. Gain insights on eliminating single points of failure. 2. Understand how to maintain deterministic tail latencies and efficient cluster utilization. 🎟️ Don’t miss out – Buy your ticket or pick a membership now. https://has.gy/Naw4 #BigData #DataScience #MachineLearning #TheFifthElephant #DataLakehouse #DeltaLake #ApacheIceberg #ApacheHudi #Kubernetes Ververica | Original creators of Apache Flink®

3

Uncover the magic of self-querying retrievers with Snowflake Cortex AI in the latest blog post from #DataSuperhero Fabian Hernandez! Imagine your natural language queries transforming into super-smart, structured searches—effortlessly pulling out the info you need.

76

Interesting post by João Salgado about partitioning vs. sharding in #Postgres which lists trade-offs for each. It's great looking at the list of cons for partitioning and seeing that #TimescaleDB actually helps with most of them - Complexity ⬇️ because we transparently manage partition creation and lifecycle for you as data is ingested - Optimization Challenges ⬇️ because we have planner hooks that help with handling data exclusion (you still do need to pay attention to your partitioning key though). - Storage Overhead ⬇️ (although Postgres doesn't really have much overhead for partitions) because you can just choose to compress your data (we often see a 95x reduction in size). Timescale 🚀

18

1 Comment

Tune in below to watch one of our founders discuss how ParadeDB is built and how you can get started hacking on Postgres yourself.

4

Customers are using Amazon Managed Service for Apache Flink to process data in real-time. Data streaming enables generative AI to take advantage of real-time data and provide businesses with rapid insights. This post looks at how to integrate generative AI capabilities when implementing a streaming architecture on #AWS. https://lnkd.in/g9wezHvt #Streaming #Data #StreamingData #DataStreaming #Kinesis #ApacheFlink #Flink #Bedrock

62

1 Comment

Question: Considering migrating to Unity Catalog? Curious about potential errors or the rollback process? Watch this quick clip of Blueprint's Databricks Champion, Shannon Lowder, explaining just how easy it is to manage and reverse a migration if needed. #unitycatalogmigration #ucmigration #unitycatalog #databricks

8

📅 Mark your calendars for April 25! Join our very own Kenten Danas and Tamara Janina Fingerlin, as they share best practices for optimizing ML/AI #workflows using the latest #Airflow features, including the recent updates in 2.9. 🚀 We'll cover all the essential elements of Apache Airflow that are crucial for developing and scaling #ML/#AI pipelines. Secure your spot today: https://bit.ly/49DadsX #dataengineering #datascience #machinelearning #artificialintelligence #datapipelines #techtalk

34

Accelerate your Oracle to PostgreSQL migrations with DMS! Google Cloud Next '24 brings Gemini AI for code/schema conversions + explanations. Simplify the process, save time, & empower your devs. #GoogleCloudNext #Oracle #PostgreSQL #DMS

1

New propeldata.com engineering blog post just dropped → https://lnkd.in/gBWpkJNt Learn how you can leverage Propel's API to build analytics with real-time data coming from Postgres via Kafka. Includes step-by-step code examples and CLI commands to set everything up. 🤩 Including: ❖ How PostgreSQL CDC with Debezium works ❖ What is the Propel Kafka Datapool? ❖ Setting up Minikube, PostgreSQL, Kafka, Kafka Connect, and Debezium ❖ How to create a Propel Kafka Data Pool ❖ How to query the data via API ❖ Visualize the data

2

We have out-of-the-box support in LlamaIndex for Snowflake SOTA arctic-embed models (which outperform other leading embedding models in benchmarks) - it's a one-liner through our Hugging Face integration! And in ~4 more lines of code, you can build a RAG indexing pipeline over your data with these embedding models. Check out the screenshot below for how to get up and running with this - we might have more detailed benchmarks soon, stay tuned 💡

215

3 Comments

What are some of the core primitives that makes Apache Hudi a robust lakehouse platform? Apache Hudi started at Uber to deal with some of the critical challenges in #datalake - bulk uploads to Parquet (120 TB of data every 8 hours) - need to re-compute entire 120 TB - data staleness (freshness would be ~24 hours) - bad ingest performance - tons of compute, I/O - no concurrency control, atomicity guarantees, among others. To tackle these challenges it was important to establish some core primitives. ✅ Upserts -> build a system that could be capable of handling updates, deletes, upserts. So, you can absorb the changes faster into the table. ✅ Incremental Reads -> consume only the 'changes' instead of the full re-compute of TBs of data ✅ Snapshot Isolation -> query a snapshot consistently with no dirty data 🔗 Docs: https://lnkd.in/gbyc9CwF 📗 Uber's Blog: https://lnkd.in/dJRY9_Nq 📱 Slack: https://lnkd.in/gZSZzdX4 #dataengineering #softwareengineering

38

Developers can now store, index, and query vectors in PostgreSQL using pgvector, eliminating the need for a dedicated vector database. This new blog from Yugabyte expert Brett Hoyer deep dives into pgvector, and explains how to store text embeddings as vectors, execute similarity search, choose between indexes, and scale with distributed SQL.💡 Find out more!⬇️ https://hubs.la/Q02CNjWd0 #YugabyteDB #pgvector #vector #AI #Postgresql #cloudnative #distributedsql

10