Shane Johnson

We are excited to announce the launch of Ursa engine, our Kafka API-compatible data streaming engine that is built on top of Lakehouse. Ursa is not something entirely new; it is built on our many years of experience building and operating #ApachePulsar and #ApacheKafka. To learn more about the #Ursa engine, I recommend our #PulsarSummit keynote presentation and announcement blog post. Here, I’d like to delve deeper into some of its features: 1. Making Kafka API a first-class citizen: In #ApachePulsar, we developed a unified messaging model combining queuing and streaming semantics into one protocol, aimed at enhancing data sharing between business applications. Although powerful and increasingly adopted by enterprises and unicorns, the requirement for application rewrites is impractical in today’s economy. Instead of building data gravity around a protocol, we open up access to the robust stream storage layer by extending support to the Kafka protocol, transitioning from a single protocol to a multi-protocol platform, facilitating greater flexibility and integration. 2. Achieve Stream-Table duality via Lakehouse Storage: Data streams commonly end up in data lakehouses, where integrating and maintaining them can be complex and resource-intensive. Ursa simplifies this by storing data streams as lakehouse tables in object storage. This integration not only streamlines operations but also allows seamless access to operational data for analytical processes. Thanks to its multi-modal storage engine which fosters data sharing across teams and departments. 3. Object storage as the primary storage solution: To better align with cloud economics, particularly for large workloads, Ursa makes ZooKeeper and BookKeeper optional and utilizes a cost-optimized write-ahead log based solely on object storage with leaderless and stateless brokers. This architecture leverages object storage for both data replication and storage, minimizing inter-AZ data transfers and reducing costs for latency-relaxed, high-volume workloads. By integrating these features at the multi-tenant level rather than at isolated cluster levels, Ursa eliminates costly data duplication and movement across clusters, allowing teams to map directly to tenants and be able to share data across teams and departments more efficiently. We believe that the future of data streaming platforms lies in being open and multi-faceted, fundamentally different from traditional data platforms. They are designed not just to store and process data, but to enhance organizational capabilities by fostering data sharing across the organization and integration with other systems. More and more data streaming platforms will look more like Ursa, with multi-protocol, multi-tenant, and multi-modal capabilities. Check out our announcement blog post about Ursa: https://lnkd.in/ghEKTYE6 #ApacheKafka #Lakehouse #DataStreaming

100

1 Comment

It is amazing to see the integration between Snowflake and LANDING AI working so smoothly. It unlocks a whole new world of possibilities, and I'm really proud of being part of the team making this happen! Now you not only have the possibility to run LandingLens in your own private Snowflake account, but also integrate it with the whole ecosystem of Snowflake solutions, including Cortex Co-pilot. In the video below, Jeff Hollan shows how to use a simple streamlit app and Cortex co-pilot to chat with the inferences generated by LandingLens. #AI #computervision #datascience #dataengineering

10

Moderated a session this morning on applied GenAI/LLMs for a broad group of folks, thanks to Hydrolix.io for sponsoring the event outside RSA. As many folks think about building out "chain of thought" RAG pipelines, I wanted to mention there was a cool effort by a group to expand the context window of LLAMA 3 from 8k to 1M tokens. https://lnkd.in/eK8U3Pms I started to test with a typical needle in a haystack test, but discovered we need a better way to test efficacy- that more accurately aligns with the way we retrieve data from files and content blocks provided to the model. I will share more as a solution is fully identified. BUT, this Gradient model is a great potential addition to those using GenAI Assistant and Agents with local enterprise data, such as large log datasets housed in systems such as Hydrolix.io.

7

Included in this article is the mention of vector databases. Unsurprisingly, Michael Stonebraker and Andy Pavlo feel there is not a need for vector-specific databases. They believe relational databases will adopt their features, which is already happening. Check out the details here: https://lnkd.in/gyq_4WjP

27

1 Comment

Anthropic just launched Claude 3.5 Sonnet, which beats both OpenAI GPT-4o and Anthropic’s previous paid model Claude 3 Opus on most benchmarks. Claude 3.5 Sonnet can execute and debug code, and has improved image understanding capabilities, including text transcription. A new feature called “Artifacts” enables users to view generated code, documents and websites in a separate window next to the chat. Claude 3.5 Sonnet is available for free to all users, but paid users get higher usage caps. Did Anthropic just take the lead back from OpenAI? It seems that ChatGPT’s Code Interpreter tool, which used to be one its main advantages, has finally met its match, but ChatGPT still has the advantages of real-time Internet access and custom GPTs which Claude lacks… What do you think? https://lnkd.in/d_hk86cf

10

3 Comments

Ready to explore the journey of adopting Generative AI? Check out our latest blog to navigate the roadblocks, pit stops, and destinations of integrating AI into your business. Kyle Gershuny shares valuable insights on how to harness AI's power while avoiding common pitfalls. Learn how to tackle challenges like data privacy, trust, and skill gaps. 🌟 See how Vectara can guide you smoothly, ensuring efficiency and reliability in your AI projects. 👉 Ready to get started? Take Vectara for a test drive: https://gag.gl/liLvrm?

24

In data engineering, a lot of time is spent ensuring data integration between companies, organizations, applications, and clouds. Making sure data is securely accessible by connecting to internal and external systems while monitoring to make sure data arrives when expected consumes an oversized amount of data engineering time and mindshare. I’ve been spending time recently learning about Vendia and its ability to connect disparate data sources for real time secure data sharing and processing.

42

The USPTO team integrated Sonatype solutions and revolutionized their approach to software development. By automating the critical aspects of the #softwaredevelopmentlifecycle, they’re not just keeping pace with technology—but leading the charge in IT innovation. Curious about their journey and how we're tackling challenges with #opensource dependencies head-on? Dive into our story 👉 https://bit.ly/4aSrCz2

8

Unapologetically Technical's newest episode is now live! In this episode of Unapologetically Technical, I interview Hubert Dulay the author of Streaming Data Mesh and Developer Advocate at StarTree. We talked about his early experience with web backends like CORBA and SOAP and how those prepared him for data work. He shares his advice for those with web development skills to transition into data and what it's like for a person leaving a company after a long tenure there. We discuss his time at Cloudera and Confluent and how the data industry has changed over time. We talk about his recent experience with Flink and StarTree. We go in-depth into streaming data mesh, covering what it is and how to create real-time data products. We discuss the ways teams can create joined or enriched data products across domains. We round out the discussion with some tips on how to get data mesh adoption. Watch the full episode here:

10

1 Comment

Great SRE Interview Questions 

4

I want to highlight a new feature in our latest release also included in our free Community edition ! Offset-Based Retrieval in Kamiwaza's Full Gen AI Stack: Seamless RAG Flow at PB+ Scale! A significant advancement in our Full Gen AI stack at Kamiwaza: out-of-the-box offset-based retrieval support, tailored for Enterprise-scale Retrieval-Augmented Generation (RAG) flow. #### What Does This Mean? - **Seamless Pagination:** Offset-based retrieval enables effortless pagination through massive datasets. This feature lets your enterprise handle enormous data scales, efficiently delivering specific subsets of data based on your application's requirements. - **Optimized Performance:** Our implementation ensures that even with petabyte-plus data volumes, your queries are streamlined for speed. Retrieve exactly what you need, without overloading your infrastructure or consuming unnecessary resources. - **RAG Flow Ready:** Retrieval-Augmented Generation (RAG) is critical for enterprises leveraging AI to augment their workflows. Our stack’s native offset-based retrieval ensures that RAG systems can navigate vast datasets, retrieving relevant information with unparalleled efficiency. #### Why Offset-Based Retrieval? Handling large-scale data is challenging, particularly when your AI systems require real-time responses. Offset-based retrieval allows you to: - **Navigate Efficiently:** Skip irrelevant data and jump straight to the relevant subset for your application. - **Scale Seamlessly:** Manage and manipulate data even at petabyte-plus scale, keeping your AI applications responsive and efficient. - **Maintain Consistency:** For use cases like search, recommendations, and dynamic data exploration, consistency is key. Offset-based retrieval helps manage this seamlessly. #### What to Expect? With Kamiwaza’s Full Gen AI stack, enterprises can: - **Empower AI Workflows:** Quickly and efficiently support retrieval-augmented workflows, ensuring your AI models access the most relevant data. - **Enhance Decision Making:** Our optimized data retrieval enables faster, more informed decisions driven by the power of your data. - **Streamline Operations:** Offset-based retrieval reduces the complexity of managing vast data landscapes, allowing your teams to focus on deriving insights. We believe that this feature will significantly enhance the performance of our AI stack, helping enterprises unlock new levels of productivity and data-driven insights. #GenAI #1TrillionInference #KamiwazAI #EnterpriseAI Check out https://lnkd.in/gAwbGdGH for the latest updates!

3

3 Comments

This is the initial infrastructure Tradescoop was built on. A few things have changed, some additions and removals; for example, we’ve gone from MongoDB on Digital Ocean to Mongo Atlas, Redis cloud to Memorystore, but the general appraoch remains. Simplicity. Shouts to Karl Obinna Amalu, CSM for cleaning it up and adding color ✨ #gcp #infrastructure #devops

19

1 Comment

Cohere Command R/R+ models are now available on Amazon Bedrock. The models are particularly well suited for RAG use cases and are multilingual. #AWSome #GenAI #AmazonBedrock #Cohere https://lnkd.in/dTJC5Pix

4

What OpenAI's GPT-4o release means for the future of SaaS -- I'll break down the threat in my new article for Inc. Magazine https://lnkd.in/eUvZ4Ywq #innovation #technology #AI #software #saas

3

It’s hard to do math about actual human people. And developers are people! Really enjoyed this #PlatformEngineering #CaseStudy with Workhuman’s Andrew Keogh, focusing on answering four key questions: - How does a #SaaS company with more than a two billion-dollar valuation prepare to deliver on its goal of making work more human?  - How can it be sure its technology can scale without over-scaling its tech team? - How can the work of developers be more measurable — while still treating them as people not cogs of #DeveloperProductivity?  - How can #GenAI help devs focus on exciting, problem-solving work? Unsurprisingly, DX is part of the answer — as is (again not surprising) a necessary culture of #PsychologicalSafety. Cheers also to Aline De Oliveira, Jenna West, Brook Perry and Vicki Walker for helping to get this inspiring story out! https://lnkd.in/e5rj3gGD

41

2 Comments

#Automation comes in various forms. One is robotics, like the article below. Another is process automation, driven by data and software, solving the same labor shortage problem higlighted in the TechCrunch article. Entytle automates tedious, time consuming and laborious Installed Base workflows for OEMs, using data, AI and software. Different side of the same labor shortage coin. #installedbaseintelligence #oems #laborshortage #productivity

15

4 Comments

Business model innovation around Retrieval Augmented Generation from O'Reilly -- they have indexed all the content on Safari and are using it to provide answers grounded in vetted and credible content (their unique sauce). Another innovation here is to credit/compensate authors for the fraction of their work used in a generated answer. While this is progress, the more well-grounded the summaries look, the more dangerous the inevitable mistakes will be -- irrelevant documents will still lead to misleading summaries, and there will still be hallucinations because of interpolations with the text used to train Llama. At least based on the article, I don't see them employing self-checks or guardrails. I hope those are coming. https://lnkd.in/gqqHiR2D

42

1 Comment

🚀 The 2024 State of Engineering Management Report is now LIVE! Dive in to uncover how engineers are shaping innovation and growth: ➡️Engineers as strategic partners: Leading business strategy. ➡️Embracing AI: Driving productivity by 94%. ➡️Addressing burnout: A critical concern for 65% of respondents. With insights from over 600 professionals, this report offers invaluable data! Don't miss out—read the full report!

4