Marius Panga

Did you miss this awesome demo Hugo Lu did with Youssef Mrini over at Databricks 🏘 In Orchestra you can trigger and monitor Databricks Workbooks, Workflows, data quality tests, and lots more. Our users also leverage Orchestra's overarching monitoring and alerting capabilities to link upstream jobs like data ingestion (e.g. ADF) through Databricks into downstream jobs like Reverse ELT with Hightouch or Power BI dataset refreshes. Check it out here 🔗 https://lnkd.in/dU2grpum #dataengineering #databricks

7

1 Comment

Snowflake releases arctic-embed models, a family of 5 text embedding models that range from 23 to 334 million parameters. The models are the result of search knowledge, and research and development that Snowflake acquired last May via Neeva. 💵 The models outperforms retrieval performance, having one-quarter of the estimated parameters and one-third of the dimensions compared to Open AI. 📊 https://lnkd.in/gjJy9Yte Some more insights on how the technical achievements are detailed in this blog 🦾https://lnkd.in/g_aYgwrH

2

1 Comment

Check out this awesome demo Hugo Lu did with Youssef Mrini over at Databricks 🌊 🏘 In Orchestra you can trigger and monitor Databricks Workbooks, Workflows, data quality tests, and lots more. Our users also leverage Orchestra's overarching monitoring and alerting capabilities to link upstream jobs like data ingestion (e.g. ADF) through Databricks into downstream jobs like Reverse ELT with Hightouch or Power BI dataset refreshes. Check it out here 🔗 https://lnkd.in/dU2grpum #dataengineering #unifieddatacontrolplane

11

2 Comments

Great blog on Spark AQE!

13

"Databricks Ushers In New Era for Open #Data And #AI Governance With Unity Catalog", click to get the full story via the News Desk over on Elnion. https://lnkd.in/gTkxYDRB #dataaisummit #data #datalake #ai #ml #gpt #genai #cloud #analytics #bi #datascience #datascientist

1

Join SqlDBM for a live demo on Transformational Modeling (Tx) in Snowflake on Wednesday, July 17th at 9 AM PST! Key Features: ✅ Easy data transformation with templates. ✅ Full control over data creation and loading. ✅ Integrated relational modeling to save time. 💡 Q&A with Snowflake SuperHero Serge Gershkovich. Register below! #DataModeling #TransformationalModeling #Snowflake

9

2 Comments

Really interesting read on the evolution of Data Warehouses, Data Lakes and Lake Houses and how Databricks and Snowflake are battling for the top spot. https://lnkd.in/egg4P5VG This is one of the more complete analysis I've seen on the evolution of the analytics space and it's big gorilla's. In the end, it gets into the AI discussion of big LLM's vs. small models and how SLM's will have a place in the analytics world, which I completely agree with.

9

1 Comment

News: #snowflakeSummit. Open Data Revolution Continues My Analysis: Snowflake’s strategy around Iceberg is a nuanced topic with significant implications. While the simplicity of Snowflake’s compute services is enhanced by Iceberg, the real value lies in how Snowflake chooses to handle interoperability and external engine support. The Polaris news might crucial in defining its position in the ever-evolving data platform market. Polaris is another #opensource gift from Netflix. Implications: Many including us were asking..will Snowflake choose to embrace Iceberg and its open data potential, even if it means disrupting Snowflake’s current model? Looks like the team is going to disrupt themselves to bring in a new era in open data. The technology industry has a saying, especially true during major market shifts: “Better to eat your own before someone else does.” This means that companies should cannibalize their own advantages before competitors do. For Snowflake, this is happening as the fully embracing Iceberg and its open data capabilities before competitors leverage these advantages against them.

4

One of the trends emerging out of the Snowflake Summit last week is the release of companion apps for Snowflake's query engine. ➡️ We've always had excellent results with ANSI SQL and a rich native dialect; ➡️ Python/Scala/Java have been "transpiled" to SQL in recent times to support non-declarative data engineering; ➡️ Graph/Network analysis isn't supported in any way and is informally considered to be a bit too niche to be a native roadmap item; But there's a way to bypass the latter constraint now. One of the startups getting quite a bit of buzz was relational.ai. They bring a companion app to process graph analysis on data available in Snowflake. It is available as a Snowflake Marketplace App. Underlying, it is available on Snowflake Container Services. This service allows to run external software within the contract perimeters of your Snowflake tenant. As a result, we're now able to store and query relational data using declarative (SQL) as well as graph queries (using RelationalAI's REL language). This comes in addition to the recently released support for vector similarity and text search using the native Vector datatype and the upcoming Cortex Search indexer. It makes the "store once/process many" marketing claim just about as broad as it gets. Our first testing results are VERY promising, both from a technical as well as a governance point-of-view. Not ready to refactor your existing network analysis apps to REL? Apparently Neo4j has a similar marketplace app in the making. #snowflake #relationalai

14

Unity Catalog has been open sourced???!?! Maybe the most shocking announcement to come out of #DAIS. Great move by Databricks. UC has been criticized for being a proprietary, Databricks Only, catalog. Databricks open sourced Delta Lake in 2019 and it quickly became one of the most common lakehouse storage formats.. Now in 2024 Unity Catalog is being open sourced... how long will it take for UC to become one of the most adopted catalogs? #databricks #unitycatalog #oss

45

7 Comments

Off to Data+AI 2024 by Databricks ✈️ Looking forward to meeting folks in person & tuning into some of the amazing sessions! I will be talking about Lakehouse table formats (Apache Hudi, Apache Iceberg & Delta Lake) & most importantly interoperability with Apache XTable (Incubating) #dataengineering #softwareengineering

48

5 Comments

Informatica's GenAI assistant for data management is now GA💥 This will be an ideal case study for the risks and rewards of domain-specific GenAI. The rewards: more productive data teams, more accurate and more timely data to drive analytics. The risks: hallucinations, PII exposure, broken data pipelines. Informatica comes at this from a point of strength - broad portfolio, deep installed base, lots of metadata to train GenAI language models. Gaurav Pathak gave a compelling introduction and demo here at Informatica World. The stakes are high. Check out this article by Eric Avidon of TechTarget to learn more. Excerpts below. https://lnkd.in/gkrb3eRd "Informatica unveiled Claire CGPT and the low-code/no-code development interface during Informatica World, the vendor's annual user conference in Las Vegas. "Now, Claire GPT is generally available while the and the low-code/no-code development interface is in preview. Both make it easier to use a platform that plays an important role in helping organizations manage and operationalize data and are therefore significant, according to Stephen Catanzano, an analyst at TechTarget's Enterprise Strategy Group. "Kevin Petrie, an analyst at BARC U.S., noted that Claire has enabled data teams to prepare and govern data with automation, AI and metadata since it was first released seven years ago. Claire GPT builds on Claire's capabilities. "This announcement extends the role of GenAI within Claire -- a vision they articulated last year -- by using a natural language interface to handle commands that previously required mouse clicks or manual scripting," Petrie said. "Beyond addressing ease-of-use and productivity like many generative AI assistants, Claire GPT has the potential to be more accurate than some other natural language tools given Informatica's access to metadata, he continued. "Informatica's extensive installed base gives it a rich metadata repository on which it can train and fine-tune GenAI language models," he said. "This differentiates Informatica because they can produce models that are less likely to hallucinate, expose sensitive data or cause other governance issues." "Petrie, however, cautioned that although the low-code/no-code interface for generative AI development is compelling and addresses a real need, trained experts will still need to be involved in the application development process. "The tool will still require expert human oversight," he said. "Data engineers, data stewards, or business-oriented data consumers will need to carefully inspect the quality of outputs they get from these language models before they put anything into production. "The caveat of data democratization [is that] you still need … experts to ensure the artifacts and data outputs are trustworthy." Shawn Rogers Timm Grosser Ankur G. Sarath Chunchu Suresh Thottempudi #data #genai #informaticaworld

78

Data is the foundation of AI NEXT Presentation > https://lnkd.in/griR2Dy8 with Deepak and Tim Velasquez Get started with BigQuery Studio > https://lnkd.in/ghttE9dK Get started with Gemini in BigQuery > https://lnkd.in/ggUDtpB5

23

I look so much forward to discuss data architecture with two bright minds in the field, James Serra and Guillaume Bodet. I will interview James about his great book, Deciphering Data Architectures from O'Reilly, and then well contextualize these data architectures to our data marketplace in Zeenea. I'm sure architects and data practitioners will get a lot out of this webinar! #datamesh #datalakehouse #datawarehouse #datafabric #dataarchitecture #datamarketplace

13

1 Comment

Snowflake Arctic, a new large language model that’s open, enterprise-grade, and efficient, is now available on Microsoft #Azure AI.   This state-of-the-art model uses a Mixture-of-Experts architecture that activates 50% or fewer tokens than comparably sized models. Arctic excels in a variety of tasks, including SQL code generation, sophisticated instruction following, and complex data analysis and is available for unrestricted personal, research, and commercial use.   We’re upholding our promise to deliver the best selection of high-quality models so our customers have choice and flexibility.     https://lnkd.in/g3mxCMfa #SnowflakeArctic#EraOfAI #GenAI

417

7 Comments

NEW BLOG: Decoding Delta Lake Compatibility Between Fabric and Databricks 🧐 OneLake in Fabric is a game changer... we can now virtualize our Delta tables into OneLake and enable all sorts of powerful Delta-centric compute options, Power BI Semantic models included (DirectLake).. but how can we tell whether our existing Delta tables are compatible with Fabric Apache Spark Runtimes? Read this post and you'll even come away with a python library to help navigate your way into Fabric. #fabric #databricks #deltalake #dataengineering

85

12 Comments

10 years on, car manufacturers are still dealing with the defective #Takada airbags, impacting an estimated 1.6 million vehicles. Why is it estimated, and how can Snowflake help prevent these data problems? Find out in Saša M.'s video (with cameo by me ;-) and Medium articles. Spoiler: find out how Snowflake is extremely fast at storing, calculating, and serving up hierarchies, ragged, historical, security, or otherwise. It can use parent-child, layered, or arrays. #mydata #manufacturing #sap #tableau #powerbi #sigma #automotive

65

2 Comments

"Databricks Ushers In New Era for Open #Data And #AI Governance With Unity Catalog", click to get the full story via the News Desk over on Elnion. https://lnkd.in/gDnArSJ5 #dataaisummit #data #datalake #ai #ml #gpt #genai #cloud #analytics #bi #datascience #datascientist

1