Sohel Khan, Ph.D., MBA

Top 5 LLM Testing and Evaluation Tools: 🔭 Galileo: An end-to-end platform for GenAI evaluation, experimentation, and observability AgentOps.ai: Build reliable AI agents with monitoring, testing, and replay analytics. Toloka: A data-centric environment to support fast and scalable AI development with the help of human insight. Gantry: Powerful observability, analytics, and evaluation for your AI-powered products. RagaAI Inc: Fix AI issues in three easy steps - Detect, Diagnose and Fix. #AI

42

Ali Azhar covers our news from #CCSummit: "The Confidential AI platform was developed at the Berkeley RISELab, a world-renowned lab known for developing technologies such as Apache Spark and Databricks. It was at this lab where the breakthrough MC2 (Multiparty Collaboration and Competition) platform was created, incubated, and open-sourced. In 2021, this platform served as the foundation to build the Opaque platform. Building on Opaque’s existing services that facilitate secure collaboration with cryptographic verification of privacy, the new platform allows organizations to unlock business insights securely and efficiently from sensitive data that wasn’t fully utilized before. Last year, Opaque announced key innovations to the platform including broader support for confidential AI use cases and new safeguards for ML and AI models from exposure to unauthorized parties. “Opaque offers a breakthrough for organizations struggling with the tension between innovation and security. By embedding privacy and security into every step of the ML pipeline, we enable enterprises to accelerate AI adoption confidently,” said Chester Leung, co-founder and Head of Platform Architecture at Opaque.  “Our confidential AI platform uniquely enables the processing of encrypted data without a noticeable performance hit at cloud scale. With Opaque securing entire data workloads, companies can unlock new business opportunities and manage risks effectively, all while maintaining absolute control and privacy of their data.” Use cases for the new platform span across various industries. In the high-tech sector, Confidential AI can be used to secure data pipelines for analytics and ML workloads and enable dynamic model training on encrypted data.   Users in the manufacturing sector can use the platform as a confidential control plane to enforce data governance rules. Financial services can also benefit from the platform through secure data sharing and collaboration across business units.  Human resources professionals can harness the power of the platform to securely share and analyze employee data across multiple data silos and ensure enterprise compliance with data privacy regulations. With the launch of the new platform, enterprises could finally have a solution to eliminate the tradeoff between innovation and security. As more organizations use the new platform, we will have a better understanding of the performance of Confidential AI in terms of integration with existing ecosystems and scalability."

16

3 Comments

Anthropic has achieved a breakthrough in understanding large language models like Claude 3 Sonnet. They identified how millions of concepts are represented within the model, using dictionary learning to uncover patterns in neuron activation. These features represent a wide array of entities such as cities, people, and scientific fields, and can respond to both text and images in multiple languages. Notably, they also found abstract features related to concepts like inner conflict and logical inconsistencies. By manipulating these features, they could change Claude's behavior, such as making it repeatedly mention the Golden Gate Bridge. This discovery enhances our understanding of AI models and could improve their safety in the future. For more details, visit the articles on Anthropic's website and Transformer Circuits. https://lnkd.in/eAp4TybE #AI #Claude #Anthropic #blackbox

31

2 Comments

Snowflake 🤝 Houseware 🤝 LangChain While building AI agents on Snowflake, we realized how the Langchain ecosystem didn't have established connectors to Cortex LLM models. Nipun & Shubham from the team recently built out a Chat Model integration for Snowflake Cortex inside Langchain's community libraries, and raised a pull request just earlier today(link in comments). This abstraction would allows developers to easily build AI applications on top of Cortex LLMs using the Langchain ecosystem, making Cortex a first-class citizen here onwards. OSS FTW 🚀

59

4 Comments

Our yearly report on the State of Spatial Data Science is out! Co-authored with our partners at Databricks, it will give you endless conversation topics for your summer barbecues on how the adoption of Spatial Data Science is growing across different industries. Check it out here: https://lnkd.in/dU4bFVdi

34

SAS, in collaboration with Coleman Parkes Research, surveyed business organizations about generative AI early adoption. Data analytics tell a story, and this one is no different. Take a peek at the report to see where your region lands on the leaderboard 🌏. http://2.sas.com/6047i2dlB #ArtificialIntelligence #GenAI

7

Route LLM can route queries between different LLM models based on human preferences. By routing between GPT-4 and Mixtral-8x7B, we demonstrate cost reductions of over 85% on MT Bench and 45% on MMLU while achieving 95% of GPT-4's performance.

2

My BHI colleagues just published a very interesting piece on “human in the loop” for AI mantra... To get maximum effect from AI automation - get human out of the loop!

5

This is a huge step towards aligning the open data lakehouse formats. Recently Microsoft announced compatibilty with Snowflake for their fabric OneLake which was already compatible with Databricks. The future of data lakehouses is open and interoperable… your storage format should not constraint you from using your favorite compute engine..

14

Interesting approach and interesting insights into how to obtain a #dataset to train #LLM : "Recently, we released 🍷 FineWeb, a new, large-scale (15-trillion tokens, 44TB disk space) dataset for LLM pretraining. FineWeb is derived from 96 CommonCrawl snapshots and produces better-performing LLMs than other open pretraining datasets."

3

�� I'm excited to launch the new Cube Semantic Catalog and AI Assistant today! 🤖 📒 I've thought for some time that having a catalog centred around the semantic layer makes the most sense. The semantic layer is the point at which data consumers and business users access data. They need to know what is available to use here, and how it relates to downstream data assets like dashboards and graphs in BI tools. Data engineers need to know how their work will affect the consumption of assets in the semantic layer, and beyond - they need to be able to maintain their API. The Cube Semantic Catalog enables both ends of this spectrum and users in between like Data Analysts and Scientists who need to understand the provenance of data they are using. On top of the Semantic Catalog, we are also launching our new AI Assistant. Users can ask questions about what objects are in Semantic Catalog but also ask data questions and receive answers and results as line or bar charts here. This is a semantic data intelligence experience - a way to discover your data and use it in the same place with no technical knowledge required. You can read more about today's launch in my blog post here: https://lnkd.in/gS8p7kQA #DataIntelligence #SemanticLayer #AI

82

2 Comments

Large language models #LLMs exhibit various skills to different degrees and this has enthralled the whole world leading to the flowering of a million startups, but do they have the ability to acquire skills? That ability to generalize from limited examples is needed for #intelligence! The #ARCPrize is based on #FrançoisChollet's paper "On the Measure of Intelligence" where he introduced a benchmark to measure the efficiency of AI skill-acquisition on unknown tasks: #Abstraction and #Reasoning Corpus. https://arcprize.org/arc (If you have to look him up, he is the creator of #keras, the deep learning framework.) If you are in the early stages of your career and are interested in research breakthroughs, you would be well advised to go after some of these ideas that need to be developed for getting to #AGI - LLMs can certainly do a lot and bring about another technological revolution but are not the panacea that many think they are : the lay person conflates language skills with reasoning ability. #MiraMurati (CTO of #openAI) got a lot of flak recently for saying that many creative jobs will be replaced by #ChatGPT and its successors. But she has a point: Are these jobs truly creative? As a society are we conflating skill with intelligence? https://lnkd.in/g5xR7wb4 As I think about this, it raises a different question, about how we train humans - are our schools #teaching skills or teaching skill acquisition? In the age of #generativeAI, are we training the next generation to be carbon based robots instead of intelligent thinkers? Should schools only teach #criticalThinking and do away with many traditional subjects? There is a secondary question: Is that appropriate for all kids? Should education style and content be differentiated by the abilities of the children so we meet each child where they are? Most schools do not do this right now - we live in the era of industrialized education. #noChildLeftBehind just means every child is reduced to being taught at the lowest possible skill level. (*) #joeLonsdale recently wrote up a tirade against #equity - how it is distorting behavior. The examples he has about college and school education are spot on. https://lnkd.in/gMTaDQ7t What revolution in #edTech is needed to change how we teach? How will the next generation compete with AI agents? (*) Here is a view point on why modern society has so few geniuses: the end of aristocratic tutoring, resulting in a move away from expensive customized 1:1 interaction with children for their development towards the mass education system in vogue now. https://lnkd.in/gYkk2n9z

12

Simple messaging, Andrew Ng knows how to simplify complicated topics, here he speaks of AI Agents ... not new but a nice breakdown of agent reasoning design patterns. Consider Reflection, Tool Use, Planning and multi agent collab can help aid solns to be more robust. Some excellent examples that are presented simply. I had the rare pleasure to work with Andrew and many of his graduate students in 2014-2015 around much smaller DNNs for ASR and IRC where we thought a billion params was a lot... how far we have come. https://lnkd.in/gK8DAQMv

23

2 Comments

One of the unintended (but beneficial) outcomes of the flexibility provided by transformer architecture is what I would call modeling convergence. Case in point is multi/cross-modality. However, extending to cybernetic scenarios like robots in real-world setting would require some clever thinking on cost and latency esp. when it comes to inferencing. Or, maybe a new architecture? https://lnkd.in/evhacsGV

37

2 Comments

Falkonry (Suhas Mehta and Ashis Khan) and iba AG (Eric Snyder) will present how steel makers are pivoting to AI & Data driven decision support system at AISTech 2024 today (May 6, 2024) - please join us if you are interested. I co-authored this paper with Eric Snyder of iba AG.

66

5 Comments

Data Lineage- The Key to Data Accuracy Lukas discusses high-quality data annotation, AI application shifts, and RLHF's role in refining large models. https://buff.ly/3yUezPU

3

How do you scale your AI Agent experiment into a real solution for production-level usage? The AI conversation reminds me a lot of the Big Data conversation from a decade ago. The initial concept works great but scaling it beyond just a few use cases falls flat. This doesn’t mean that AI can’t be useful, it just needs supporting technology to be truly effective. Many of these same AI problems existed in Big Data and were solved by clever and robust technologies. You can move more quickly and build better by leveraging tried and tested technologies! #ai #agents #scale #efficiency #bigdata

22

2 Comments

Curious about supercharging your ML workflows with #Spark on #Databricks? 🚀 Join Stuart Lynn, Sepideh Ebrahimi, and me as we dive into Spark for efficient parallelisation, ML, and MLOps. Discover how you can leverage Spark to transform your ML projects! 📺 Check it out: https://hubs.la/Q02xbXZL0 👇🏽 #MachineLearning #DataScience #AI

13