Deepak Kumar

https://lnkd.in/gWUpZECK Mixture of Agents (MoA) is an approach to harness the collective strengths of multiple LLMs to improve state-of-the-art quality. According to Together AI, it achieves a score of 65.1% on AlpacaEval 2.0, surpassing prior leader GPT-4o (57.5%).

5

FOSSA has published a lot of content over the past year around the topic of SBOMs and what they're useful for. Today, we're rolling it all into one easy learning portal meant to serve as a comprehensive at-a-glance of key concepts starting with: 1. Elements 2. Formats 3. Security use cases 4. Compliance use cases 5. Tooling / best practices We're going to keep building upon this over time and updating it as standards, regulations and best practices change. If you're interested in following the topic, give this a bookmark! https://lnkd.in/exyftA_z

42

3 Comments

There is some confusion in the industry about causal reasoning versus other methods involved in observability. It's nuanced, but also clearly differentiated. If you want to see it in action, here's a blog on it https://lnkd.in/dk9xn33K

7

1 Comment

Metrics for LLM. Nice article explaining the Grounding, hallucination, data correctness for fine-tuning, Prompt correctness, and RAG effectiveness by computing relevant Context awareness. https://lnkd.in/gFQMQ_ZR

32

Is the age of Superintelligence coming? "Imagine 100M human research equivalents running 24/7 at 100x more than at current human speed. What kinds of breakthroughs will we see?" This video is long but if you want to get a sense of the scale has the potential to reach it is worth a watch. Mind blowing stuff. https://lnkd.in/eA-MK_yK #AI #ChatGPT #Gemini

8

Looking forward to this panel where Divakar Tantravahi, Meduri Ravi Kumar, and Cosimo Spera will talk about the latest AI technologies that are changing product and customer support, along with their real world examples and practical strategies. If you have not registered, use this link - https://lnkd.in/gdTMgryA #ai #businessoutcomes #product #customer

22

LLM Critics Help Catch LLM Bugs The Paper highlights the problem of human evaluators' limited capacity to assess the outputs of large language models. It introduces the concept of training LLM critics using RLHF to provide natural language feedback on code, showing that these model-written critiques are often preferred over human critiques. Key Findings : ���� Model-Written Critiques Preferred: Critiques generated by CriticGPT were preferred over human critiques in 63% of cases. 🔹Higher Bug Detection Rate: CriticGPT identified more bugs in code than human contractors. This includes detecting hundreds of errors in ChatGPT training data rated as "flawless." 🔹Human-Machine Teams: Teams consisting of humans and CriticGPT provided more comprehensive critiques while reducing hallucinations compared to models alone. 🔹Comprehensive vs. Precise Critiques: Critiques that were more comprehensive (covered more points) tended to also have more hallucinations or nitpicks. FSBS helped balance this tradeoff by generating critiques that maximized comprehensiveness while minimizing spurious claims. 🔹Improvement with RL Training: Training with RLHF significantly improved the performance of CriticGPT across different model scales, reducing the rate at which bugs were missed. 🔹CriticGPT Outperforms ChatGPT: CriticGPT critiques were substantially preferred over those from prompted ChatGPT and caught more bugs. Human Inserted vs. Detected Bugs: Human-inserted bugs were harder to detect and more severe than naturally occurring (detected) bugs in the ChatGPT pipeline. 🔹Generalization Beyond Code: CriticGPT also performed well in identifying issues in non-code tasks, suggesting its utility beyond code reviews. 🔹Adversarial Tampering: The tampering process, where contractors inserted bugs into code, provided a high-quality source of difficult bugs, grounding the evaluation of critiques and improving model training. 🔹Impact of Model Scale: Larger models improved bug detection rates, but CriticGPT achieved high performance without needing significantly larger compute resources. 🔹FSBS Effectiveness: Force Sampling Beam Search (FSBS) allowed CriticGPT to generate more comprehensive critiques with a reduced rate of hallucinations compared to a simpler sampling strategy. The paper concludes that LLM critics significantly enhance the evaluation of AI outputs, particularly in code generation tasks. This improvement is crucial for developing more reliable and safer AI systems. The need for scalable oversight methods is emphasized, ensuring that AI behavior aligns with human values and expectations even as models become increasingly intelligent. #GenAI #LLM #NLP #datascience #ML #AI #RAG https://lnkd.in/gGeJ_pnk

3

1 Comment

"Productization is an affirmation," writes Adrian Bridgwater, "when any good or service evolves to a point where it can be productized and boxed, it denotes a certain solidity and branded consumability." In our conversation, I was delighted to share how #SAS equips organizations with readily deployable AI technology to productionize real-world use cases. https://lnkd.in/enUGCgAi #AIML #sasusers

9

Had a great conversation with Jabari Young from Forbes about the $13.8 million AI Squared just raised in Series A funding, led by Ansa Capital and New Enterprise Associates (NEA). It’s exciting to watch AI Squared grow, because helping organizations deploy and streamline their use of AI is something I’ve been working on for a long time. During my time at the National Security Agency, I started thinking on how to accelerate and streamline the way organizations adopt AI. Leading their data science team, I saw first hand that many of the AI/ML solutions developed were never actually able to keep people safer - and that mattered to me, because I had two brothers deployed overseas. I knew then I wanted to fix this problem. I went to Databricks to better understand how to incubate a technology and bring it to market - and worked with Matei Zaharia, Ali Ghodsi, Ion Stoica, Pete Sonsini, Greg Papadopoulos, Luke Cooper and so many others who helped me understand what it is to build a great business. I work now with talented people who bring experience from the data science and technical teams of Microsoft, JPMorgan Chase & Co. Chase and the United States Department of Defense, and I am grateful for the growing number of people invested in AI Squared’s success. As part of today’s investment, Roger W. Ferguson, Jr., former Vice Chairman of the Federal Reserve and board member of Alphabet Inc., has also invested and will join the company’s Board of Directors. Ansa Capital co-founder and General Partner Allan Jean-Baptiste will also join AI Squared's board. I’d like to thank Ansa Capital for leading this funding round and existing investors New Enterprise Associates (NEA), Latimer Ventures and Ridgeline. We’re excited to use this funding to expand our team and mature our platform - because we’re invested in helping businesses truly maximize the value of AI. Read more about the funding round and AI Squared of Forbes here: https://lnkd.in/edTdauaZ

406

68 Comments

5 Ways Telecommunications Companies Can Monetize AI https://lnkd.in/gkFwWcg2 #telecommunications #AI #monetize #business #artificialintelligence #services #networkoptimization #personalizedmarketing #frauddetection #digitalcommunications #technology #innovation

5

♟️ Don't play quantum games -- you'll win and lose at the same time. Today's dispatches from quantum: ⚆ Riverlane collaborates with global leaders to reduce the environmental impact of quantum computing through the Quantum Energy Initiative ⚆ A new recurrent quantum embedding neural network method combines quantum computing with NLP to improve memory efficiency and detection accuracy in vulnerability detection ⚆ An experimental demonstration of the odd-cycle game shows a quantum advantage over classical strategies using entangled ions with high detection efficiency ⚆ Queen Mary University of London upgrades quantum research capabilities with the state-of-the-art ProteoxMX dilution refrigerator ⚆ EV Group and Fraunhofer IZM-ASSID develop advanced bonding and debonding technologies to improve semiconductor integration for quantum systems ⚆ NBC News takes a tour of Google's Quantum AI Lab ⚆ Alberto Prado from Unilever discusses the transformative impact of quantum computing and the importance of data management at the Ai Summit London ⚆ A feedback-based quantum algorithm introduces an operator to encode solutions for quadratic constrained binary optimization problems, improves performance and reduces resource usage ...and more. 📰 Stay quantum-savvy. The Daily Qubit offers daily insights into the rapidly evolving world of quantum computing. Join leaders in academia, founders, and industry experts in staying informed. Sign up at www.thedailyqubit.com and never miss a qubit.  #quantumcomputing #quantumphysics #innovation

30

1 Comment

In the rapidly evolving landscape of Large Language Models (LLMs), traditional benchmarks like the MMLU (Massive Multitask Language Understanding) have been the go-to for evaluating AI capabilities. However, a growing body of research suggests that relying solely on these scores to select LLMs for specific applications could be misleading and costly. 🔍 𝐖𝐡𝐲 𝐁𝐞𝐧𝐜𝐡𝐦𝐚𝐫𝐤𝐬 𝐌𝐢𝐠𝐡𝐭 𝐌𝐢𝐬𝐥𝐞𝐚𝐝: A recent study, "When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards," underscores a critical vulnerability in the standard benchmarking approach. This research demonstrates how minor methodological changes can significantly shift LLM rankings by up to eight positions on leaderboards. Such volatility raises questions about the stability and reliability of benchmark-based assessments. 👩‍💻 𝐓𝐡𝐞 𝐋𝐢𝐦𝐢𝐭𝐚𝐭𝐢𝐨𝐧𝐬 𝐨𝐟 𝐌𝐂𝐐𝐬: Leaderboards often rely on multiple-choice questions (MCQs) to gauge model performance. While MCQs offer a straightforward and quantifiable way to test certain model capabilities, they fall short in measuring comprehensive performance. The research highlights that models might perform well on MCQs by recognizing patterns rather than genuinely understanding content, which doesn’t necessarily translate to effectiveness in practical applications. 🔄 𝐓𝐡𝐞 𝐬𝐭𝐮𝐝𝐲 𝐜𝐚𝐭𝐞𝐠𝐨𝐫𝐢𝐳𝐞𝐬 𝐚𝐝𝐣𝐮𝐬𝐭𝐦𝐞𝐧𝐭𝐬 𝐢𝐧𝐭𝐨 𝐬𝐞𝐧𝐬𝐢𝐭𝐢𝐯𝐞 𝐚𝐧𝐝 𝐮𝐧𝐬𝐞𝐧𝐬𝐢𝐭𝐢𝐯𝐞 𝐩𝐞𝐫𝐭𝐮𝐫𝐛𝐚𝐭𝐢𝐨𝐧𝐬 𝐭𝐨 𝐞𝐱𝐩𝐥𝐨𝐫𝐞 𝐭𝐡𝐞𝐢𝐫 𝐢𝐦𝐩𝐚𝐜𝐭 𝐨𝐧 𝐦𝐨𝐝𝐞𝐥 𝐫𝐚𝐧𝐤𝐢𝐧𝐠𝐬: - Sensitive Perturbations: Modifications like shuffling choice order, changing symbols, and alternative scoring drastically alter model standings, pointing to a fragility in how performance is currently measured. - Unsensitive Perturbations: Adjustments that tweak the relevance of in-context examples show minor performance variations and help mitigate bias, suggesting a need for more nuanced testing methodologies. 𝐅𝐨𝐫 𝐨𝐫𝐠𝐚𝐧𝐢𝐳𝐚𝐭𝐢𝐨𝐧𝐬 𝐥𝐨𝐨𝐤𝐢𝐧𝐠 𝐭𝐨 𝐢𝐧𝐭𝐞𝐠𝐫𝐚𝐭𝐞 𝐋𝐋𝐌𝐬 𝐢𝐧𝐭𝐨 𝐭𝐡𝐞𝐢𝐫 𝐨𝐩𝐞𝐫𝐚𝐭𝐢𝐨𝐧𝐬, 𝐭𝐡𝐢𝐬 𝐫𝐞𝐬𝐞𝐚𝐫𝐜𝐡 𝐬𝐞𝐫𝐯𝐞𝐬 𝐚𝐬 𝐚 𝐜𝐚𝐮𝐭𝐢𝐨𝐧𝐚𝐫𝐲 𝐭𝐚𝐥𝐞: - Custom Evaluation: Assess LLM performance using your specific data and scenarios rather than relying exclusively on generalized benchmarks. This approach ensures the model aligns with your unique requirements. - Holistic Assessment: Consider a broader range of evaluations that go beyond MCQs to include tests that measure real-world task performance and contextual understanding. - Continuous Benchmarking: As LLM technology evolves, so should the methods we use to evaluate it. Businesses must stay updated on the latest research and adapt their evaluation strategies accordingly. Do you think traditional benchmarks suffice, or is there a need for innovation in evaluation methods? #ai #genai #innovation #technology #llms

21

2 Comments

Quick tip - DDL for vector indexes https://lnkd.in/gwtNtE_r

38

🏅 Large Language Model Leaderboard 🏆 When it comes to evaluating different Large Language Models (LLMs) for your use case, understanding their relative strengths is key. 🏋‍♂️ Here's a concise breakdown: 1️⃣ Multitasking Understanding: Claude 3 Opus from Anthropic and GPT-4 from OpenAI lead in the MMLU benchmark for general knowledge. This makes them strong candidates for projects requiring a wide-ranging understanding. 2️⃣ Arithmetic Reasoning: In arithmetic, Claude 3 Opus scores highest, ideal for tasks that involve math or logical problem-solving. 3️⃣ Coding Proficiency: For coding tasks, Claude 3 Opus and Meta Llama 3 70B show top performance, perfect for software development support. 4️⃣ Context Window: Google's Gemini 1.5 Pro offers the largest context window, able to process extensive data at once, beneficial for complex tasks involving large volumes of text. 5️⃣ Cost-Efficiency: Meta Llama 3 8B stands out for its low input cost per token, offering a budget-friendly option without compromising too much on capability. Choosing the right LLM involves balancing these attributes against your specific requirements. Subscribe to Generative AI with Varun for more insights on LLMs: https://lnkd.in/dNZ4b7ix #GenerativeAI #LLM #OpenAI #Anthropic #Google #Meta

23

3 Comments

Avoid taking leadership advice from extremely lucky companies.

7

1 Comment