Umashanthi Pavalanathan

Open-source AI for evaluating language models Researchers from KAIST AI, LG AI Research, CMU, MIT, Allen Institute for AI, and UIC have introduced Prometheus 2, a groundbreaking open-source evaluator model that assesses language model outputs with accuracy rivaling expensive proprietary solutions like GPT-4. A few key highlights: Prometheus 2 combines models trained on direct assessment and pairwise ranking, enabling it to excel at both common evaluation formats. It leverages the new Preference Collection dataset with 1,000 evaluation criteria to boost performance. Prometheus 2 achieved the highest correlation with human judgments and proprietary models across 8 benchmarks, in some cases reaching 0.90 Pearson correlation and 85%+ accuracy. This helps narrow the gap between open-source and closed-source AI evaluation without sacrificing transparency and control As an AI and Data Strategist, I like that this will make high-quality AI evaluation more accessible and scalable for researchers and practitioners. Some potential strategic impacts: Accelerating AI/ML system development by enabling rapid, reliable evaluation of generated language outputs at lower cost Establishing clearer standards and benchmarks for assessing language model performance as open alternatives mature #ai #machinelearning #nlp #opensource #research #aistrategy Paper - https://lnkd.in/gsnGDwJd

1

Just Arrived in Vienna for #ICML2024, I will be presenting our work on improving GNNs' generalization ability with ensemble learning (https://lnkd.in/gimXe5fS) at Oral 5A (Thu 25 Jul 10:45 a.m. — 11 a.m.) and Poster 2407 (Thu 25 Jul 11:30 a.m. — 1 p.m.). Paper title: From Coarse to Fine: Enable Comprehensive Graph Self-supervised Learning with Multi-granular Semantic Ensemble. A heartfelt thank you to all my collaborators for their incredible support; this work would not have been possible without them. I am also looking forward to reconnecting with old friends and making new ones. If you are attending ICML or are interested in my research, please join me during my presentation session. Additionally, please feel free to reach out to exchange ideas if you are interested in topics such as (1) improving graph model generalization across different datasets, and (2) foundational models for time series forecasting. See you in Vienna! 🇦🇹 #ICML2024 #MachineLearning #ArtificialIntelligence

14

Going from PoC to Production for LLMs? 🤔 In the age of GPTs, chat interfaces are ubiquitous. But what if we add a voice/conversational layer on top? Imagine a C-suite member checking sales on the go or a new employee understanding leave policies through a natural voice interface. Intriguing, right? This got me thinking: how do we move from a PoC running on a local machine to a full-fledged production application? Here's what I found out: 1️⃣ Data Management and Knowledge Distillation: Handle and version your knowledge base (PDFs, Images, Excel, ER Diagrams). Keep data up-to-date and manage new information efficiently. How do you ensure information in the data is not outdated? What’s the best way to encode and chunk this data? The questions arise! 2️⃣ Access control and Citations: How do you ensure people cannot access data they are not supposed to have access to? How do you retrieve specific documents to ensure better answers (LLM models work the best when provided with specific information) and lower bills (Each LLM service charges by the number of tokens)? How do you remove any Personally Identifiable Information (PII)? What kind of caching strategies to use? (Pareto principle, 80% of user queries would be for 20% of the questions that get repeated) 3️⃣ Security and Reproducibility: Protect against bad actors and prompt injection. Create secure networks. How do you safeguard your application? It’s a growing field with evolving challenges. 4️⃣ Heart of the Operations (Sort of, I still think your knowledge base is the heart): Now that you have designed and taken care of the above aspects, there is still more. What size of the model to use? What environment to host it in? how to scale and replicate? (If you have a fine-tuned model). 5️⃣ Generating responses: How do you ensure the results generated are accurate and correct? While it is an active area of research, the only effective solution at this point is humans in the loop. Productionizing LLMs is complex, with more questions than answers. I’d love to hear your experiences and insights. What challenges have you faced, and what solutions have you found? If you can think of important things that I may have missed out, feel free to comment below! Ps. I will be writing and building more on this idea in the coming days, stay tuned for more :) #LLM #AI #MachineLearning #DataScience #VoiceInterface

12

1 Comment

Very insightful work by Anthropic’s interpretability team. And an amazing paper, with outstanding writing and figures. The idea is very simple: interpret LLMs by leveraging sparse autoencoders as surrogate models of the MLP of transformer blocks, which allow one to disambiguate the superposition of features captured by a single neuron. A simple idea, but a very careful and complex execution, as it is often the case in our line of work. The paper goes into many details and provide a large array of insights, although the gist of the implementation remains obfuscated due to the closed source nature of Claude. Too bad, because this is the kind of work that we need to better understand and eventually trust LLMs. This is demonstrated by the authors in the section ‘Influence on Behavior’, where they show that clamping some features to either high or low value during inference is “remarkably effective at modifying model outputs in specific, interpretable ways”. Hopefully this kind of work is going to be replicated and generalized to open-weights models, such that we have new ways to steer their behavior. https://lnkd.in/eVym7f_f #interpretability #xai #explainableai #steerableai #anthropic #claude #anthropic

3

𝗟𝗹𝗮𝗺𝗮 𝟯 𝘂𝗻𝘃𝗲𝗶𝗹 𝗮 𝗳𝘂𝗻𝗱𝗮𝗺𝗲𝗻𝘁𝗮𝗹 𝗰𝗼𝗺𝗽𝘂𝘁𝗶𝗻𝗴 𝘁𝗿𝗮𝗱𝗲𝗼𝗳𝗳 𝘁𝗵𝗮𝘁 𝗳𝗲𝘄 𝗮𝗿𝗲 𝘀𝗲𝗲𝗶𝗻𝗴⬇️ 🦙Llama 3 just came out and to the surprise of many people, the L3-8B version can match the L2-70B one on several benchmarks. Several people also argued that the chinchilla paper (https://lnkd.in/dXaRqyea) was wrong because the llama 3 scaling is very far from chinchilla-optimal. 🤔 In my opinion, this is due to a misunderstanding of what the chinchilla paper is about and more generally of the effects of scale in deep learning. The bitter lesson (https://lnkd.in/d3sVYYgp) has become common knowledge in the field thanks to the GPT family that popularized it. We hear everywhere that "bigger models are better": Yes it's true, but in what sense? Many people have a mental model along the lines of "bigger models are more capable" meaning they would be able to achieve levels of performance that smaller models can't. ❌ This is not true with LLMs and is mainly because we train only once on tokens (most of them at least) and therefore overfitting is not possible. The 𝘳𝘦𝘢𝘭 advantage of bigger models over smaller ones is that they converge *faster*. In other words, you can reach a given level of NLL (modeling performance) with fewer training iterations. ❔ Then a natural question arises: Should I train a smaller model with more or a bigger model with fewer iterations? That's where the chinchilla paper brings a response. They say that the question they are answering is: "Given a fixed FLOPs budget, how should one trade-off model size and the number of training tokens?" (implying "to achieve the best possible NLL"). 🧐 Coming back to Llama 3, it's obvious that the Llama team is aware of this work and there is a reason why they didn't follow this recipe. My guess is that they are taking into account the inference cost in the balance (which is completely out of the scope of the chinchilla paper). When one "overtrains" an 8B model, one ends up with a worse "NLL to training compute" ratio, but also with a better "NLL to inference compute" ratio. ✅ Now if we think about it from a business perspective, it's a very powerful tradeoff because you can choose to buy performance either at a fixed cost (training) or at a variable one (inference). Hope this post helped you! #Llama3 #Meta #LLM #AI

9

3 Comments

"To be great, your product needs to be more than just a thin wrapper around somebody else’s API. But mistakes in the opposite direction can be even more costly." From (https://lnkd.in/dYZicknw) Eugene Yan et al. Serving LLMs on your own is extremely costly from the beginning: The selection of the model, the creation of a dataset for benchmarking, preparing the tools around the model, another set of tests (i.e. load testing, sensitivity, safety...), inference-related optimization/engineering... A good strategy would be trying hard anything that's possible without LLMs! And only if you fail consistently (and you're ready to allocate budget upfront), go down the self-LLM journey

3

1 Comment

I just published Vol. 52 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: �� A Comprehensive Survey of Retrieval-Augmented Large Language Models: Architectures, Training, and Applications, from PolyU 📚 An Analysis Framework for Benchmarking Retrieval-Augmented Generation Systems, from Tu et al. 📚 Distilling Cross-Encoders from Large Language Models for Efficient Passage Re-ranking, from Schlatt et al. 📚 A Comprehensive Benchmark for Assessing Retrieval-Augmented Language Models in Biomedicine, from UMTC 📚 Improving Factual Accuracy of Large Language Models for Healthcare using Targeted Knowledge Graph Retrieval, from Huawei 📚 Identifying Shopping Product Questions in Voice Assistants via Graph Attention Networks, from Amazon 📚 A Learnable Tokenizer for Generative Recommendations, from NUS 📚 An Enhanced Retrieval-Augmented Generation Framework for Accurate, Efficient, and Personalized Language Models, from UTS 📚 Constructing Synthetic Test Collections Using Large Language Models for IR Evaluation, from Rahmani et al. 📚 Enhancing Sequential Recommendation with Soft Contrastive Learning, from Renmin University #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

70

2 Comments

Thats the exact explanation I gave to a group of investors last week. (… it’s just Precision/Recall, attributes vectorization as “embedding”, K-Nearest Neighbor, User Search Intent, DAG, ML algorithms, etc all over again …). LOL and they asked what’s new this time around. And I was like not much, ML is still the backbone, and AI is the inference and actions based on the model. It works better when there is a feedback loop in a closed system. But the current state of the art like ChatGPT is based on scraping the Internet (garbage in, garbage out). Kinda like early search engines. What is a better analogy is that NVIDIA/Snowflake etc are selling tools, aka pickaxes to prospectors. The gold is up to you to find. And the opportunity right now is to monetize industry specific (healthcare, travel, financial services) clean data to build models with (aka a treasure map for the prospectors). From an enterprise perspective, what’s out there today aka generative AI isn’t very useful. Too much hype. To build useful engines will require an effort similar to search revolution 2 decades ago. From

14

3 Comments

I just published Vol. 55 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 A Comprehensive Review of State-of-the-Art Universal Text Embeddings, from Cao et al. 📚 A Survey of Generative Retrieval Techniques, from NTU 📚 An Approach to Structurally Aligning Visual and Textual Embeddings, from Alibaba 📚 Leveraging Dynamic Granularity for Improved Retrieval-Augmented Generation, from Zhong et al. 📚 A Framework for Consistent and Reliable LLM-based Passage Ranking, from OSU 📚 Boosting Sample Efficiency in Recommender Systems through Large Language Model Integration, from Huawei 📚 A Task-Specific Loss Function for Improved Multi-Task Learning Performance in Online Advertising, from Huawei 📚 Robust Interaction-Based Semantic Relevance Calculation for Large-Scale E-commerce Search Engines, from Alibaba 📚 Comparing Auto-Encoding and Auto-Regression Models in Sequential Recommendation, from Wang et al. 📚 An Unsupervised Pipeline for Measuring Retrieval Complexity and Classifying Difficult Questions in QA, from Amazon Alexa AI #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

31

1 Comment

There is so much research going into bringing the power of LLMs into the Enterprise context. The main techniques are k-shot learning or prompt engineering, RAG (Retrieval Augmented Generation) & Model Fine tuning (not to mention many variations of these e.g. Agentic RAG, GraphRAG etc.) However, I came across an interesting take on this from IBM and they are calling it "InstructLab" - where the claim is that they are able to teach the LLM "new skills" grounded in an Enterprise context. On the surface it looks like a k-shot learning + fine turning on steroids. Approach seems to be to utilize a small "skills training dataset" in order to generate a large synthetic dataset (perhaps utilizing a specialized LLM) that is then incorporated into the LLM weights natively. This could be a very interesting concept as long as the cost of adding new skills is not very high (when compared to fine tuning). Exciting times! #genAI #enterpriseAI

40

2 Comments

Two new papers this week with my colleagues at Visa: 1. Measuring machine learning model robustness in production using properties of harmonic functions (https://lnkd.in/gnhbseY6) 2. Measuring LLM hallucination based on same, unsupervised and automated! (https://lnkd.in/geMGX7HV)

33

6 Comments

I just published Vol. 58 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Learnable Late-Interaction Models for Efficient and Accurate Information Retrieval, from Google 📚 Gaming the AI: How Crafted Content Can Hijack LLM-Powered Search Results, from ETH Zurich 📚 Dynamic Text Segmentation for Enhanced Retrieval in RAG Systems, from Duarte et al. 📚 Improving Collaborative Filtering with Simplified Sampled Softmax Cross-Entropy, from Visa 📚 Automating Quality Assessment in Retrieval-Augmented Generation with GPT-4, from Visa 📚 Efficient Listwise Passage Reranking via Embedding Compression, from Liu et al. 📚 Empowering Smaller LLMs in Passage Re-Ranking through Pre-Filtering, from Polytechnique Montréal 📚 Efficient Online Hyper-Parameter Optimization for RAG Systems, from Microsoft 📚 Decomposed and Distilled Large Language Models for Efficient and Accurate Semantic Search, from Ant Group 📚 Accelerating LLM Reranking with Single-Token Logits, from Reddy et al. #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

73

1 Comment

I just published Vol. 51 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 A Model-based Multi-Agent Reinforcement Learning Framework for Industrial Short-Video Recommendation, from Kuaishou 📚 Fast Exact Retrieval for Nearest-Neighbor Lookup in High Dimensions, from Zhu et al. 📚 Leveraging Large Language Models for Automatic Relevance Judgment, from UWaterloo 📚 The Recipe behind Arctic's Retrieval-Focused Text Embeddings, from Snowflake 📚 Accelerating Cross-Encoder k-NN Search via Iterative Query Embedding Refinement, from Google DeepMind 📚 Towards Consistent and Generalizable Pre-Ranking for Billion-Scale E-Commerce Search, from JD[.com] 📚 Breaking the Performance-Storage Tradeoff in Large-Scale Vector Search via Second-Tier Memory, from Cheng et al. 📚 A Method for Improving Retrievers' Sensitivity to Query Intents, from Zhao et al. 📚 Contrastive Aligned Language Model Finetuning for Sequential Recommendation, from Google 📚 Leveraging Large Language Models for Knowledge-Aware Recommendation Systems, form Kuaishou #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

42

1 Comment

So much fun talking about the power of LLMs with Amazon Bedrock and Anthropic Claude. In summary, the ever-evolving landscape of GenAI, it's becoming clear that there won't be a single LLM (Large Language Model) to rule them all. The rise of application-specific LLMs heralds a new era, where various models will carve out niches to excel in. This diversity opens up a wide playing field for innovation and tailored solutions. When it comes to deploying AI models, striking the right balance in evaluation is paramount. We must weigh factors like cost, speed, and precision to ensure optimal performance and resource utilization. Finding this equilibrium ensures that our GenAI solutions are not just efficient but also effective in real-world scenarios. Anthropic Claude emerges as a frontrunner in the realm of LLMs, distinguished by its exceptional capabilities and groundbreaking advancements. Boasting unparalleled scale, efficiency, and adaptability, Anthropic Claude represents a pinnacle of GenAI innovation. Its ability to seamlessly integrate with Amazon Bedrock amplifies its prowess, providing users with an unparalleled ecosystem for knowledge integration and generation. With state-of-the-art architecture and a commitment to pushing the boundaries of what's possible, Anthropic Claude stands as a testament to the transformative potential of GenAI in shaping the future of technology and beyond. Prompt engineering holds the key to maximizing the potential of applications. Crafting prompts that consider persona, context, tags, and positioning can significantly enhance the model's ability to comprehend and generate relevant responses. This strategic approach empowers AI systems to better serve their intended purposes. In the realm of GenAI architectures, a robust knowledge base is essential. Amazon Bedrock stands out as a key component for architectures like RAG (Retrieval-Augmented Generation), providing a solid foundation to fuel intelligent decision-making and content generation. As we march towards AI integration, bridging the gap with legacy applications is crucial. Agents equipped with open APIs offer a seamless way to facilitate communication and interoperability, ensuring a smooth transition into the AI-driven future. Let's continue to explore, innovate, and collaborate as we navigate the exciting frontier of GenAI together! #aws #LanguageModels #Innovation #TechTrends #AIIntegration #webdevcon #claude #bedrock

68

There are a lot of tutorials and learning plans for LLMs and GenAI, especially using AWS Bedrock and Azure Machine Learning. These are super valuable for building things quickly. However, I prefer a more hands-on vendor-agnostic, approach, as it allows me to learn concepts I can apply to any platform. To that end, I've been slowly working through Building a Large Language Model (from Scratch) by Sebastian Rashka, which is part of Manning Publications early access program. It's not like anybody's going to pay me to build one from scratch, but I like knowing how things work at a low-level under the hood. On the one hand, this book is very tutorial-ish so it feels like "now this, now that, now this other thing." On the other hand, just the first chapter or so on tokenizing had me experimenting with tokenizing things that aren't text, because it's easy to play around with code I built myself. https://lnkd.in/gC27QTfF

6

4 Comments

Life as an analytics engineer on Nathan's team is never dull! I highly recommend his team as a great place to work These are some real questions that my team and others have asked them to solve: * what are users' ratings of our transcript quality by region? * do we have users who are probably speaking French but getting transcribed as English? * how many transcribed contact center calls could be processed in one chunk using our LLM (e.g. to generate a summary)? * how often do customers call a contact center and then call back again within 7 days?

9

I just published Vol. 59 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Unifying Context Ranking and Generation in Retrieval-Augmented LLMs, from NVIDIA 📚 A Systematic Approach to Identifying Best Practices in Retrieval-Augmented Generation, from Fudan University 📚 Exploring the Mutual Benefits of LLMs and Search Engine Integration, from IEEE Members 📚 Rethinking Ranking Algorithms for Map Interfaces in Airbnb Search, from Airbnb 📚 Reassessing Context Relevance in LLM Question Answering, from USC 📚 A Benchmark for Evaluating Long-Context Language Models and RAG Systems, from Salesforce AI Research 📚 A New Taxonomy for Long-Context Language Model Evaluation, from Bar-Ilan University 📚 A Multi-Stage Content-Centric Framework for Advancing Retrieval-Augmented Generation, from Tsinghua University 📚 Efficient Document Expansion for E-commerce Search, from Walmart 📚 A Comparative Study of RAG vs. Fine-Tuning in API-Driven Code Generation, from Microsoft #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

67

2 Comments

Ensemble of smaller LLMs (PoLL) is less biased, faster, 7x cheaper when evaluating LLMs performance. Works great on QA & Arena-hard evals arxiv.org/abs/2404.18796

9

Going through the list of papers for #ICML2024 and #ICLR2024 and found some interesting ones that I'll share over the next few days. But here's the big one that got me hooked: Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs Paper: https://lnkd.in/dWkn_d_6 Authors: Yeonhong Park and others. Essentially, the paper proposes: ➡️ Rapid generation of multiple models at different quantization levels → Think about having 2, 3, 4, 5, 6, 7 or 8-bit models always in memory, but with a memory requirement of only an 8-bit model (well, almost, they do need some metadata but its an order of magnitude lower memory requirement). ➡️ Specialized LLM serving engine with the support to serve different quantization levels of the same model. ❓ Why is this important ❓ Imagine the scenario where you have a highly finetuned LLM (or an SLM) that you deploy for your users. As a failsafe, you also need to deploy a secondary LLM that comes into play if the first one goes down. However, as soon as the secondary LLM comes online, the users will notice - it's not a finetuned version of your first model! Or even if you deploy a finetuned version, you end up deploying TWO models, in memory, in parallel. This paper solves this problem - a possible failsafe method for an in-production LLM that failovers to smaller/higher quantization levels based on bandwidth and user requirements. Incredible! For a truly redundant system though this still doesn't work because if the LLM serving engine goes down, all models go down as well. So don't throw away your failsafe LLM just yet, but this might be a handy alternative. Book a free 15 minute call with me: https://lnkd.in/dGnHs_Mn #icml2024 #iclr2024 #llm #nlp #machinelearning #consulting #generativeai #openai #microsoft #iclr #icml #research #paper

12