Nasrin Mostafazadeh

In computational linguistics, researchers are focused on enhancing language models' ability to extract precise information from extensive textual data. Recent studies introduce "retrieval heads," specialized attention mechanisms that selectively focus on critical segments of large texts, improving targeted data retrieval. These heads, integrated into transformer-based models like LLaMA and Yi, significantly enhance accuracy and efficiency, particularly in handling long-context scenarios. The methodology involves thorough experiments across various models, including the Needle-in-a-Haystack test, which embeds specific information to measure precision. Results show models with active retrieval heads outperform those without, with accuracy dropping notably when these heads are masked. The empirical evidence underscores the effectiveness of retrieval heads in improving information retrieval precision and reliability. This research deepens our understanding of attention mechanisms in large-scale text processing, paving the way for more efficient and accurate language models. https://lnkd.in/dNNPvMXw https://lnkd.in/ddbHX6EA Innovation Hacks AI #aiinnovation #generativeai

1

Sage have created a very nice interactive overview of the median prediction of forecasters on Metaculus and Manifold Markets on various questions related to AI capabilities, harms, and societal impacts. It's well worth checking out if you are interested in this sort of stuff. See: https://lnkd.in/eUgkmzX9 Follow MIT FutureTech if you are interested in technological trends and their social implications. #AI #technology #economics

12

🚨 **Addressing the Reproducibility Crisis in ML-Based Science** 🚨 Another fascinating post from “AI Snake Oil“, where Princeton based Arvind Narayanan and Sayash Kapoor argue that ML-based science is facing a reproducibility crisis. Here's a summary of their findings and proposed solutions: 🔍Key Issues Identified: *Data leakage is a major cause of reproducibility failures. *Complex ML code and lack of standardization complicate performance evaluation. *Overoptimism* and biases in publication contribute to the crisis. 📊 Evidence of the Crisis: *41papers from 30 fields show widespread errors affecting 648 studies. *Common pitfalls include no train-test split, non-independence, and improper feature selection. 🔧 Proposed Solution: *Develop a taxonomy of data leakage to understand and mitigate errors. *Introduce model info sheets to connect ML model performance with scientific claims, enhancing transparency and reproducibility. 💡 Authors demonstrate their observations over an interesting case study: *Civil war prediction studies claiming superior ML model performance were found to be flawed due to data leakage. *According to authors, correcting these errors showed no substantial improvement over traditional models! In summary, Narayanan and Kapoor urge the adoption of best practices and ongoing scrutiny to ensure the reliability of ML-based scientific research. I highly recommend following these two and #AISNAKEOIL #MachineLearning #Reproducibility #DataScience #ResearchIntegrity #DataLeakege https://lnkd.in/dhrCthMm

2

2 Comments

Unveiling Chain-of-Thought Reasoning: Exploring Iterative Algorithms in Language Models Quick read: https://lnkd.in/eEe4EdAN Paper: https://lnkd.in/eb_D4Dk8 AI at Meta

18

CMU Researchers Propose a Distributed Data Scoping Method: Revealing the Incompatibility between the Deep Learning Architecture and the Generic Transport PDEs Researchers from Carnegie Mellon University present a data scoping technique to augment the generalizability of data-driven models forecasting time-dependent physics properties in generic transport issues by disentangling the expressiveness and local dependency of the neural operator. They solve this problem by suggesting a distributed data scoping approach with linear time complexity, strictly constraining information scope to predict local properties. Numerical experiments across various physics domains demonstrate that their data scoping technique significantly hastens training convergence and enhances the benchmark models’ generalizability in extensive engineering simulations. They outline a generic transport system’s domain in d-dimensional space. Introducing a nonlinear operator evolving the system, aiming to approximate it via a neural operator trained using observations from a probability measure. The discretization of functions allows for mesh-independent neural operators in practical computations. The physical information in a generic transport system travels at a limited speed, and they defined the local-dependent operator for the generic transport system. They also clarify how the deep learning structure of neural operators dilutes local dependency. A neural operator comprises layers of linear operators followed by non-linear activations. As layers increase to capture nonlinearity, the local-dependency region expands, potentially conflicting with time-dependent PDEs’ local nature. Instead of limiting the scope of the linear operator to one layer, they directly limit the scope of input data. The data scoping method decomposes the data so that each operator only works on the segmentation. Quick read: https://lnkd.in/g3ERrWzq Paper: https://lnkd.in/g9GeqBDA Machine Learning Department at CMU #ai

19

QoQ and QServe: A New Frontier in Model Quantization Transforming Large Language Model Deployment Researchers from MIT, NVIDIA, UMass Amherst, and MIT-IBM Watson AI Lab introduced the Quattuor-Octo-Quattuor (QoQ) algorithm, a novel approach that refines quantization. This innovative method employs progressive group quantization, which mitigates the accuracy losses typically associated with standard quantization methods. By quantizing weights to an intermediate precision and refining them to the target precision, the QoQ algorithm ensures that all computations are adapted to the capabilities of current-generation GPUs. The QoQ algorithm utilizes a two-stage quantization process. Initially, weights are quantized to 8 bits using per-channel FP16 scales; these intermediates are further quantized to 4 bits. This approach enables General Matrix Multiplication (GEMM) operations on INT8 tensor cores, enhancing computational throughput and reducing latency. The algorithm also incorporates SmoothAttention, a technique that adjusts the quantization of activation keys to optimize performance further. Quick read: https://lnkd.in/gtTBazPJ Paper: https://lnkd.in/gTK_T7Ea Github: https://lnkd.in/gdz9XnB2 #ai #largelanguagemodel #llms

14

https://lnkd.in/eSrt4ey7 "Llama or Large Language Model Meta AI is a family of LLMs introduced by Meta AI in February 2023. The first version of the model was released in four sizes — 7B, 13B, 33B, and 65 billion parameters. Reportedly, the 13B model of Llama outperformed OpenAI’s GPT-3 which had 135 billion parameters. To simplify, parameters here are a measure of the size and complexity of an AI model and generally, a larger number of parameters means an AI model is more complex and powerful." Meta AI at Meta #Llama #AI #LLM #LargeLanguageModel

1

For his debut TDS article, Jeremi Nuer presents a thoughtful walkthrough of a recent paper on model interpretability: "We want to have finer-tuned control over LLMs. Being able to detect when certain features are active, and tune how active they are in the middle of generating output, is an amazing tool to have in our toolbox."

19

New AI Agentic course! Learn to use LangGraph to build single and multi-agent LLM applications in AI Agents in LangGraph. This short course, taught by LangChain founder Harrison Chase and Tavily founder Rotem Weiss, shows how to integrate agentic search to enhance an agent's knowledge with query-focused answers in predictable formats. Also learn to implement agentic memory to save state for reasoning and debugging, and see how human-in-the-loop input can guide agents at key junctures. You'll build an agent from scratch, then reconstruct it with LangGraph to thoroughly understand the framework. Finally, you'll build a sophisticated essay-writing agent that incorporates all the learnings from the course. Sign up here! https://lnkd.in/gsQnySCA

5,197

136 Comments

In a recent synthetic data workshop, our guest co-host and Dir. of Google's AI/ML & GenAI, Ali Arsanjani, PhD, shared insights on the major architectural building blocks for creating production-ready end-to-end GenAI applications. He noted that the old "linear" method of "let's just get the data, tune the model, then serve it" is no longer viable. #GenAI model development demands a more iterative and "interconnected" approach, where you are constantly evaluating, observing, and optimizing each component. Consequently, preparing data for multiple points in this process is crucial. As Ali put it, this GenAI framework means "we can attack the model from any angle". This is where high quality privacy-preserving #syntheticdata can play a key role. As teams often lack access to real data for AI training, the ability to create customizable synthetic datasets that are fit-to-task is "paramount" to this iterative anddynamic process of improving GenAI model performance. ▪ Read more about Ali's GenAI reference architecture: https://lnkd.in/eCVxHhJf ▪ Watch our workshop w/ Ali on Anonymizing Financial Data with a Fine-Tuned SLM: https://lnkd.in/eqje5mcn

25

Aligning Large Language Models with Diverse User Preferences Using Multifaceted System Messages: The JANUS Approach Quick read: https://lnkd.in/gQAsT95M Paper: https://lnkd.in/gkBAuWiC

19

AI Reading for Monday June 3! https://lnkd.in/eh-5mrsF Nvidia builds a platform for robots. - Firstpost Huang: the future is robots building robots. The robot singularity is near! - Reddit video Sam Altman borrows money to make big venture bets that pose a conflict of interest with his work at OpenAI. Bold strategy, Altman. - WSJ Jamie Dimon says JPM is up to 800 use cases for AI. - eFinancialCareers McKinsey: 12m US layoffs due to AI in next 5 years, most affected will be employees like admins, customer service, food service, and manufacturing. - Business Insider Retraining workers to use AI. - FT SEC to hold virtual meeting Thursday about AI on Wall Street. - InvestmentNews Researchers crash Baidu self-driving cars with tinfoil, cardboard and drones. - The Register Doctor says AI helped him step away from the keyboard and rekindle his love for medicine. - LA Times AI matches the quality of human phishing scams. Will need spy-vs-spy AI to keep up. - Bruce Schneier Gurman: big AI updates to Siri might not ship til early 2025; Using Apple's new OLED iPad Pro as a laptop replacement. - Bloomberg X is OK with p***y in bio if it's labelled and Musk gets his cut. Super attractive to mainstream advertisers. - TechCrunch AI safety becomes a political battlefield, and a minefield for Google and Microsoft. - Axios The Daily Caller is big mad that OpenAI doesn't want to license its high quality content. - Futurism Intel has their work cut out for them between ARM and now AMD SOCs, data center CPUs and GPUs. - Tom's Hardware AMD powers laptops with 50 TOPS of NPU and 320 TOPS of RTX4070. Won't match the battery life of the ARM laptops ofc. - Digital Trends Stable Diffusion 3 due in a couple of weeks. - The Register With LinkedInfluencer, ChatGPT will help you create posts like, "what helping an old woman cross the street taught me about B2B sales." Start influencing today! - Reddit

2

Yann LeCun advises students entering the AI field to avoid working on large language models (LLMs). Speaking at VivaTech in Paris, he emphasized that developing next-generation AI systems that overcome LLM limitations would be more impactful. LeCun highlighted the need for innovation in AI architectures beyond LLMs and warned against the dominance of a few AI assistants in controlling digital interactions, stressing the importance of diversity and open-source development in AI. #YannLeCun #AIAdvice #VivaTech #NextGenAI #AIInnovation #BeyondLLMs #AIResearch #OpenSourceAI #AIDiversity #FutureOfAI #TechLeadership

1

Can LLMs reason and plan? In a talk given at Google deep mind Subbarao Kambhampati says “not really” when doing this autonomously and supports his argument with an extensive list of experiments from his team and others. He proposes the LLM-modulo framework where you use external verifiers and solvers, to build on what LLMs can do well, such as guessing plans, guess domain models and elaborate problem specifications. Some other insights worth noting and some warnings -LLMs are expansive system 1 thinking equivalents -having LLMs self verify autonomously is a recipe for compounding errors -if you buy the argument that LLMs can’t plan autonomously, letting them decide plans and execute them by calling external tools/agents is like, quote “leaving a loaded gun in the house with a toddler” -elaborate prompt engineering efforts to get valid plans/reasoning exposes us to the ‘clever Hans’ effect TLDR- Don’t trust, verify and use humans in the loop Link to video in the first comment

8

1 Comment

Kyri Baker joins Farhad Billimoria, Conleigh Byers, Ahlmahz Negash, PhD, and Paul Dockery to discuss the use of AI to solve AC optimal power flow problems faster and better. The discussion starts with explanations on the different forms of artificial intelligence including machine learning, physics informed neural networks, and large language models. You can find Prof. Baker's recent work on emulating AC OPF solvers with neural networks in the show notes. https://lnkd.in/gXYC57V4

7