David Castillo, PhD

Interesting paper; if the interaction between layers is linear, there are possibly new ways of optimizing them. We have just started with these.

3

An apt representation …and that’s true for any technology innovation. However risks are much higher and its impact is much broader in case of ai #aiforgood #artificialintelligence #ai

6

Small models becoming smarter! Phi-3 reinforces the importance of data quality.

24

Very insightful work by Anthropic’s interpretability team. And an amazing paper, with outstanding writing and figures. The idea is very simple: interpret LLMs by leveraging sparse autoencoders as surrogate models of the MLP of transformer blocks, which allow one to disambiguate the superposition of features captured by a single neuron. A simple idea, but a very careful and complex execution, as it is often the case in our line of work. The paper goes into many details and provide a large array of insights, although the gist of the implementation remains obfuscated due to the closed source nature of Claude. Too bad, because this is the kind of work that we need to better understand and eventually trust LLMs. This is demonstrated by the authors in the section ‘Influence on Behavior’, where they show that clamping some features to either high or low value during inference is “remarkably effective at modifying model outputs in specific, interpretable ways”. Hopefully this kind of work is going to be replicated and generalized to open-weights models, such that we have new ways to steer their behavior. https://lnkd.in/eVym7f_f #interpretability #xai #explainableai #steerableai #anthropic #claude #anthropic

3

Fine tuning is probably not a smart strategy to allow LLMs to understand up to date facts, as Ethan indicates in this post with attached paper. However it's also very easy to cause "catastrophic forgetting" by overly fine-tuning on too large a dataset with too many epochs. So you don't do that. Fine tuning isn't a way to get a model to know updated facts. It's a way to get desired response formats easily (more than anything else) I personally have a pretty big problem with the use of huge context windows as a solution to hallucination. Here's an analogy... I give you the Encyclopaedia Britannica and all you want is what's the capital of Zaire. With RAG, you know Zaire is in the last volume near the front, and quickly find the answer. With "large context windows" you scan page by page from the first page, reading every word... Until you find the answer Large context windows as a solution to hallucination are environmental vandalism, as well as a waste of time waiting for a response I believe RAG still is the best solution to most "current knowledge" problems, but some types of questions still aren't easily answered using RAG. Fine tuning can help, by making models more likely to say "I don't know" (as a very simple example). There are many other approaches to improving RAG for specific types of questions and specialised domains. What do you think?

38

2 Comments

If you are developing Gen-AI applications, LLM Agent is the direction in which the future is headed. I cannot advocate more for the LangChain and LangGraph frameworks. Regardless of whether GPT-5 or GPT-10 emerges in the upcoming years, they will not have access to your data if it is protected. Therefore, you will always require your custom agents to bridge the knowledge or functionality gap for the LLMs to do things that you want them to do. The use of agents transforms LLMs into an operating system capable of understanding what, when, and how to invoke tools/apis based on simple natural language commands. Gen-AI is a triangle made up of data, algorithms, and compute infrastructure. Lang Chain tools act as a sophisticated orchestration engine at the heart of this triangle, facilitating the creation of practical Gen-AI applications. A big thank you to Harrison Chase and Andrew Ng for providing this technology and related knowledge into the opensource and open-knowledge domain.

31

Excellent report from General Dynamics Information Technology John P. Curran regarding adoption and trends of AI in the Govt. I used Meta to generate summary below and included link to original article. Main AI use cases are chat and creating/updating computer code. Note also the 23% failure rate of AI projects due to user expectation misalignment. This can be overcome with thorough protoyping and establishing use case metrics. - Successful AI project completion timelines are narrowing, with an average of 14 months from pilot to production. - Future demand for AI projects is building around chat and coding-related functions. - 54% of surveyed AI experts said projects currently under development involve improving computer code. - 59% of projects involve "research and understanding", which may relate to creating chat functions. - 23% of AI projects fail due to misalignment between AI solutions and user expectations. - Partnership efforts are required for successful AI projects. Data diversity is crucial, and incorporating diverse data sources can lead to better AI outcomes. Trends: - AI adoption is increasing across the Federal government, with a growing focus on chat and coding-related functions. - Agencies are leveraging cloud-native services, tooling, and other capabilities to accelerate AI project development. - There is a growing interest in translating older code to newer languages, such as Python or cloud-based analytic tools. - Domain-specific chat functionality is becoming a priority, with many agencies planning to implement chat services in the next two years. Statistics: - 325 AI experts surveyed across the Federal government - 14 months: average time from pilot to production for AI projects - 54%: percentage of projects currently under development that involve improving computer code - 59%: percentage of projects that involve "research and understanding" - 23%: percentage of AI projects that fail due to misalignment between AI solutions and user expectations. Bill Casola John Grim Leslie Anne McIndoe Kim Giffin Michael S. Freeman III Tobias Quaranta https://lnkd.in/eqvmAVFG

4

Check out our latest post on Clappy.ai, "Leverage Layout-Aware RAG for More Accurate Retrieval of Knowledge from Documents." Discover how to harness the power of Retrieval-Augmented Generation (RAG) for documents in PDF, Doc, and Markdown formats. Dive into the specifics of implementation and explore our reference repository to get started with advanced RAG for your knowledge base. Don't miss out on this essential read! Continue reading here for all the details: The Power of Hierarchical Indexing with RAG. (https://lnkd.in/gbiVHfcH)

2

New Mindset Needed for Large Language Models - https://buff.ly/3XbN4vp #AI #ML #LLMs #genAI

An era is ending and most people don't even know it. Zilog announced last call for their venerable Z80 processor after an astounding 48 years of production. "Federico Faggin, an Intel engineer, founded Zilog in 1974 after his work on the Intel 4004, the first 4-bit CPU. The Zilog Z80 was then released in July 1976, conceived as a software-compatible "extension" and enhancement of the Intel 8080 processor." One of my first computers was a Kaypro II which ran on the Zilog Z80 chip. It's been decades since that computer and technology has changed in astounding ways. Trying to compare processing power between a Z80 and a modern Intel or AMD CPU is an exercise in futility the performance differences are so great (for example the Z80 has 8,500 transistors compared to an AMD Ryzen 7 5700G which has 10,700,000,000 transistors - 99.9999% fewer transistors in the Z80). The Z80 was and still is found in all sorts of devices - early 1980's video games (including Pac-Man) contained Z80 CPU's. Other game consoles (Sega's Master System and Game Gear consoles as well as the Genesis) all contained Z80 chips. Graphing calculators from companies such as Texas Instruments use (yes, still use at least as of 2023) the Z80 or a derivative chip. In addition to the fun historical information there is something very informative and interesting about this news. Namely, people tend to get caught up in the latest and greatest CPU's from companies like Intel or AMD (or GPU's from NVidia) but the vast majority of CPU's in the devices we use every day are not at that level. They are older, well tested CPU's that get the job done.They are not the fastest but they don't need to be - they need to provide the functionality required to accomplish whatever the goal of the product is. Sometimes you may need the latest and greatest but more often than not a chip that's been around for almost half a century meets your needs. While the end of the Z80 era was inevitable it does bring back some memories.

6

3 Comments

LoRA — Intuitively and Exhaustively- Gem for Intelligent Document extraction Good to Read!! if you’re willing to invest additional effort into refining your data extraction process, consider fine-tuning your chosen LLM. Methods such as Low Rank Adaptation (LoRA) can be highly effective. By training the LLM on a series of prompt-output pairs consisting of Raw OCR dumps and the output being how you desire the OCR to be formatted, you can significantly enhance its ability to parse OCR dumps and generate more consistent JSON outputs. This fine-tuning can tailor the model to be more adept at understanding and structuring the specific types of data you’re working with. 1) Named Entity Recognition (NER): Task: Identifying and classifying entities (e.g., names, dates, organizations) within documents. LoRA's Role: LoRA allows fine-tuning an LLM on an NER-specific dataset by injecting low-rank matrices into the model’s attention and feed-forward layers. This results in efficient adaptation to recognize entities pertinent to the domain of the documents (e.g., legal, medical). 2) Key-Value Extraction: Task: Extracting pairs of keys and corresponding values from structured or semi-structured documents (e.g., invoices, forms). LoRA's Role: By fine-tuning an LLM with LoRA, the model can be specifically adapted to recognize and extract key-value pairs with high accuracy. The adaptation focuses on the patterns and context specific to key-value pairs in the document layout. 3) Table Extraction: Task: Extracting tabular data, including rows and columns, from documents such as reports and spreadsheets. LoRA's Role: LoRA can fine-tune the LLM to understand the structure and semantics of tables within documents, enabling precise extraction of tabular data while preserving the relationships between data points. Courtesy: https://lnkd.in/dwpGuGFu

24

2 Comments

Excited to be open-Sourcing long context (128K) Llama-3 70B - Llama-3-Giraffe-70B and Giraffe-Instruct From Abacus.AI The biggest issue with the Llama-3 family is that they are fairly small context models. So, even though they are as performant as GPT-4 on critical benchmarks, they can only be used in the real world with a bigger context. We just open-sourced two 128K long-context fine-tunes for Llama-3-70B, which effectively puts it on par with GPT-4 regarding context length support and real-world usage. At Abacus AI, we already use Llama-3 on a few production workloads as a great base model for our customer fine-tunes. Links to both models - https://lnkd.in/gPX_XpD6 https://lnkd.in/gaeEq94t

3

Anthropic has achieved a breakthrough in understanding large language models like Claude 3 Sonnet. They identified how millions of concepts are represented within the model, using dictionary learning to uncover patterns in neuron activation. These features represent a wide array of entities such as cities, people, and scientific fields, and can respond to both text and images in multiple languages. Notably, they also found abstract features related to concepts like inner conflict and logical inconsistencies. By manipulating these features, they could change Claude's behavior, such as making it repeatedly mention the Golden Gate Bridge. This discovery enhances our understanding of AI models and could improve their safety in the future. For more details, visit the articles on Anthropic's website and Transformer Circuits. https://lnkd.in/eAp4TybE #AI #Claude #Anthropic #blackbox

31

2 Comments

Our paper, "QGen: On the Ability to Generalize in Quantization Aware Training," (https://lnkd.in/gXXvS_9Q) is now available on Arxiv. This research delved into the generalization capabilities of quantized models, a relatively unexplored domain, examining them through the lens of sharpness in the loss landscape during optimization. I found the experiments and mathematical proofs particularly fulfilling. We're hopeful for acceptance at the conference to which it's been submitted. 🤞

47

My most recently published podcast episode captures a fun conversation with Marc Planagumà i Valls: "Decentralizing Data: Navigating the Shift to Data Mesh" He's doing some very forward-looking work with data and building a #DataMesh using existing commercial and open source tools; whatever is best for the job. #AddDot (Sorry it took so long to publish, Marc. Sometimes it just gets too busy for anything but the busy parts.) *** Link in comment ***

12

2 Comments

Is it just me, or is anyone else finding it hard to reconcile Sam’s appearance of altruism in this article with the reality that simultaneously, WorldCoin is taking serious advantage of the desperation of populations in places like Argentina and Venezuela by getting them to trade iris scans for a mere $30 USD (which to those populations is like a week’s earnings in the desperation of their fallen economies. ) “Worldcoin is giving anyone that scans their iris 25 WLD tokens, which is roughly equivalent to USD $30 at the current exchange rate. In other words, a person can receive the equivalent of 1 week’s pay just by scanning their iris, in a country where 40% of the population lives in poverty. They don’t want their irises scanned, they are desperate for anything they can earn in USD. Worldcoin is a convenient way to get USD $30.” All the while other countries throughout the globe have banned worldcoin due to privacy concerns. https://lnkd.in/g5Wqfvbq The stated altruistic persona doesn’t exactly seem aligned with this practice.

10

4 Comments

You are seriously underestimating the surprising capabilities of the current LLMs - very Large Language Models based on the Transformer design. Consider Stanford University’s comprehensive “Artificial Intelligence Index Report 2024”: https://lnkd.in/g24NDe7X On page 81, it discusses the State of AI Performance stating “As of 2023, AI has achieved levels of performance that surpass human capabilities across a range of tasks.” “Over the years, AI has surpassed human baselines on a handful of benchmarks, such as image classification in 2015, basic reading comprehension in 2017, visual reasoning in 2020, and natural language inference in 2021. As of 2023, there are still some task categories where AI fails to exceed human ability. These tend to be more complex cognitive tasks, such as visual commonsense reasoning and advanced-level mathematical problem-solving (competition-level math problems).” Then on page 115, the paper discusses the GPQA benchmark: “GPQA: A Graduate-Level Google-Proof Q&A Benchmark. In the last year, researchers from NYU, Anthropic, and Meta introduced the GPQA benchmark to test general multi-subject AI reasoning. This dataset consists of 448 difficult multiple-choice questions that cannot be easily answered by Google searching. The questions were crafted by subject-matter experts in various fields like biology, physics, and chemistry. PhD-level experts achieved a 65% accuracy rate in their respective domains on GPQA, while nonexpert humans scored around 34%. The best-performing AI model, GPT-4, only reached a score of 41.0% on the main test set.” Note, that GPT-4 does better than highly skilled non-expert humans. But this was last year. There is an article published a few days ago that discusses the latest results: https://lnkd.in/g53hdRQR “Last month Anthropic announced that Claude 3 scored just under 60%” (using chain-of-thought prompting and 5 examples).” Note, this is almost at the level of PhD experts “in their respective domains”! And just days ago, OpenAI announced a new version of GPT-4 that they claim outperforms Claude 3 in reasoning abilities - reclaiming the lead its held for the last year.

Yet another sleepless night catching up on GenAI research papers, and my mind is blown! My co-founder Matthew Wallace came across a paper suggesting that LLMs can scale to billions of parameters without using MatMul. arxiv.org/abs/2406.02528 Further it says due to the scaling properties shown MatMul-FREE models would actually out perform current model architectures as they scale even larger. This breakthrough, once proven and adopted, could lead to human-level brain efficiency for AGI. Let’s dive in: Implications: 1. GPUs in their current usage are heavily under-optimized. This breakthrough could lead to a massive increase in their CURRENT capabilities when optimized. Jevon's Paradox !!!! as the increase in efficiency would just lead to more and more usage of current and new GPUs to produce more and more LLM capabilities well ahead of current predictions. 2. CPUs and FPGAs could become far more efficient in training LLMs and even more efficient in inferencing them. Imagine your personal AGI on a USB stick! 3. AMD (Xilinx) and Intel (Altera) can potentially counter the NVDA competition with FPGAs. 4. True real-world real-time inferencing at the edge for nearly any use case becomes possible. First, what is MatMul? MatMul, or matrix multiplication, involves multiplying large matrices, which dominates memory usage and computational resources. This process is why such large clusters of GPUs and extensive memory are needed for training modern LLMs. Reducing or eliminating MatMul could drastically cut down on the hardware requirements and costs associated with training these models. - By using an optimized kernel during inference, memory consumption can be reduced by more than 10× compared to unoptimized models. 🤯 - MatMul-free LLM: Replaces MatMul operations in dense layers with ternary accumulations using weights constrained to {-1, 0, +1}. This reduces computational cost and memory usage while keeping the network effective. - Self-Attention Optimization: Uses a modified GRU, the MatMul-free Linear GRU (MLGRU), which relies only on element-wise products, simplifying the model and enabling parallel computation. - Channel Mixing: Adapts the Gated Linear Unit (GLU) to use BitLinear layers with ternary weights, removing expensive MatMuls while still mixing information effectively. - Hardware Efficiency: Introduces a fused BitLinear layer that optimizes RMSNorm and BitLinear operations, improving training speed by 25.6% and reducing memory usage by 61%. - Experimental Results: Shows the MatMul-free LLM performs competitively with current models on various tasks, with potential to outperform them as it scales. - Custom FPGA Accelerator: Built to leverage the lightweight operations of the MatMul-free LLM, processing billion-parameter models efficiently at 13W—demonstrating human brain-level efficiency and showcasing the potential for brain-like efficiency in future lightweight LLMs. Exciting time to be alive! #5iR KamiwazaAI

14

4 Comments

Today hundreds of thousands of customers are choosing Amazon SageMaker to build, train, and deploy their AI models. Here is a great video from eWEEK on why Amazon SageMaker is a great choice for customers to build high-performing #genAI models.

4