Matthew Katzman

Much has been said about many companies’ desire for more compute (and data) to train large foundation models. I think it’s under-appreciated that we also have nowhere near enough compute available for inference on foundation models. Years ago, when I was leading teams at Google, Baidu, and Stanford that focused on scaling up deep learning, many semiconductor makers, data center operators, and researchers asked me if AI would continue to make good use of more compute if they kept delivering it. For many desktop workloads, like running a web browser, a faster CPU doesn’t help much beyond a certain point. So do we really need faster and faster AI processors? Each time, I confidently replied “yes!” and encouraged them to keep scaling up compute. (Sometimes, I added half-jokingly that I had never met a machine learning engineer who felt like they had enough compute. 😀) Fortunately, this prediction has been right so far. However, beyond training, we are also far from exhausting the benefits of faster and higher volumes of inference. Today, a lot of LLM output is for human consumption. A human might read around 250 words per minute, which is around 6 tokens per second (250 words/min / (0.75 words/token) / (60 secs/min)). So it might seem there’s little value to generating tokens much faster than this. But in an agentic workflow, an LLM might be prompted repeatedly to reflect on and improve its output, use tools, plan and execute multiple steps, or implement multiple agents that collaborate. So, we might generate hundreds of thousands of tokens or more before showing any output to a user. This makes fast token generation very desirable and makes slower generation a bottleneck to taking better advantage of existing models. That’s why I’m excited about the work of companies like Groq, which can generate hundreds of tokens per second. Recently, SambaNova also showed it can hit hundreds of tokens per second. Incidentally, faster, cheaper token generation will also help make running evaluations (evals), which can be slow and expensive today since it involves iterating over many examples, more palatable. Fortunately, both training and inference are rapidly becoming cheaper. I spoke with Cathie Wood and Charles Roberts of the investment firm ARK, which is famous for its bullish predictions on tech. They estimate that AI training costs are falling 75% a year. If they are right, a foundation model that costs $100M to train this year might cost $25M to train next year. Further, they report that for “enterprise scale use cases, inference costs seem to be falling at an annual rate of ~86%, even faster than training costs.” I don’t know how accurate these specific predictions will turn out to be, but with progress in both semiconductors and algorithms, I do see training and inference costs falling rapidly. This will be good for application builders and help AI agentic workflows lift off! [Original text: https://lnkd.in/dJ9tVGh7 ]

2,903

80 Comments

Google DeepMind just dropped #AlphaFold3 [1] and it looks like it's going to be a BIG deal for AI-driven drug design... BUT as Demis Hassabis said in his interview with Bloomberg [2] - the real impact of these technologies will be measured in the clinic and we're at least "a couple of years away." It's a good reminder that the true test of AI for drug development will not be how well models generalize or perform on benchmarks, but how interdisciplinary teams are able leverage them to develop drugs that are safer, more effective and more accessible to patients. The DeepMind team knows this which is why they founded Isomorphic Labs to develop a therapeutic pipeline with Eli Lilly and Company and Novartis. But while we wait to see how those programs progress, we're going to have plenty of exciting results to look out for... 👀 Just last week - the results of Recursion's successful Phase I trial for REC-994 were published in the Journal of the British Pharmacological Society [3], making it one of only a small number of AI-designed drugs to be tested in humans and advance to PhII [4]. While it may not be as exciting as a major new model drop, these are the kinds of milestones we should be celebrating if we want to measure AI progress in patient lives (instead of model parameters :) Big congrats to Chris Gibson, Ron Alfa, Shafique Virani, Tim Considine JD, Tony Donato, Joel Ellis and everyone involved in this work and of course major props to the Isomorphic Labs and Google DeepMind teams who are investing in both technology and clinical translation. Can't wait to see how this changes what we can do for patients. [1] https://lnkd.in/gW4UCddC [2] https://lnkd.in/gfWsgu4n [3] https://lnkd.in/gqKrMC_C [4] https://lnkd.in/gDXkwDyA cc Andrew Dunn, Andrii Buvailo, Patrick Malone, Elliot Hershberg, Leonard Wossnig, Jacob Rinaldi

229

7 Comments

There seems to be a “philosophical” war between Bayesian and Frequentist MMM. I don’t want to discredit other ways of thinking, but I do want to share why Recast is Bayesian. There are a couple of important reasons: 1. The statistical challenge of MMM is squarely in the regime where a Bayesian model will shine. Media mixed models at their heart are time series models, where you have a very limited number of observations. Daily data over 2 years, that’s about 800 steps of data – that’s not that much. But we do have a lot of parameters we want to estimate: if you have 30 channels and you want to estimate their ROI every day and how they change over time, now you have about 25,000 parameters to explain 800 observations. That's way too many parameters to fit way too little data. Now, using priors, you can constrain the model so that it's not really 25,000 parameters. It's closer to somewhere between 75 and 150 effective parameters. That's what priors allows you to do — to impose structure on the model so that it's consistent with other things you may know about, but still get sensible results and make accurate and verifiable predictions. 2. The software to estimate Bayesian models is light years ahead of where it was just 10 years ago. There's been a revolution in Bayesian computation. So platforms like Stan, PyMC3, Turing for Julia, and others like BlackJAX are ways that you can now estimate a Bayesian model with parameters in the tens or even hundreds of thousands of parameters. This is several orders of magnitude more advanced now than what it used to be when you'd be hard-pressed to estimate a model that had several hundred parameters efficiently. That rate of improvement has unlocked Bayesian MMM and it’s one of the reasons why we use this approach. If you want to learn more about how we do things at Recast, check us out here: https://lnkd.in/eUpbB2BR

104

3 Comments

Today Random processes shape science and math: Researchers propose a unified, probabilistic framework by Santa Fe Institute SFI Professor David Wolpert, a physicist, and former SFI Omidyar Fellow David Kinney, a philosopher and cognitive scientist, propose a unified, probabilistic framework to describe math, the physical universe, and even describe how humans reason about both. To illustrate, say you answer a mathematics question now, and in the faraway future, some mathematicians answer the same question. The correctness of your answer depends on how your probabilities match theirs. (In physics, your correctness would involve physical experiments, not future physicists.) Randomness pervades every part of the question-answer process, from question selection to the answer given. This results in an important benefit of the proposed framework: a novel, formal justification for two common-sense shortcuts in science and math—believing more strongly in ideas supported by multiple lines of evidence, and believing more strongly in ideas that best explain something you already believe.

1

This one goes into my scrapbook: > The resulting time-causal limit kernel that obeys this property constitutes a canonical temporal kernel for processing temporal signals in real-time scenarios when the regular Gaussian kernel cannot be used, because of its non-causal access to information from the future, and we cannot additionally require the temporal smoothing process to comprise a complementary memory of the past beyond the information contained in the temporal smoothing process itself, which in this way also serves as a multi-scale temporal memory of the past. Interesting interpretation of the result: > We conclude by stating implications of the theory for modelling temporal phenomena in biological, perceptual, neural and memory processes by mathematical models, as well as implications regarding the philosophy of time and perceptual agents. Specifically, we propose that for A-type theories of time, as well as for perceptual agents, the notion of a non-infinitesimal inner temporal scale of the temporal receptive fields has to be included in representations of the present, where the inherent nonzero temporal delay of such time-causal receptive fields implies a need for incorporating predictions from the actual time-delayed present in the layers of a perceptual hierarchy, to make it possible for a representation of the perceptual present to constitute a representation of the environment with timing properties closer to the actual present. I came to a similar conclusion back in 2018 when I started thinking about distributed computer systems in terms of statistical mechanics. What I discovered then was that knowing the probability for the initial state of each system (independently of all the other systems) was not enough to predict the statistics for how the systems evolve. The missing information that is so critical is the correlation between states which is marginalized over in order to obtain the individual system probably distributions. (To make an analogy back to physics, my first attempt ended up recapitulating in new dress the observation that truncating the BBGKY hierarchy at the equation for a single particle distribution is generally not adequate for modeling interesting physics. If you know what that means you'll understand why I didn't rush to publish this result. 🥱 My particles are just way more complicated—a classical computer is just a whole lotta macrostates with highly constrained interactions.) The point was that I had to add a form of memory to my model. In that case, it took the form of a “network entropy” which changes at the linearization points of each process sending and receiving a message. This change effectively interposed the network as a mediating field for the interactions between processors: there's a linearization point between sender and network, and a (later ordered) one between receiver and network. I essentially introduced a memory of prior interactions to encode the missing correlations. #physics

2

I've just published an in-depth exploration of these questions on Silicon Sands News. https://lnkd.in/ebYFwBGW In this piece, I delve into: - The impact of frontier AI models on investment strategies - Critical factors for AI investment, from technical innovation to ethical considerations - The crucial role of expert general partners in venture capital firms - Future trends shaping the AI investment landscape, including quantum AI and Web3 convergence Whether you're an investor, founder, or AI enthusiast, this article offers valuable insights into the future of AI and how we can responsibly embrace this technological revolution. 🔗 Read the full article here: https://lnkd.in/ebYFwBGW #ArtificialIntelligence #TechInvestment #AIFuture #VentureCapital"

24

1 Comment

Yes I dont want LLMs to always provide a neutral view on every question. Allowing models to mirror diverse characters, instead of consistently targeting a middle ground could enrich the interactivity of conversations. Integrating these character traits into the alignment training data could be a more effective approach than merely crafting system prompts for specific characters, which might still inherently constrain the models to limited viewpoints. Source: https://lnkd.in/eu9YP26u #claude #openai

8

Excellent work on this topic. Also wildly dangerous LLM scaling. https://lnkd.in/gYEiMj9c

30

2 Comments

Going through the list of papers for #ICML2024 and #ICLR2024 and found some interesting ones that I'll share over the next few days. But here's the big one that got me hooked: Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs Paper: https://lnkd.in/dWkn_d_6 Authors: Yeonhong Park and others. Essentially, the paper proposes: ➡️ Rapid generation of multiple models at different quantization levels → Think about having 2, 3, 4, 5, 6, 7 or 8-bit models always in memory, but with a memory requirement of only an 8-bit model (well, almost, they do need some metadata but its an order of magnitude lower memory requirement). ➡️ Specialized LLM serving engine with the support to serve different quantization levels of the same model. ❓ Why is this important ❓ Imagine the scenario where you have a highly finetuned LLM (or an SLM) that you deploy for your users. As a failsafe, you also need to deploy a secondary LLM that comes into play if the first one goes down. However, as soon as the secondary LLM comes online, the users will notice - it's not a finetuned version of your first model! Or even if you deploy a finetuned version, you end up deploying TWO models, in memory, in parallel. This paper solves this problem - a possible failsafe method for an in-production LLM that failovers to smaller/higher quantization levels based on bandwidth and user requirements. Incredible! For a truly redundant system though this still doesn't work because if the LLM serving engine goes down, all models go down as well. So don't throw away your failsafe LLM just yet, but this might be a handy alternative. Book a free 15 minute call with me: https://lnkd.in/dGnHs_Mn #icml2024 #iclr2024 #llm #nlp #machinelearning #consulting #generativeai #openai #microsoft #iclr #icml #research #paper

12

Just a few months ago, we were training on a curated set of genetic research papers from arXiv. These LLMs have been invaluable for our internal research, empowering scientists to understand how different traits and SNP interact for a particular health objective. However, keeping pace with this ever-evolving field presents two key challenges: The data deluge: arXiv churns out thousands of new papers daily, making it impossible to constantly retrain our LLMs with the latest findings. The model release frenzy: New LLM architectures and fine-tuned models appear on platforms like Hugging Face at a dizzying rate. How can we leverage these advancements without starting from scratch every time? How do you solve this problem?: model merging! Imagine this: hundreds of specialized models, some focusing on the exact dataset we need, are readily available on Hugging Face. What if we could combine the strengths of these pre-trained models with our own, leveraging tools like "mergekit"? This powerful tool allows anyone to merge a million-dollar trained model with a specialized one on a regular 16GB CPU machine using sharding. The entire process can take as little as 30 minutes! This opens up a world of possibilities: Reduced Training Costs: Say goodbye to expensive, time-consuming retraining! Merging pre-trained models allows us to incorporate the latest advancements efficiently. Faster Innovation: By leveraging existing models, we can focus on fine-tuning and customization, accelerating our research and development cycles. Democratization of AI: Mergekit empowers researchers of all levels to tap into the power of advanced LLMs without needing access to massive compute resources. The future of LLM development is collaborative and efficient. By embracing model merging, we can unlock the full potential of this technology and accelerate scientific progress! #AI #LLMs #MachineLearning #Research

25

3 Comments

I had a discussion with a Ph.D. in Neuroscience from Princeton back in 2015. It was predictable that deep learning would take over in less than a decade from then, which is sort of happening now. I'm still investigating BCI to improve the human mind to prevent this, just kidding. It's already happening, and there's no stopping it. That's why I propose joining the fun and embracing Gemini, D-WALL, GPT-4, and nobody cared. Just wait until AI fully becomes cryptos. Oh wait, that's been happening since 2018, and I bought them. If only people weren't so slow and their useless brains could be transformed into something useful. Hold on, Neuralink and DARPA have been trying to do this since 2016 and 2011, respectively. If only people had seen that or my article in a Chinese newspaper. The last time I visited the University of Toronto as a guest speaker in 2019, I received questioning looks from the audience about my take on anything revolutionary. That's when I stopped. OpenAI was right in its post. People laugh at revolutionary ideas, then fight against them, question them, and finally join them. Those who have been advocating for at least a decade (publicly, I had it in my notes since 2013) are either silenced or ridiculed. Don't worry, people. You're too late, and there's no one to save you. I'm still frustrated that I couldn't get private access to SORA, but long story short, I recommend only programmers use AI tools for about two years. It's likely that the next evolution will fully automate most tasks that can be done with computers, which is most jobs anyway. It's time to move forward. Otherwise, how can I get my next wave of funding for my BCI startup? Oh, I still need a job in the meantime. People somehow only respond when I get hired by Google, as if that even mattered. I foresee two startups surpassing it in 15 years, and by that time, even if I become an executive, it won't matter because people will have moved on. (this is written by AI obviously, who the hell is gonna take time to write these unimportant and useless things on the social media. when I figure out how to mass produce AI videos, I will use youtube to get enough funding to keep my research so my job hunting can slow the f*** down)

2

Comet's Head of Research, Douglas Blank, has spent decades conducting research on what it means for a computer program to think and how robots can learn. He recently sat down with Harald Carlens to discuss some of his learnings in the fields of cognitive science, developmental robotics, and #ArtificialIntelligence. From (arguably) the first neural network coded in #Python, to what makes #ML competitions valuable, to Douglas Hofstadter’s “Strange Loop,” and whether or not Jupyter notebooks are actually useful, they cover it all. 👉 Read the full article from ML Contests:

28

Very insightful work by Anthropic’s interpretability team. And an amazing paper, with outstanding writing and figures. The idea is very simple: interpret LLMs by leveraging sparse autoencoders as surrogate models of the MLP of transformer blocks, which allow one to disambiguate the superposition of features captured by a single neuron. A simple idea, but a very careful and complex execution, as it is often the case in our line of work. The paper goes into many details and provide a large array of insights, although the gist of the implementation remains obfuscated due to the closed source nature of Claude. Too bad, because this is the kind of work that we need to better understand and eventually trust LLMs. This is demonstrated by the authors in the section ‘Influence on Behavior’, where they show that clamping some features to either high or low value during inference is “remarkably effective at modifying model outputs in specific, interpretable ways”. Hopefully this kind of work is going to be replicated and generalized to open-weights models, such that we have new ways to steer their behavior. https://lnkd.in/eVym7f_f #interpretability #xai #explainableai #steerableai #anthropic #claude #anthropic

3

Massachusetts Institute of Technology let's do an earthshot! The institute plans to clarify it's decarbonization plans for MIT’s Campus by the end of Fall 2024 and I challenge them to be ambitious! Background: Last week a few students got together to write a letter to MIT to be inclusive and ambitious in their goals and I 100% support them. In all my time at MIT folks always an attitude of pushing the envelope, wholly knowing they may not be successful, but that the juice was worth the squeeze. On this issue, MIT has the opportunity to be a leader. The letter, linked below, calls on MIT to aim for an “Earthshot” goal of 2035 for campus decarbonization, to set intermediate milestones, and to be more inclusive of students in climate initiatives. Please consider reading through and signing on if you’d support these goals and being part of the decarbonization efforts on campus! Please read the full letter here: https://lnkd.in/gmRiC9iw If you're affiliated with MIT, consider signing the letter here: https://lnkd.in/gm6Ppc5w

13

I'm going to jump 500 years into the future and talk about the Force and AI, mixing two of my favorite subjects (Star Wars and Computational Intelligence) This paper suggest that it's plausible that humans can experience the Force (Telekinesis and ESP) under some conditions. I say if humans can then so can AI or your toaster oven, or anything for that matter, with effects growing in power into the observable realm, I suspect with the growth in intelligence. Now imagine an ASI that somehow had access to the Force. We'll be all toast. It's not me; it's this paper:

7

8 Comments

🌟 𝗜𝘁𝗲𝗿𝗮𝘁𝗶𝘃𝗲 𝗦𝗲𝗹𝗳-𝗥𝗲𝗳𝗶𝗻𝗲𝗺𝗲𝗻𝘁: 𝗔 𝗛𝘂𝗺𝗮𝗻-𝗜𝗻𝘀𝗽𝗶𝗿𝗲𝗱 𝗔𝗽𝗽𝗿𝗼𝗮𝗰𝗵 𝘁𝗼 𝗘𝗻𝗵𝗮𝗻𝗰𝗶𝗻𝗴 𝗔𝗜 𝗢𝘂𝘁𝗽𝘂𝘁𝘀 Ever felt stuck rewriting an email to get it just right? 📝 Large language models (LLMs) experience similar challenges, struggling to perfect outputs on the first try. This hit home while laboriously tweaking LLM outputs—repetitively generating, questioning ("Is this tone executive-appropriate?"), and refining based on feedback. It was draining for both me and the LLM. Enter "𝗦𝗲𝗹𝗳-𝗥𝗲𝗳𝗶𝗻𝗲," a concept that has me utterly captivated. Why? Because it automates the very process I was wrestling with! Imagine an LLM that can not only generate text, but also analyze it critically and refine it based on its own assessment – all without a single human intervention. This is the magic of Self-Refine: a method that empowers LLMs to become their own editors, constantly striving to improve their work through a human-like iterative process. 🔄 🛠️ 𝗗𝗲𝗹𝘃𝗶𝗻𝗴 𝗶𝗻𝘁𝗼 𝘁𝗵𝗲 𝗧𝗲𝗰𝗵𝗻𝗶𝗰𝗮𝗹 𝗪𝗶𝘇𝗮𝗿𝗱𝗿𝘆 𝗼𝗳 𝗦𝗲𝗹𝗳-𝗥𝗲𝗳𝗶𝗻𝗲 The elegance of Self-Refine lies in its simplicity and effectiveness. Here's a breakdown of the core functionalities that make this human-like iterative process a reality: 🔍 𝗜𝗻𝗶𝘁𝗶𝗮𝗹 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻: The model receives an input and generates an initial response. This could be anything from crafting a creative marketing blurb to writing a line of code. 🎯 𝗦𝗲𝗹𝗳-𝗖𝗿𝗶𝘁𝗶𝗾𝘂𝗲: The model then analyzes its own output, identifying areas for improvement in terms of quality, accuracy, or adherence to specific criteria. Imagine the LLM dissecting its email draft and questioning the clarity of certain sentences or the overall tone. ✍️ 𝗦𝗲𝗹𝗳-𝗥𝗲𝗳𝗶𝗻𝗲𝗺𝗲𝗻𝘁: Leveraging the self-generated critique, the model refines the initial output to enhance its performance. This could involve rewriting sentences, adjusting the technical approach in the code, or tailoring the marketing message for a specific audience. 🔄 𝗜𝘁𝗲𝗿𝗮𝘁𝗶𝘃𝗲 𝗣𝗿𝗼𝗰𝗲𝘀𝘀: This cycle continues until the model determines no further improvement is possible or a predefined number of iterations is reached. Just like revising an email draft until you're satisfied, Self-Refine allows the LLM to iteratively refine its work until it achieves the desired outcome. 💡 𝗕𝗲𝗻𝗲𝗳𝗶𝘁𝘀 𝗮𝗻𝗱 𝗔𝗱𝘃𝗮𝗻𝘁𝗮𝗴𝗲𝘀 𝗼𝗳 𝘁𝗵𝗶𝘀 𝗦𝗲𝗹𝗳-𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗔𝗽𝗽𝗿𝗼𝗮𝗰𝗵 🚀 𝗥𝗲𝗱𝘂𝗰𝗲𝗱 𝗥𝗲𝗹𝗶𝗮𝗻𝗰𝗲 𝗼𝗻 𝗧𝗿𝗮𝗶𝗻𝗶𝗻𝗴 𝗗𝗮𝘁𝗮  📈 𝗗𝗲𝗺𝗼𝗻𝘀𝘁𝗿𝗮𝘁𝗲𝗱 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗚𝗮𝗶𝗻𝘀 🌐 𝗕𝗿𝗼𝗮𝗱 𝗔𝗽𝗽𝗹𝗶𝗰𝗮𝗯𝗶𝗹𝗶𝘁𝘆 As Agentic AI workflows continue to evolve, I believe Self-Refine has the potential to become a cornerstone technology, driving significant advancements in LLM output quality and ultimately contributing to the success of Agentic AI applications. For a deeper dive : https://lnkd.in/g_X3JCGA #AI #ArtificialIntelligence #GenerativeAI #vertexai

8

I was fed up with looking at prompt engineering hacks. I found myself elated with the discovery of DSPy, a new paradigm for prompting developed by Stanford University AI researchers. So, let's see what this magic is about. #AI #ArtificialIntelligence #LLM #LLMs #GPT #ChatGPT #MachineLearning #ML #DeepLearning #DL #DataScience #DataScientist #prompting https://lnkd.in/eSQQayB9

1