Hoa Vu

My team and I at Superpowered AI (YC S22) have been working in the RAG space for a little over a year now, and we’ve recently decided to open-source all of our core retrieval tech. spRAG is a retrieval system that’s designed to handle complex real-world queries over dense text, like legal documents and financial reports. As far as we know, it produces the most accurate and reliable results of any RAG system for these kinds of tasks. For example, on FinanceBench, which is an especially challenging open-book financial question answering benchmark, spRAG gets 83% of questions correct, compared to 19% for the vanilla RAG baseline (which uses Chroma + OpenAI Ada embeddings + LangChain). You can find more info about how it works and how to use it in the project’s README. We’re also very open to contributions. We especially need contributions around integrations (i.e. adding support for more vector DBs, embedding models, etc.) and around evaluation. Finally, if you have an unstructured retrieval problem that you’re struggling to build a reliable solution for, feel free to reach out to me and I’d be happy to help and talk through whether spRAG is the right solution or not. #llm #rag #ai #developer https://lnkd.in/gpGcCxb7

12

1 Comment

Today, I found a fascinating way to think about “Tokens” in LLMs. It came from a conversation between Jensen Huang and Patrick Collison at a recent Stripe event. When asked about the future of compute capacity in five years, Jensen smartly dodged that question a bit, but then gave an excellent explanation and analogy about why tokens are the new forces that will power the next decades of humanity. He explained that we're now producing something unprecedented (and at scale): floating point numbers that possess “value”, which we now call tokens. These tokens are valuable because they encapsulate “intelligence”. People are now taking these tokens and transforming them into English, French, images, videos, chemicals, proteins, robotic movements, etc. And many are working hard to expand the range of concepts and ideas we can create with these tokens. He then goes on to made a compelling comparison between tokens and electricity. In the past industrial revolution, we successfully found a way to convert “atoms” into “electrons” (by boiling water to power electricity turbines). And now, we've discovered a way to convert "electrons" into "tokens" (by using energy to power data centers that train and run LLMs). When electricity was first introduced, few people understood its value. Today, paying for kilowatts is routine. The same will happen with tokens. Right now, only early adopters and builders are paying tokens. Soon, everyone will be paying for tokens on a daily basis to supercharge productivity and power new products and services. Many new industries will be born from and built on top of tokens. When I first heard about this way of thinking, I got goosebumps. Perhaps it’s because Jensen has a good storytelling skill that helps him sell what he builds, but I have to admit this way of thinking gave me a brief surge of pride. I'm proud of humanity's collective effort and creativity, which have taken us from living under rocks to building machines that can "think." That's nothing short but a miracle.

18

5 Comments

I have to write outreach emails/messages regularly to potential clients. I wanted to automate the writing process. I could use LLMs to write it for me, but I hate it when it sounds more like a robot and it's even worse when it hallucinates 😞 So I created a RAG-based LLM application (running locally) using LlamaIndex that takes in all my previous emails, proposals, and portfolio projects and generates content that sounds more like me 😎 . To top it off, it often identifies and highlights connections between our previous work and the client's industry (See the demo below for a closer look.) Note: This is just a demo, I haven't provided it with much data yet. I think it can do even better with more personalized content. #LLM #RAGtechnique #AI #outreach #codeclever

26

7 Comments

Its sometimes hard to believe that we live in an era where driverless vehicles aren't just a vision anymore but a reality. Waymo driverless cabs for example, are transforming the San Francisco streets forever! It was a wonderful experience working on an application of autonomous vehicles as part of my final project in the course CS267: Applications of Parallel Computing at the University of California, Berkeley. Our team, comprising Terence N., Zeid Solh, and myself, focused on parallelizing stereo vision for driverless applications. Stereo vision is crucial for self-driving cars as it builds a 3D understanding of the environment by using two cameras. However, real-time processing of these images is computationally expensive, posing a significant challenge for autonomous vehicles. We tackled this by exploring various parallel computing techniques to improve the speed of calculating the Disparity Map algorithm. We used popular techniques such as #SIMD (Single Instruction, Multiple Data), #OpenMP (multi-threading), and CUDA (NVIDIA GPU acceleration), and benchmarked the performance. By distributing the computational load across multiple cores and processors, we achieved substantial performance improvements, enabling real-time processing of high-resolution images and videos. The results were promising! Utilizing CUDA, we saw significant speedups, demonstrating the potential for efficient, real-time depth perception in autonomous systems. This project signified the importance of parallel computing in advancing autonomous driving technologies. I'm incredibly proud of our team's hard work and the insights we gained through this project. Check out our GitHub repo: https://lnkd.in/gYHrEU9z for more details on our implementation! #UCberkeley #CS267 #ParallelComputing #StereoVision #AutonomousDriving #DeepLearning #ComputerVision

50

3 Comments

If you are a data scientist looking to work on the hardest challenges at the edge of implementing AI models in the enterprise. If you believe that AI Agents will make us 100X more productive at our work. If you want to live in a future where an Agent can go through all of your main KPIs and provide you a summary every morning with suggested key actions. If you are convinced that companies should have ultimate freedom and control over the agents and models they deploy - without having to deal with the technical complexities this entails. If you understand that the only way to make this happen lies in helping companies integrate fragmented data, create a semantic layer, and effectively orchestrate LLMs. If you want to work with exceptional leaders like Raul Mosqueda - and you understand the deep technical achievements presented in his demo: Please reach out. Arkham is your home.

19

1 Comment

Jensen: "Most code will become API calls to these powerful LLMs." It really does feel like there are just 2 strategies for building in AI 1) better LM ==> better product 2) better LM ==> product less useful?? At Crosshatch we're all in on (1). We're betting that AI gets faster, cheaper, and more capable. With AI changing so fast, Jeff Bezos' now-famous quote feels important "'What's not going to change in the next 10 years?' [This question] is actually the more important [one] because you can build a business strategy around the things that are stable in time." So with everything changing so fast right now, what's not going to change? - People want to feel known and understood. - We want to save time and money. - We want to feel in control. Excited for AI to help deliver a true hyper-personalized cyberspace.

8

At booxar, we wanted to solve the issue of LLMs hallucinating when people interacted with books. Integration with LlamaIndex.TS helped us address this problem. Now we are able to provide contextual responses to our readers by leveraging a Retrieval-Augmented Generation (RAG) system. It ensures our LLMs have up-to-date, semantically relevant context, which improves the accuracy of our Question and Answer systems and chatbots. LlamaIndex is helping us ingest, structure, and access contextually relevant data. With LlamaIndex.TS, we're enhancing the reading experience by making interactions with books more reliable and informative Check it out at https://beta.booxar.com #booxar #LlamaIndex #RAG #AI #ReadingExperience #ContextualResponses

9

I wrote a blog post on a thorny but underappreciated problem in ML-based marketing personalization: event attribution. It's how you go from the actions you take and customer behavior you observe to creating training experiences for your model. Good attribution makes a huge difference in driving uplift, there are a lot of degrees of freedom in configuring it, and at the same time, it's rarely discussed in ML/bandits/RL literature.

38

4 Comments

PSA: Stop using the original OpenAI CLIP weights. I see a lot of people using CLIP for multi-modal retrieval. What many people may not know is that there are 100’s of CLIP models now since OpenAI’s original release. These newer models offer significantly better performance across many benchmarks. For product search related tasks we recommend the following. + Fastest inference: ViT-B-32 [laion2b_s34b_b79k] + Best balanced: ViT-L-14 [laion2b_s32b_b82k] + Best-all round: xlm-roberta-large-ViT-H-14 [frozen_laion5b_s13b_b90k] You might notice these are not the ones that perform the best on the benchmarks. Why? Because the benchmark only test for very specific tasks which are often quite different from search and recommendations. These are ones we have found to be the best for product related search and retrieval tasks and generally very good first choices if you are unsure where to start (even for non-product related search). Here is a hugging face space with all the data: https://lnkd.in/gxu5UYkt Read more: https://lnkd.in/g5_D4ccs

292

22 Comments

New iterative reasoning technique could improve RAG app accuracy by 50% (vs. Naive RAG)! Another solid example of the rapid innovation cycle on RAG research. BTW, you can quickly find out answers to question like "Why did combining RAG with Fine-Tuning and Iterative Reasoning, such as OODA, improve accuracy by 50 percent?" in Epsilla (YC S23)'s "RAG Research Paper" smart search app at: https://lnkd.in/gVgzpbue , with number of research papers included exceeding 360. #genai #rag #smartsearch #innoation

18

The most robust, reliable, and productive uses of AI are when many models are used in coordination with each other, with "regular" programming mixed in. This leads to more capable, more reliable, and more interpretable AI systems. Most people building with AI already know this. But (unless you work for Google) the main barrier to realizing multi-step AI workloads is an infrastructure one. For the rest of us, we’re left creating an unwieldy mess of chained API calls to multiple providers, with many roundtrips in between. This state of affairs is stifling progress. Nobody wants to deal with more tooling and infrastructure… but everyone would benefit from simple, intuitive interfaces that abstract away a powerful system underneath. Substrate is the first inference API optimized for multi-step AI workloads. With Substrate, you can write less code, run more inference, and build high performance AI applications with zero infrastructure to manage. No tooling, no infrastructure – just elegant abstractions. Excited to support Ben Guo, Rob Cheung and the amazing Substrate team on their mission to democratize how AI applications are built.

40

4 Comments

Very useful insights for folks who are starting to dig further and further into the realm of LLM. An easier GUI approach for fine tuning LLMs.

9

Human in the loop with AI: Best-in-class LLMs are now at ~80th percentile in output quality. That is excellent for first drafts and being better than average, but still not great for delivering a final product of specialized, expert level work. #LLMs #AI #GenAI #futureofwork https://lnkd.in/gpe6BJjx

12

Causal review: "I've tried almost every tool in this space... you guys have solved a couple of things that have been very very difficult for me to solve in other places" — Anik Khan, CEO at MaxRewards One of the best things about running Causal is getting feedback from → savvy users who've spent their careers in Excel, → and have tried other tools in the space, → but didn't find what they needed until they used Causal 🤩 Check out Anik's full thoughts below 👇

37

3 Comments

📣 With this workflow, we enable our users to effectively A/B test between different versions of a prompt and measure the respective accuracy. This empowers developers to make data driven decisions about choosing a prompt when building with LLMs.

6

Meta's latest release of Llama 3 is yet another incredible moment in the state of AI. It's not only best-in-class for open source models but exceeds most other proprietary models as well in performance. This has stunning implications for the ecosystem -- developers, investors, and customers -- when you can have an open source model that performs as well or better than closed approaches. But at a more macro level, what's amazing is that we have now essentially all major tech companies spending in aggregate $100s of billions of dollars on advanced AI R&D and infrastructure, that eventually accrues as value to all developers out there. Given the battle for the major tech platforms to lead in AI, we're going to only continue to see the rate of advancements and leap-frogging continue to accelerate. The implications of this rate of change is to ensure that you have an architecture that lets you eventually take advantage of these improvements over time, and one that anticipates major breakthroughs in the future. Work backwards from a world where models could be 10X higher performance or cheaper than they are today, and build for that world. And ultimately, there's really never been a better time to be in tech than right now.

408

19 Comments

LLMs can take gigabytes of memory to store, which limits what can be run on consumer hardware. But quantization can dramatically compress models, making a wider selection of models available to developers. You can often reduce model size by 4x or more while maintaining reasonable performance. In our new short course Quantization Fundamentals taught by Hugging Face's Younes Belkada and Marc Sun, you'll: - Learn how to quantize nearly any open source model - Use int8 and bfloat16 (Brain float 16) data types to load and run LLMs using PyTorch and the Hugging Face Transformers library - Dive into the technical details of linear quantization to map 32-bit floats to 8-bit integers As models get bigger and bigger, quantization becomes more important for making models practical and accessible. Please check out the course here: https://lnkd.in/g66yNW8W

10,609

143 Comments