Y Combinator’s Post

Arrived at that point, when LLM is a choice of cost and speed.

When we first met Daniel, he showed us the first version of Unify - a single API for developers to easily integrate multiple LLMs. It was clear this was a powerful tool for developers, but integrating across multiple LLMs wasn’t a hair on fire problem for companies - yet. At that time, OpenAI had just launched ChatGPT (only a few weeks after W23 started). Companies scrambled to get any AI wrapper to market as fast as they could, and gpt-3.5-turbo was simply the default choice. Plus, there were only a small handful of alternative LLMs to choose from. Today, those companies that started early have become far more sophisticated in their AI tech stack. And there are hundreds of LLMs now, with new models being launched weekly. While some tasks can be solved efficiently by smaller models, others require a larger model like opus, gpt-4o, gemini, etc. For example, you don't need gpt-4 to summarize simple documents if llama-3-8b can do it just as well and is 15x faster and 100x cheaper. But for extracting data from 100+ page PDFs, or for generating the best multilingual translations, cost probably isn’t as important. Choosing the right model for your specific task is harder and with higher stakes than ever. To get this right you need a rigorous and manual testing process that only few companies have the resources to do. But even for these companies, it’s a hard game to play. By the time you’ve defined your test cases, set up your evals, put together a benchmarking framework, and gathered enough data to make a decision, a new model is already launched. Many companies don't even bother trying, and just opt for the latest and largest model, passing on the costs to their users. Unify just launched a next-gen LLM router to help solve this. You focus on building your LLM app, and Unify automatically routes prompts to the right LLM to make sure you're using the best models, with the fastest providers, at the lowest cost. It also provides endpoints for private and self-hosted models, so you can deploy the router in your own infrastructure. Excited to be a part of the ride along with Orange Collective. Congrats to Unify and Daniel Lenton. Keep building 💪

Every AI engineer building LLM apps for prod should practice “Evaluation Driven Development” (EDD) 🧑🔬 ✅ Define eval metrics to target (faithfulness, relevancy etc. but also latency/cost) ✅ Define eval dataset ✅ Try out different approaches, compare against baseline The stochastic nature of LLMs means having a rigorous eval-first approach may be even more important than TDD in traditional software! Wenqi Glantz has a wonderful blog series on using EDD in LlamaIndex to 1) try out different retrieval approaches, and 2) to try out different LLMs and embedding models. Check them out! Retrieval methods: https://lnkd.in/ggCkmCk4 Comparing LLMs (e.g. zephyr-7b): https://lnkd.in/gzazguEA Want to learn more about EDD? We’re hosting a LIVE workshop this Thursday 9am PT. Don’t miss out 👉: https://lu.ma/4xhzyv9n

Evaluation driven development is it!

One of the best ways to use LLMs is to get perspective on use of language. For example, people often don’t know what things mean in an article, video, or, in this case, a joke told via word and image. I didn’t get this joke image at first, and I had to read the comments. Sure enough, gpt4o had no problem. With the advent of tech like VoT, we don’t need APIs to integrate software anymore if we pipe sensory inputs to the LM so it can “see” what’s on-screen or “hear” the audio-out. https://lnkd.in/ekzNVHCH My OS, on a phone or desktop, can relay those inputs and the LM can simply relate the information in its own interface, without integration. Is this already a defined segment of software? It seems like it will be, if not already.

Today OpenAI released the Assistants API! 🤖 The Assistants API allows you to build AI assistants within your own applications. An Assistant has instructions and can leverage models, tools, and knowledge to respond to user queries. The Assistants API currently supports three types of tools: Code Interpreter, Retrieval, and Function calling 🚀

DeepSeek-Coder-V2 might be taking the king’s crown 👑 from Llama 3 70B soon UPDATE: STRIKE THAT, it took the Crown! https://lnkd.in/dj89nitt but Anthropic's Claude 3.5 Sonnet is on its tail, and might be even better in quality. Still crunching the results. Still, worth reading details in the comments of this post, and the follow up post. Lots to learn! - 📈 Shares almost same score (19980) with Claude 3 Opus (19954) a leap over GPT4-o (19236) - 💵 However, FAR CHEAPER: $0.42 vs $90 vs $20 - 🔓 Open-weight and allows commercial use This is an excerpt of the preliminary results of the DevQualityEval v0.5.0 which is testing real world software development tasks and cases. We are evaluating +180 LLMs with different providers and tools. From OpenRouter over Hugging Face to Ollama. Let us know if we are missing a model you want to see evaluated! Deep dive blog post soon 🏇 Some more details in the comments 👇

ok then GPTQ for inference now

🚀 Exciting News from Anthropic: Introducing Claude 2.1! 🚀 🔍 What's New in Claude 2.1? ⬇ *Enhanced Reliability* 📉 Reduced False Statements: 2x fewer than Claude 2.0 📚 Better Accuracy: 30% reduction in incorrect answers 🕵️ Sharper Analysis: 3-4x improved in assessing document support *More Affordable* 💲 Lower Costs: 27% cheaper than Claude 2.0 📈 Exceptional Value: 1M tokens at $16 vs. $30 with GPT-4 Turbo *Advanced Function Calling* 🛠️ Integration Power: Seamlessly integrate with other products and APIs - personally I am really excited about this one 🌐 Web Search Enabled: Access and call custom functions or APIs 📚 Private Knowledge Access: Utilize private knowledge bases effectively *Extended Context Window* 📚 Massive Capacity: Handles about 150k words or 500 pages 🧠 Deep Analysis: Ideal for codebases, financial reports, or entire books 📈 Versatile Applications: Summarization, detailed Q&A, trend forecasting, and more

2023 was the year of GPT Wrappers. 2024 is shaping up to be the year of AI Agents going mainstream. These are networks of LLM prompts/fine-tunes that talk to each other, plan next steps, retrieve relevant context, query external APIs, double check their work, and loop in humans when needed. With that comes a lot of new tooling for scaffolding these AIs, visualization, stepwise evaluation, drift prevention, and continuous improvement pipelines. Backend engineers were the stars last year but a lot of this new stuff is going to require the help of frontend and data engineers to help make build vs buy decisions on all the above internal tooling, setup the required data pipelines, and manage what is essentially maturing into a whole new capability stack. Exciting times.