Francisco Marco-Serrano

One library was missing on the list. It leverages constrained grammar sampling from llama.cpp to produce structured outputs with any LLMs... https://lnkd.in/etv3QWH4

The Benchmarking Challenge ‼ As generative models like LLMs demonstrate remarkable capabilities in language understanding and complex reasoning, the need for evolving our evaluation methods becomes crucial. How do we effectively measure the progress of these increasingly sophisticated systems? Comprehensive benchmarking of LLMs at scale presents obstacles that must be overcome: 1️⃣ Cost Efficiency: Large benchmarks demand substantial energy and compute. How can we make benchmarking faster and more affordable without compromising quality? 2️⃣ Evaluating Complex Tasks: LLMs excel at multi-task, open-ended problems beyond traditional narrow benchmarks. How do we assess these broader capabilities? 3️⃣ Specialized Domain Performance: As LLMs advance in areas like code generation, how can AI community ensure benchmarks stay relevant and accurately assess evolving performance in specialized tasks? While benchmarks play a crucial role in evaluating AI systems, it's important to remember that they aren't the be-all and end-all of GenAI progress. They're an important factor in assessing AI capabilities, but not the only one. ❗ They're like academic grades in the real world - important, but not everything. We say "grades don't define you," yet we can't help comparing them. Similarly, the AI community both critiques and relies on benchmarks. 🔑 Real-world application, ethical considerations, and adaptability to diverse scenarios are equally crucial in determining #GenAI system's true value and impact. What are your perspectives on this rapidly evolving landscape? Keen to hear from researchers, developers and leaders tackling these issues.

5

1 Comment

Apparently, someone (maybe KPMG) is running a supply chain due diligence on pydantic on behalf of NASA. The thread is interesting. I link in the comment, but I summarise in a couple of points: - "This is so dumb. Do they know what OSS is?" Dumbness is not an issue in regulations. There was literally a supply chain attack recently on Linux, and if successful, a state actor would have gained access to half the internet (which makes you wonder what else is happening out there). You can read more about this if you search "hacker news xz," which is a fascinating story; I might write about it. - "KPMG ripping of NASA and the taxpayers" Maybe the astronauts and the rocket scientists don't want to spend their time on this? Outsourcing these are pretty standard to those poor people whose entire job is just this. (Shed a tear for their souls) - "Can you do it for free for NASA?" Why would he? They have an enormous budget. Pydantic must have a good place for that money. - "Did not OSS maintainers suffer enough?" They can ignore the request, and NASA will have no other choice but to either fork pydantic or recreate it from scratch. They have no other choice. They are not doing this because they have nothing better to do. - "AI will replace these people, can't wait for it." That's just mean. - "This is so dumb." They plan to send people to the Moon, not exactly a "move fast and break things" mission. Look at what happened when this principle was ignored at Boeing. NASA wants to ensure that if there is a bug or issue with pydantic, someone will be there to fix it. They also want to ensure that no nefarious actors have a chance to take down anything serious or at least have a contained blast radius. This is a negotiation. How much the maintainer can charge for this? What is the alternative? Recreate pydantic from scratch? How many lines of code? How many engineers and how much time to recreate? How much time does it take to review a fork and maintain compatibility? This will provide a BATNA ("Best Alternative to Negotiated Agreement) that you can use as a baseline. On the other hand, you have your costs to actually fulfil the above agreement. Some quoted lowball numbers, which indicate that the entire deal can be done in 1-2 weeks. It won't, and you will regret accepting the deal. There is a huge crisis in OSS, especially with single-person maintainers. Huge companies rely on these softwares, but because there is no business behind them, the individual probably can't or doesn't want to deal with the commercial side. They started doing it in their spare time and now things go out of hand. [continued in comments]

90

40 Comments

🎉 PyTorch just made available the alpha release of ExecuTorch library. 🦋 What it does: 🎈 Helps deploy Large Language Models (LLMs) on edge devices (including mobiles) 👉 Since, mobile devices are highly constrained for compute, memory, and power, this introduces a considerable amount of challenge in bring the LLMs to such devices. To pack these models appropriately, the PyTorch team heavily leveraged quantization and other techniques. 🎁 What it offers/supports: 👉 supports 4-bit post-training quantization using GPTQ 👉 production tested, ExecuTorch is used by Meta for hand tracking on Meta Quest 3 and a variety of models on Ray-Ban Meta Smart Glasses 👉 supports Llama 2 7B efficiently on iPhone 15 Pro, iPhone 15 Pro Max, Samsung Galaxy S22, S23, and S24 phones and other edge devices 👉 early support for Llama 3 8B 👉 broad device support on CPU via dynamic shape support and new dtypes in XNNPack 🚀 Blog post: https://lnkd.in/gTFxdDhP 👉 GitHub: https://lnkd.in/gN-Yjj4A 👉 Docs: https://lnkd.in/gVD-2Tzt #LLMs #edgedevice #deploment #ml #pylib #executorch #pytorch

8

Insightful work applicable to augmenting clear thinking decision making that is robustly efficient. Could also be said it is an “Interpretation of how metric results can be used for characterizing the causal context, thus informing risk “performance” management decision-making”. Look forward to the further work of linking CaNeTA with dynamic risk analysis models to illustrate the effectiveness of risk/performance assessment strategies.

3

Organisations have faced challenges in operationalising machine learning models for years. With the rise of LLMs, the stakes are even higher! Many organisations are under pressure to launch these complex models into production, even whilst they continue to struggle with smaller, simpler ones. What does this mean? The spotlight is finally on ML/AI engineering, a role that’s been flying under the radar… until now. #data #ai #ml #engineering

21

5 Comments

Is it possible to find valid instrumental variables (IV) with ML? Martin Huber and co-authors present a framework to do so. This novel approach is an interesting mix between causal inference, ML and causal discovery (relies as well on conditional independence (Aleksander Molak, what do you think?)). It took me three years to find a valid and strong IV to measure the effect of weapons imports on conflict during my Ph.D. I discussed with numerous various experts from weapons trade, to production etc. I even went to a shooting range to familiarize myself with AK-type weapons and more and discussed with arms enthusiasts to try to find an idea. Anyone who works with IV knows how tough it is. Hence, this is a very important step. Of course, there is no silver bullet with causal inference. As Martin told me, you need a lot of data (usual with ML), and some assumptions on the causal structure (usual with causal inf.). I still have to read it more carefully and most importantly test the method on my own. What do you think? Will we need less and less "domain knowledge" in causal inference due to causal ML or not? ------------------- 🧠I help you go from intuition and correlational analysis to causal analysis, leading to effective data-driven decisions. 🤝Connect with me to learn the causal mindset through my daily posts. 𝐓𝐡𝐢𝐧𝐤 𝐂𝐚𝐮𝐬𝐚𝐥𝐥𝐲, 𝐀𝐜𝐭 𝐖𝐢𝐬𝐞𝐥𝐲 #causalinference #causality #ai #ml #causalai #causalml #statistics #econometrics #datascience #causaldiscovery

144

13 Comments

"Ai is two markets, training, and inference. Inference is going to be 100 times bigger than training." On his All In Podcast, major investor Chamath Palihapitiya, known for his investments in AMD and Groq, discussed several key topics related to Ai development. One of which was Meta's "overspending" on the training of its Llama 3 and the growing impact of open-source developments in the LLM marketplace- which was stated as "catching up to that of OpenAI" Any thoughts on the potential impact of open-source advancements in the AI industry?

1

We have two excellent tools to parse PDFs: LlamaParse and PyPDF - built with LLMs and for LLM use cases. But which one is supreme? LlamaParse excels in: Structured Data Extraction: If you need to extract precise data points from complex PDFs, such as financial files, LlamaParse is probably better. It appears to be intended for tasks such as table recognition and data validation. Consider tables full of numbers; LlamaParse delves into, analyses them, and extracts the important information you require. PyPDF excels at: Basic PDF Manipulation: Pypdf is a suitable solution if you only need to read text content, navigate pages, and conduct basic PDF structure modifications. Need a quick text snippet or want to browse a digital document? Pypdf is your go-to. So, Who Wins? Essentially, consider: - Complexity of Data: If your PDFs have complicated layouts and require organized data extraction, use LlamaParse. - If you only require basic functionality, such as reading text, use pypdf. Comment below if you've tried something else and believe it's better than these two. I would like to do more exploring. #LlamaParse #PyPDF #LLM #pdfParser

34

4 Comments

AI Reading for Friday May 3! Is OpenAI about to launch a Google (Perplexity really) competitor next week? And, using ChatGPT for academic research. https://lnkd.in/e-eAAv2u Will OpenAI launch a Perplexity/Google competitor? And is Google vulnerable, with search quality having taken a back seat to revenue extraction? - BGR AI chatbot for consensus academic research - Consensus.app This one is in the GPT Store, uses an external RAG engine to summarize academic papers - chat.openai.com This one is for doctors, asked me for an NPI number. Give me free medical advice you cowards! - OpenEvidence Let us meticulously delve into the intricacy of using AI to write research papers. - The Register Joanna Stern takes on the Human AI pin and Rabbit R1, finds them to be science projects more than useful companions, Meta AR glasses somewhat better. - WSJ Going 'dark and deep' on job applicants with facial recognition and AI. No way there could be false positives or hallucinations, right? Welcome to the Orwellian future. - NewsNation Automate your job search with AI, then go down to the pub. - Cult of Mac Chinese unicorns vie to be China's OpenAI. - FT Apple hypes AI plans, quarter not as bad as expected, stock pops $10. - Yahoo Finance Microsoft bans using Azure for services helping US police departments implement facial recognition, possibly because Axon was doing that. - TechCrunch Using ChatGPT to plan your wedding. Or 'Chat GBT'. - Peoplemag Using Mindtrip AI to plan your vacation travel itineraries. - Mindtrip.com

3

1 Comment

Parsing structured data from LLMs can be frustrating for anything beyond toy problems. I recently compared three frameworks: Instructor, Fructose, and Langchain, on three problems that are more complex than the normal toy examples in tutorials. https://lnkd.in/eJCNndyj 1. Extreme Multilabel Classification with 60 classes 2. 3-level deeply nested Named Entity Recognition 3. 3-level deeply nested Synthetic Data Generation Recently, Khuyen Tran shared about Mirascope, a relatively new framework for structured data parsing: https://lnkd.in/eE6MssQ6, so I had to test it out using the same methodology I had used to crown Fructose as the best out-of-the-box framework for these tasks. Unfortunately, Mirascope did not perform as well as Fructose. Details below. Following the documentation on the respective frameworks, I ran 20 runs of the same input and collected some metrics, as shown in the picture. - Instructor: Has concise code, uses Pydantic for data structure definitions, has good latencies, and is excellent for extreme multilabel classification tasks. - Mirascope: Has slightly more verbose code but not by much, and it also uses Pydantic for data structure definitions. However, it has comparable or worse accuracy than Instructor with generally worse latency. - Fructose: Has concise code, generally good latencies (except for synthetic data generation) and is excellent for all three complex tasks. However, it uses Dataclasses, which I’m not a big fan of. - Langchain: I want to love Langchain, but it struggles out of the box with more complex problems like nested fields. Also, debugging is a challenge with LCEL, as I couldn’t figure out how to place breakpoints. The documentation is also vague and broken at times. In conclusion, ✅ Fructose is still the most well-rounded out-of-the-box option for Multilabel Classification, deeply-nested NER, or Synthetic data generation. ✅ Instructor is good, too, if you want to use Pydantic instead of Dataclasses. ❌ Mirascope is comparable to or worse than Instructor for these tasks, so I’d stick to Instructor if I needed to use Pydantic instead of the Dataclasses that Fructose requires. ❌ I don’t recommend Langchain for these tasks. 🔗 Medium friend link to all the code and evaluation methodology: https://lnkd.in/gUcWrZNx 🌟 Fructose Github: https://lnkd.in/gadEKyNZ 🌟 Instructor Github: https://lnkd.in/g__dT2Gr 🌟 Mirascope Github: https://lnkd.in/efp9FDNH Follow me for more tips on building successful ML and LLM products! Medium: https://lnkd.in/g2jAJn5 X: https://lnkd.in/g_JbKEkM #generativeai #llm #nlp #artificialintelligence #mlops #llmops

147

19 Comments

As large language models (LLMs) continue to advance and find applications across various industries, it's encouraging to see the community embracing the BOLD benchmark (https://lnkd.in/gcwwGqsy) as a standard for evaluating biases in these powerful AI systems. Responsible AI practices, including rigorous bias assessments, are crucial as we increasingly rely on LLMs for diverse tasks and build autonomous LLM agents. By systematically measuring and mitigating biases, we can work towards building more fair and ethical AI systems that benefit society as a whole.

20

Notebook on instruction fine tuning of LLMs. #LLMs #finetune #instruct #tutorial #notebook #python

18

🐣 What is the probability of A given B has happened? This kind of conditional thinking is at the core of Bayesian Problem statement. There are many such real world scenarios that could be modeled using Bayesian thinking. 🤗 PyMC is a Python library that allows you to conveniently express such probabilistic problems in a programming construct. ⚡️PyMC 👉 GitHub: https://lnkd.in/gVj-eX-S 👉 Docs: https://lnkd.in/gfJJczQg 👉 Intro to PyMC 4.0: https://lnkd.in/gqmcbwf4 #python #pymc #bayesian #ml

5

AI: the bubble that burst What’s wrong with the current unscientific AI. Let’s summarize the WSJ article: * Uncertain return to investors * Unrealistic goals * Not delivering on its promises * Narrow set of practical applications * Many model variations of the same algorithm * Taking a long time, or never, to realize the business case * Spending does not guarantee a return * Pace of improvement slowing, or too many errors * Imperfect data; data being appropriated from others; * Open source options catching up with paywall models * Excessively expensive to run and maintain for meager results * Public starting to realize of its limits: self-driving cars fatal accidents; poor search results; lawsuits by owners of the data; driven by personalities not science; increasingly taking more electrical power from the grid Finally, at the end of the article, WSJ closes with: “The problem is that the current level of investment—in startups and by big companies—seems to be predicated on the idea that AI is going to get so much better, so fast, and be adopted so quickly that its impact on our lives and the economy is hard to comprehend. Mounting evidence suggests that won’t be the case.” Source: The Wall Street Journal https://lnkd.in/gg-7hJjN #artificialIntelligence #ai #machineLearning #dataScience #spe #petroleumEngineering

155

61 Comments

Just found an excellent Towards Data Science article by Mariya Mansurova on Medium about causal analysis with controlled experiments and observational data: https://lnkd.in/e2fq76hZ A few highlights: 1️⃣ The treatment effect size and confidence determined with classical (frequentist) A/B/n tests are equal to the respective coefficient value and significance in a linear regression 2️⃣ Adding more independent variables to the linear regression increases confidence in the treatment (assuming more of the observed residual variance can be explained by those variables) 3️⃣ We can reduce bias from confounding variables (that impact the treatment and our target metric) by also adding them as features to the linear regression 4️⃣ When applying the above to observational data, we can come to the same causal conclusions if we include all confounders (of course this won't be entirely feasible and as much art as science in a lot of cases) As a bonus, Mariya shares her Python notebook code to reproduce the described analysis incl. many beautiful graphs. #experimentation #causalinference #linearregression #statistics #machinelearning

18

1 Comment

Interesting early approach by Anthropic for reducing transformers models complexity, helping with interpretability and (consequently) reducing hallucinations and errors #GenerativeAI #largelanguagemodels #artificialintelligence #airevolution

53

1 Comment