Meghan Heintz

I’m honestly excited that AI model development has slowed over the last year. It’s giving us a chance to really explore what can be done with LLMs. example: I wrote earlier about extracting emotion from LLM embeddings https://lnkd.in/ezPZ2W-H example: Some great work around explaining the behavior of LLMs https://lnkd.in/eQMHfTxX Most important, we’re learning all the ins and outs of deploying & running AI in production, and all the quirks that go with that. For a while there, it seemed like everyone was just banking on the next GPT to make everything perfect (kind of like how performance didn’t matter in the ‘00s, we just waited for newer faster CPUs), now it feels like we’re ready to actually get nice things.

4

This is more of a "talking to myself in public" thing than any kind of fully fleshed out hypothesis, so take this with the appropriate grain(s) of salt. But if I were interested in AI and were looking for an angle to explore that might yield something interesting, I'd start working on learning "Multiscale Mathematics" (also known as "Multiscale Modeling"). This is rooted in my personal belief that there has to be a "multi-scale" aspect to intelligence because there is such an aspect inherent in our underlying physical reality.

2

#RAG DOES NOT REDUCE HALLUCINATIONS IN #LLMS: Think I've called this out a few times, but the validation is nice - https://lnkd.in/eAi6zxW2. The article makes two well established points: 1) "Hallucination is a problem of reasoning and not relevance! Any amount of relevant text fed through RAG to a machine will retain the original perplexity and entropy in the system to hallucinate independent of the text"; and 2) "The initial challenge arises from the retrieval phase’s reliance on semantic similarity, which is quantified by inner product spaces or cosine similarities. This metric, while capturing semantic closeness, does not differentiate between factually accurate and inaccurate information." This should be clear to everyone, but the industry still shills flimsy RAG scaffolding anyway (shovels to gold miners).

104

34 Comments

This writeup is a goldmine of knowledge on building with LLMs ✨ My most important takeaway being: “Build eval framework. Can be super basic, but have one” https://applied-llms.org/

35

Today's problem with AI output being unoriginal is our little access to the deeper, higher quality data within the model. For example, don't prompt "write me a lyric about x". Ask an LLM to sit next to a lake, look at a scene and describe it. Then ask it to use the text to compare the scene to how one can deal with loss of a friend in a lyrical structure. The path to access comes is in a domain-specific understanding of prompts. Frameworks behind prompts are going to be the official language of the future to learn. e.g. study the "approach" to how a painting was created or what the stages of writing a paper are, rather than asking AI "write me a paper". Humans are about to become 100x more curious about deeper layers of "how x is made", to be able to speak to AI and capture unique/original output. #LLM #promptengineering #prompts #latentspace

10

1 Comment

I wrote a blog post on a thorny but underappreciated problem in ML-based marketing personalization: event attribution. It's how you go from the actions you take and customer behavior you observe to creating training experiences for your model. Good attribution makes a huge difference in driving uplift, there are a lot of degrees of freedom in configuring it, and at the same time, it's rarely discussed in ML/bandits/RL literature.

38

4 Comments

Pydantic is sponsoring Frank Hoffmann because inline-snapshots is transformative to how we write unit tests in Python! https://lnkd.in/en6HtXCF If you're using pytest, and you're not using inline-snapshots, you're wasting your time; simple as that.

336

19 Comments

For many of us that don't have the time or attention span to a read miriad of research papers to stay up to date on rapidly changing world of AI - you can now listen to summaries as a podcast:-) #airevolution #artificialintelligence #largelanguagemodels

46

1 Comment

Very useful insights for folks who are starting to dig further and further into the realm of LLM. An easier GUI approach for fine tuning LLMs.

9

This new little product demo is awesome! Our new Rev AI Transcript Assistant will save you TONS of time. Easily pull out nuggets of info from your transcripts, making valuable content more accessible than ever. https://lnkd.in/gzzFpcDa #transcription #asr #accessibility #transcript #summarization #ai #assistant

31

2 Comments

𝒀𝒐𝒈𝒂 𝑨𝒔𝒔𝒊𝒔𝒕𝒂𝒏𝒕 𝑪𝒉𝒂𝒕𝒃𝒐𝒕 𝑷𝒂𝒓𝒕 2: 𝑭𝒊𝒏𝒆-𝒕𝒖𝒏𝒊𝒏𝒈 𝑳𝑳𝑴𝒔 Although I initially planned to update this series weekly, it seems more realistic to do so every 2-3 weeks 😬. In Part 1, I introduced how to fine-tune image models effectively in 2024. Now, in Part 2, I'm excited to cover the fine-tuning of LLMs, using the recent release of Llama v3 with the torchtune framework as an example. We've got a fun mix of coding tutorials, data prep tips, and some in-depth explanations. Grab a cup of coffee, get comfy, and check it out! Github: https://lnkd.in/g5Wg3zXr 瑜伽聊天機器人系列第二篇，介紹如何準備自己的資料集來微調LLMs，同時整理了常見的微調方法、原理與好用的Libraries，完整中文內容可以參考： Medium: https://lnkd.in/gn9tRmdg

11

4 Comments

Good post and thread by Barak Turovsky. #RAG is just flimsy scaffolding thrown up hastily to enable developers to experiment with #GenAI. It's pretty ironic that Deep Learning was mated with manually scripted pipelines with static encoded DSLs to derive "context" from unstructured documents for one-off siloed applications. As you suggest, #GraphRAG is more work with nominal improvement and questionable ROI. Extending RAG just creates more convoluted Rube Goldberg machines. In most cases, simplistic graphs are being used to help with tagging/chunking docs - again fairly primitive stuff. The answer for GenAI and AI generally is graph, but true ontologies, for #NeuroSymbolicAI. In a multi-model, hybrid AI world, an #LLM is just an element to be managed as part of a broader solution architecture with an #ontology providing #domainknowledge as context for one-many use cases. The business needs an #abstraction layer between AI, GenAI and analytics, and its information sources and its capabilities. I refer to this as a "Domain-oriented #GenerativeAI Architecture". https://lnkd.in/ekQD3Sgw

8

"OpenAI basically set back progress towards AGI by probably 5 to 10 years. They caused this complete closing down of frontier research publishing. And now LLMs have sucked the oxygen out of the room" Provocative soundbite from François Chollet on Dwarkesh Patel's podcast. The increasing secrecy of frontier research has been a big disappointment - it is the opposite of the wonderful open source code & data momentum that has been steadily increasing over the years. On the other hand, the increased AI hype should bring more clever minds to the field, who spread their knowledge as they move employers or found their own businesses, and I would expect at least some of the capital fuelling the genAI boom will trickle into more AGI-like endeavours.

28

2 Comments

Great example that shows 1) One cap doesn't fit all. You don't need a single LLM for everything 2) With fine tuning you can get better performance from a small model than a huge LLM with trillions of hyperparameters. Result: more efficiency and lower TCO!

9

Large language models (LLMs) are known to generate plausible sounding factually incorrect responses, especially when faced with unfamiliar concepts, queries. Recent research suggests that strategic fine-tuning and in-context learning, retrieval augmentation can significantly reduce these hallucinations. #llms #chatgpt4 #llmops #aiml #gemini #phi #mlops https://lnkd.in/duRrC3yg

4

Great work complementing benchmarks which focus on output quality. Speed and cost effectiveness are equally crucial in productization to deliver customer value and business value!

7

We need more research in this direction especially for cultural adaptation of LLMs.

12

Sparse Autoencoders (SAEs) are one of the most promising and popular methods for LLM interpretability and explainability. I wrote a simple, intuitive introduction to SAEs, complete with diagrams and reference PyTorch code. Among other things, I cover why we use SAEs, how they are used for interventions, and challenges with using SAEs. https://lnkd.in/gzrvi6Kg

13