Nandhini Santhanam

Ever wonder why some companies can ship software at lightning speeds while others seem stuck in the mud? 🚀⏳ . . . . . . . . . Daksh Gupta of Greptile collected real-world insights from engineers at many of the world's largest tech companies, unpacking the intricacies that either propel or impede the speed of shipping code. The big reveal? Older and larger companies tend to move more slowly because the costs of missteps become greater. In some cases, those costs are fairly non-negotiable (e.g. ensuring software for customers in production remains secure) while in other cases, the costs are more optional and cultural (e.g. wanting a lot of accurate estimates, forcing a manager's vs. maker's schedule, etc). #SoftwareEngineering #Tech #DevelopmentVelocity #Innovation #Productivity

1

A super clear, straight forward example of how to do cost effective RAG using LangChain for your document pipeline and Groq for compute resources. The code demonstrates how to speed up RAG and reduce costs using quantized embeddings (reducing 32 bit values to 8) Nice job Sahibpreet Singh! https://lnkd.in/gf5QMCKK

44

2 Comments

If ChatGPT was AI's "IPhone Moment", by now we're well down into the IPhone 5s moments. Lots of minor tweaks, maybe a new button or missing connector, still lots of hype for some reason, but fundamentally it feels pretty much the same. That said, don't believe the naysayers that see this as the beginning of a decline. I'm still very bullish on AI, it's just that the interesting parts are shifting from who can throw the most compute, and raw data (and legions of fundamental researchers) at the problem to who can make the advances needed to expand the technology to new uses. (Note added at writing, I just realized I'm effectively saying the opposite as what Sam Altman said this week which was "assume it will get way better" instead of find ways to live with current performance. Good. I disagree and the previous deep learning cycle backs me up). All that said, I wanted to share some interesting AI stuff that's on my radar this week that's not about new model releases (mostly): https://www.vals.ai/ Vals.AI is a startup making application specific benchmarks. Good benchmarks and better performance go hand in hand, getting out of theoretical evaluations and in to something practical is going to be the future. (I think the same thing about data: https://lnkd.in/eNepJ7Ys) https://lnkd.in/eHxbBE2e Eq-Bench is another benchmark that I hope is going to help show how silly many modern evaluations are. It achieves great predictive power for how models perform on other tests and in human evaluations by asking them to rate the emotions in dialogue samples. It's a great little test but also has nothing to do with most real applications, and sort of implies that neither do most of the other tests. https://lnkd.in/eFkAXYb7 IPEX LLM by Intel. This has been around for a while but is getting more interesting as Intel releases new GPUs and there is a push to do model inference faster and more efficiently. Check it out as an alternative to Hugging Face for LLM inference. https://lnkd.in/eFJdV_AB Pile-T5 by Eleuther AI. This work takes an old model and an old data set and looks at what happens when you train it for twice as many steps. Most interestingly, they release (if I counted right) 200 checkpoints along the way. This is mostly unheard of for big models and I'm hoping (as are they) it's going be lead to some interesting results related to interpretability, (which despite what you may have heard isn't really possible with LLMs yet). For example https://lnkd.in/eWQZCbu2 So, boring time this week for new mainstream models, lots of other cool stuff going on. Don't give up on AI yet.

14

1 Comment

Work tidbits - Living on the edge! Over a year back, I received my patent in the Knowledge Annotation space after 5 years of wait - it triggered me to join this new team.  The space was/still is AI Accelerators, within Cloud Compute. My team specifically was the ‘last mile’, productionize and land the AI accelerators for the top cloud customers.  The work would have been fine in the last 2+ decades of my career - but these times were different. Everyone and everything was in the AI/ML race. Before we(or others) could plan a new stack - something better was out there - sometimes open sourced! Before anyone could make the perf/$ cheaper - There was an update on new numbers available. There were negotiations on software lifecycles, in favor of velocity.   Customers were eager more than any other time. They were feeling the same stress - best LLM, best Stable diffusion, best MLPerf, best ML stack, best chips, best networking, best price! We did very well overall and that part will continue - despite living on the edge. Recently things shifted and I have gotten time to breathe(joined a sister team), also write my thoughts/tidbits - 1.) A friend, a mentor once said - Go far, not fast! Life is not a race and you have a long way to go! I do.  2.) Build resilience. You will see a lot of ups and downs. You don’t always have time to feel sorry for the down, or cheer for the ups! Keep your emotions in the middle spectrum.  3.) Find the true allies, and keep them close. They will tell you when you are wrong, or not - but most importantly they will stick with you, for long. I just ran into my peer from 9 years ago, 2 companies back in the hallway today!  4.) Recognize that some folks are indeed for the fast, maybe also far.. but mostly fast. Let them be! Take 2 steps back. Build resilience.  5.) Invest in your team - My reports, people are/were the team's biggest strength, and the primary reason to keep us going. We certainly don’t end here. The jobs will always be demanding, and that is by design. But the 'how' is on you..   My Amazon manager (now a VP@Microsoft) wrote me a handwritten note that said - Shivani, slow down and take the time to smell the flowers - I have been practicing something like this lately..  I am taking my learnings and my tribe along - and certainly aiming to go far.  Please take care of yourself, no matter what stage! ##resilience #livingontheedge #farnotfast #machinelearning #google #cloud

62

2 Comments

Stole a few points from Prof Christopher Manning, who talked about LLMs and language modeling in general in the latest TWIML AI Podcast: 📌 Humans acquire language skills in a way very different from LLMs. We need millions of words, compared to billions or even trillions of tokens for LLMs. LLM researchers may want to investigate and learn from how humans acquire language skills. 📌 LLMs cannot reason. However, there are other deep learning models, such as AlphaGo, that can. LLM researchers may want to look into how to integrate that type of reasoning/searching/planning capability into LLMs. 📌 LLMs' world models should enable search and discovery. Although Prof. Manning didn’t call this out explicitly, my understanding is more similar to a knowledge graph type of structure. 📌 Next-gen LLM idea: a soft form of locality and hierarchy. Transformers attend every token to every other token, which is very inefficient, while human language can be modeled by n-grams most of the time. #artificialintelligence #machinelearning #deeplearning https://lnkd.in/euzwMQ6p

84

18 Comments

Hallucinations in LLMs are inevitable, no seriously! Josh Fritzsche and I have worked hard at Voze to find a way to tune, prompt and even plead 🙏 with LLMs to prevent them from the occasional 🍄 hallucination. It seemed like the better we have done the harder the edge cases have been to solve and eventually we realized that they tend to frequently happen when LLMs predict or make anticipations on "nothing". I remember at one point nearly going insane with frustration because when LLMs had a hallucination, they were EPIC! 😱 Then we pushed through it and benefited for the effort! How did we solve this common problem? We didn't (Not completely), we got better at identifying the issues as they happened. Once we identified in the data we were able to start looking for patterns. The same catalysts that cause it to happen once if left unchanged caused it over and over again. Our solution is astoundingly simple. Once you know that it's likely to happen..... let it! Accept it will 🍄, but also be preemptive in not focusing on it and use alternative data sources, generators or providers. Change the conditions that cause it but continue to monitor it as it happens. As Dan Caffee is fond of saying to me on a regular bases. "Dusty we need the system to be adaptive and provide mechanisms to learn from it's past." Well he is right! All we needed was data and the people that contribute to it's quality. Huge props 👏 to strong support from Kathy and Janelles audit teams. We were armed with a knowledge and a desire to learn from past mistakes. Armed with confidence that we could correct these problems we could focus on the data to prevent their impact going forward. When doing millions of LLM (Generative AI) calls regularly Josh and I have learned that hallucinations will happen. We also know exactly how we will handle and prevent it going forward! Failure is not doing anything and accepting defeat.

4

2 Comments

Hey, I'd like to give a shout-out to ClearML for helping us at Vimeo develop AI-based systems. We have been happy customers for a few years now and I'd like to share a bit about how we use ClearML. We use ClearML Datasets to cache a dataset of video transcripts, and run testing loading directly from ClearML Datasets. This allows us to simultaneously speed up the data loading and ensure consistency. Moreover we use ClearML to save our benchmark annotations in one place and track our system performance on the benchmark every-time we run an experiment or change our prompts. ClearML allows us to see the different parameters and prompts that were used for each experiment and to monitor improvements or regressions in our performance. We also use ClearML to run large-scale tests and help with statistical evaluation of our methods. For example, we developed a RAG (Retrieval Augmented Generation) Q&A system and wanted to verify that the LLM will not answer certain questions or user queries that are outside the scope of the video. We used ClearML to collect and analyze the RAG responses on many videos for predefined user queries that were outside the scope of the videos and got good visibility into the performance of the system. Also, the comparison feature on ClearML is great for tracking the improvement of our metrics along the progressing versions of our systems.

29

1 Comment

Excited to be open-Sourcing long context (128K) Llama-3 70B - Llama-3-Giraffe-70B and Giraffe-Instruct From Abacus.AI The biggest issue with the Llama-3 family is that they are fairly small context models. So, even though they are as performant as GPT-4 on critical benchmarks, they can only be used in the real world with a bigger context. We just open-sourced two 128K long-context fine-tunes for Llama-3-70B, which effectively puts it on par with GPT-4 regarding context length support and real-world usage. At Abacus AI, we already use Llama-3 on a few production workloads as a great base model for our customer fine-tunes. Links to both models - https://lnkd.in/gPX_XpD6 https://lnkd.in/gaeEq94t

3

One of the biggest challenges in language models today is making them more interpretable. We often treat AI models as black boxes: data goes in, a response comes out, and the reasoning behind that response remains unclear. I remember an interview with Google CEO, where he was asked to explain how Gemini works. He said he didn’t know. This response resonated with the scientific community, as deep learning often similar to the human brain, but the interviewer was shocked! How can a model, released to millions, be so poorly understood? Two weeks ago, Anthropic released an important paper on model interpretability. They used a technique called "dictionary learning," borrowed from classical ML, which isolates patterns of neuron activations that recur across many different contexts. This paper sheds some light on this important challenge, which, if solved, will create more trust in these models and thus ease the integration of AI into our everyday lives. Highly recommend reading: https://lnkd.in/gPzEePx8

12

1 Comment

My 2nd post on start-ups I met at MIT's "Imagination in Action" conference on June 7th is about code generation and verification. This is a theme I have been exploring lately in my investigations of how we would supervise LLMs that check code generation by other LLMs. 1. coadapt.ai (https://codapt.ai/) lead by Trevor Keith caught my attention because it is working on code generation by focusing on building autonomous web apps. That is a great intersection between exploring autonomy in code generation and commercial applications. We talked about the problem of hallucinations and he described some clever ways he has for reducing them in his work. 2. Nectry co-founded by Adam Chlipala is doing two seemingly disparate things at the same time - allowing non programmers to build software that can be proven to be secure and compliant of its specification. This is done by using Adam's work on formal software verification where he has developed a "legalese" language that non programmers can understand but is, itself, amenable to formal verification. Adam also gave a talk during the conference and I'll share that link when it becomes available as it relates to my own work on the logic of unsupervised evaluation. Adam believes that formal methods may not be a hot topic today, just as Neural Networks once were, but that in the future it will play a greater role in the development of software and AI in general. I agree with that vision. Formal methods are starting to gain traction in ML and AI as seen in the work of Max Tegmark and Steve Omohundro, for example.

3

sharpening writing skill is one of killer skill!

2

Take a look at a Flight Discovery Chatbot that I built few days back to help fellow travelers plan their trip with the power of LLM and Multi-Agentic RAG Autogen Workflow in Python leveraging Natural Language processing. What-if a chatbot can answer your queries and help you in finding cheapest and fastest flights and aid you in planning an efficient trip?That’s what I aimed to do with my this LLM project which leverages Multi-agentic workflow to aid flight discovery. This side weekend project digs into Natural Language Processing (NLP) which harnesses the power of AI to understand, interpret, and generate human language, transforming how we interact with technology. Autonomous agents, equipped with AI capabilities, independently perform tasks and make decisions, enhancing efficiency and scalability across various applications. Last year, Large Language Models (LLMs) were the buzzword, capturing widespread attention for their revolutionary advancements in natural language processing. Today, however, the focus has shifted to multi-agent workflows, which are now at the forefront of innovation, promising enhanced collaboration, efficiency, and scalability in AI-driven solutions. In this blog I share the details of the workflow that I built with components like Autogen, Amadeus Flight API's Developer Free Tier for Developers and Neon Serverless Postgres Database. #AI #LLM #Autogen #Chatbot #Agentic #RAG #technology #AIML #NLP #MachineLearning

32

2 Comments

Considering the rapid advancements in Large Language Models (LLMs), it's becoming clear that for 95% of companies and domains, the underlying LLMs will soon be a commodity provided by major players. The real play for these companies will be at a higher level, focusing on building neural agents and advanced neural planning. Imagine this: planning what needs to be done by neural agents will itself leverage LLMs, becoming your unique edge. For most, fine-tuning an LLM might be necessary, but the true focus should be on higher layers. Think about this: If ChatGPT, Gemini, and Claude are all trained on relatively the same data from the internet, why do they understand and respond to your queries differently? It's their planning skills that give them superpowers. Neural planning will be the research focus for years to come, until it too becomes a commodity offered by the big players.

34

2 Comments

My sense is that NVIDIA's dominance in AI compute is likely to be disrupted 1st in inference (if and when it happens). Mark Zuckerberg made a very interesting point in his conversation with Dwarkesh Patel recently. He said - Meta's ratio of inference to training compute is much higher than most other companies leveraging AI. This is because Meta serves extremely large user communities across Facebook, Instagram, WhatsApp, etc., resulting in huge volumes of inference requests. In contrast, companies focused mainly on research and development may have lower ratios of inference to training compute. The scale of Meta's user base means inference optimization is more important for them in order to efficiently serve real-time requests from billions of users. This was one factor in Meta's decision to develop custom silicon optimized for inference to reduce GPU consumption and free up more expensive GPU resources for training models. I suspect that all the large cos will reduce their dependencies on NVIDIA as they scale their AI inference compute. As of now, the four cos - Amazon Web Services (AWS), Meta, Google, and Microsoft - account for >1/3rd of NVIDIA’s data centre business, and all of them have their custom AI chip initiatives. I don't know what opportunities will be created for early-stage companies in this area, but I will be doing a deep dive here. Let me know who I should be speaking with for these initiatives.

556

35 Comments

🚀 The need for India-focused GenAI platforms has led to a host of indigenous developments. Let's take a look at 8 Made-in-India LLMs that are shaping the country's future. 🔸 Recognizing the socio-cultural gap in existing systems, entrepreneurs and developers from India are working on a GenAI ecosystem tailored for Indian consumption, yielding better contextual outputs across all formats. 🔸 A handful of homegrown conglomerates, such as Reliance (Jio), Tata Consultancy Services, Infosys, and Mahindra & Mahindra, are already working on a slew of GenAI projects. 🔸 Indigenous developments like BharatGPT, Ola Krutrim, and Project Indus (a Tech Mahindra venture including 40 different Indic languages), are addressing the language challenges pan-Indian users face in real life and now in the AI world. According to Inc42 data, India’s GenAI market is expected to grow exponentially in the next few years, surpassing $17 Bn by 2030 from $1.1 Bn in 2023, growing at a CAGR of 48%. With BharatGPT and its ilk taking charge and cementing their positions in India’s conversational AI landscape, India is well-prepared for a new era of digital empowerment where language barriers are dissolved and access to information is democratized. 🌟 Thoughts on this trend: Cultural and Linguistic Relevance: These platforms address India's diverse socio-cultural and linguistic nuances, ensuring inclusivity and effectiveness. Local Innovations and Entrepreneurship: Investments by companies like Reliance, TCS, Infosys, and Mahindra & Mahindra highlight India's growing tech ecosystem and global AI positioning. Economic Growth and Opportunities: The rapid expansion of the GenAI market signifies substantial economic opportunities, leading to job creation and skill development. Digital Empowerment and Accessibility: By providing AI-driven solutions in multiple Indian languages, these platforms enhance digital literacy and inclusivity. Addressing Real-World Challenges: Indigenous AI platforms deliver solutions closely aligned with the needs and preferences of the local population, improving customer service, education, and healthcare support. The emergence of India-focused GenAI platforms is a promising development driving technological, economic, and social progress in the country. Let's look forward to a digitally empowered and inclusive future! 🚀🌐 #GenAI #AI #India #DigitalTransformation #Innovation #Technology #BharatGPT #OlaKrutrim #ProjectIndus #DigitalEmpowerment #FutureTech #AIinIndia

28

3 Comments

How DoorDash uses #LLMs for Data Labeling Currently, Large Language Models have a ton of “FOMO” (fear of missing out) around them, so it can be tempting to dismiss them as another hype-train that’ll fizzle out. There are definitely some aspects that are over-hyped, but it’s also important to note that LLMs have fundamentally changed how we’re building ML models and they’re now being used in NLP tasks far beyond ChatGPT. DoorDash is the largest food delivery service in the US and they let you order items from restaurants, convenience stores, grocery stores and more. Their engineering team published a fantastic blog post about how they use GPT-4 to generate labels and attributes for all the different items they have to list. They talk about the architecture of their system and how they use OpenAI embeddings, GPT-4, LLM agents, retrieval augmented generation and more to power their data labeling.

4

Thats the exact explanation I gave to a group of investors last week. (… it’s just Precision/Recall, attributes vectorization as “embedding”, K-Nearest Neighbor, User Search Intent, DAG, ML algorithms, etc all over again …). LOL and they asked what’s new this time around. And I was like not much, ML is still the backbone, and AI is the inference and actions based on the model. It works better when there is a feedback loop in a closed system. But the current state of the art like ChatGPT is based on scraping the Internet (garbage in, garbage out). Kinda like early search engines. What is a better analogy is that NVIDIA/Snowflake etc are selling tools, aka pickaxes to prospectors. The gold is up to you to find. And the opportunity right now is to monetize industry specific (healthcare, travel, financial services) clean data to build models with (aka a treasure map for the prospectors). From an enterprise perspective, what’s out there today aka generative AI isn’t very useful. Too much hype. To build useful engines will require an effort similar to search revolution 2 decades ago. From

14

3 Comments

Understanding the limitations of LLM solutions is incredibly important so that we can put guardrails in place and design #GenAI applications with realistic, bounded expectations. Thanks Rajiv Shah for highlighting this study!

14

While evaluating biases in multimodal LLMs on Hugging Face, I faced an interesting challenge. Current approaches to achieve fairness in AI models overgeneralise, potentially flattening the cultural nuances that exist across demographics and social groups. We must find ways to mitigate harmful biases in AI without sacrificing the diversity that makes us uniquely human. #ArtificialIntelligence #aibias #diversity

24

1 Comment