John Hegeman

The afternoon Keynote @ Stripe Sessions was Fireside Chat between Patrick Collison (CEO of Stripe) and Jensen Huang (CEO of NVIDIA). Patrick was an excellent moderator — asked Jensen really probing, nuanced questions that Jensen answered with his usual sharp wit and unique (and impressively well rationalized) perspective. A couple of quotes among many that stuck with me: - “Current CPU installed market is $1 trillion of hardware, all of which will be upgraded to GPUs over the next 5 years. And I would estimate about another $1 trillion will be spent on top of that during the same time period as companies try to do more with AI. Add those together — and I think at least $2 trillion will be spent on GPUs in the next 5 years.” - “I like to go after Zero-Billion Dollar Markets — nonexistent markets that I envision & can rationalize well, but will be very hard, expensive & time-consuming to achieve.”

1

Having worked at Cruise and Waymo on self-driving taught me many things. I hope to bring some of these learnings to healthcare so it too can soon have the magical, "no-driver-at-the-wheel" moment. Here's a short conversation I had with Louie Cai, MD on this topic!

12

1 Comment

🍜Anybody up for some ramen? I am excited to share that I am back in the garage, building again (it’s actually a house in the Berkeley Hills). We are taking the wraps off of Supercog AI, a new startup focused on solving Application Integration using LLM powered agents. My co-founder in this new venture is a great friend of mine (and a former co-founder partner), Alex Osborne. Alex and I have worked around the application integration space for a long time, and we are unrealistically excited about applying the GenAI stack to this problem. Not many people relish the task to get some bits to move from system A to system B. But it’s a critical job to be done, and that’s led to a lot of brittle scripts, a myriad of inflexible “no code” tools, and a lot of hand crafted SQL. But the power of the Large Language Model, trained on a huge corpus of information, offers a remarkable tool to solve this problem. The LLM you use today already knows the APIs of hundreds of popular systems. It knows the SQL dialects of every major database. It understands the semantics and data schemas of many popular SaaS systems. It understands much of the specific domain in which your business operates. I like to call the AI revolution “the last platform shift”. That may turn out to be hyperbole, but I definitely believe we are in the very early stages of seeing what this new stack can enable. I know we are hardly the first folks to claim that “amazing stuff is coming!”. But this is why Alex and I have decided to focus on a real and hard problem. If we can prove that LLMs can power a new way to solve this problem, it will be the first successful *new* approach in 15 years.. We will have much more to share about this new platform soon. In the meantime, if you’re interested in getting a peek - or even better if you have application or data integration tasks that you would like some help with - please reach out because we want to help. #genai #startups #backinthegarage

164

37 Comments

Don't fight against AI. Help it to get latest info. 👇 is the talk I gave last week regarding Real-Time #RAG in Zilliz's Unstructured Data and #LLM meetup. "Talk is cheap, show me the code" This blog shows the key data flow and code blocks. The full example is available at https://lnkd.in/gUj349hh How it works? Input: Stream stories and comments from #HackerNews API. Pre-process: Retrieve updates and filter for stories/comments via #Bytewax. Retrieve Content: Download the html and parse it into usable text. Thanks to the awesome #Unstructured.io. Vectorize: Create an embedding or list of embeddings for text using #HuggingFace #Transformers. Output: Write the JSON document with #embeddings and other fields to a local #Timeplus #Proton server, into the data streams. Filter/Route: Use streaming SQL to filter/transform JSON data and send output to a local #Kafka server. Forward: Use #Milvus plugin for #KafkaConnect to forward the data to #Zilliz Cloud. Query: Build a #streamlit web application to query the live data with the same embedding model.

18

1 Comment

Hey, I'd like to give a shout-out to ClearML for helping us at Vimeo develop AI-based systems. We have been happy customers for a few years now and I'd like to share a bit about how we use ClearML. We use ClearML Datasets to cache a dataset of video transcripts, and run testing loading directly from ClearML Datasets. This allows us to simultaneously speed up the data loading and ensure consistency. Moreover we use ClearML to save our benchmark annotations in one place and track our system performance on the benchmark every-time we run an experiment or change our prompts. ClearML allows us to see the different parameters and prompts that were used for each experiment and to monitor improvements or regressions in our performance. We also use ClearML to run large-scale tests and help with statistical evaluation of our methods. For example, we developed a RAG (Retrieval Augmented Generation) Q&A system and wanted to verify that the LLM will not answer certain questions or user queries that are outside the scope of the video. We used ClearML to collect and analyze the RAG responses on many videos for predefined user queries that were outside the scope of the videos and got good visibility into the performance of the system. Also, the comparison feature on ClearML is great for tracking the improvement of our metrics along the progressing versions of our systems.

29

1 Comment

A super clear, straight forward example of how to do cost effective RAG using LangChain for your document pipeline and Groq for compute resources. The code demonstrates how to speed up RAG and reduce costs using quantized embeddings (reducing 32 bit values to 8) Nice job Sahibpreet Singh! https://lnkd.in/gf5QMCKK

44

2 Comments

I wrote a blog post on a thorny but underappreciated problem in ML-based marketing personalization: event attribution. It's how you go from the actions you take and customer behavior you observe to creating training experiences for your model. Good attribution makes a huge difference in driving uplift, there are a lot of degrees of freedom in configuring it, and at the same time, it's rarely discussed in ML/bandits/RL literature.

38

4 Comments

Workflows v2 just dropped including multiple paths, multiple charts, loops, and a brand new refined UX. Grab more details at https://lnkd.in/gEN_9Gvy and come give it a try! Observability reimagined just keeps getting better. 🚀

27

Check out the "Titan Text Premier Demo" built by my team on this page

9

Congrats the team on ICML acceptance of the Wukong paper: https://lnkd.in/g-MyahNH, which represents a breakthrough of recommendation model design and efficiency. Recommendation models traditionally suffer from not able to utilize the full computational power of GPU (only 5-10% of peak GPU FLOPS while LLMs can reach >50%), due to their sparsity. Wukong is first of its kind by pushing toward 40% GPU peak FLOPS though model-infra codesign. One of my teams consisting of GPU experts is building a high performance model component libary where we’ll provide highly optimized model components (eg sequence summarization, x-sequence interaction, x-GPU layout scheme) that could be composed - with the goal that models built from this library have high performance out of box to scale the model-infra codesign effort. We are also scaling training beyond the limits that are typical of recommendation models - from hundreds of GPUs to 10s of thousands. If this is the kind of work that excites you, come join us!

264

3 Comments

Day 3/3 GenAI Summit San Francisco Causal AI, causal grounded RAG - cRAG, scaling from POC to production in AWS, a deep dive into the evolution of Llama models and what's coming next, synthetic data in the age of genAI, PII, and domain randomization. I can't count how many times I've heard the words AI, genAI, data, cloud, RAG, LLM, and multimodal in the last 3 intense days. And when I finally went to have lunch at a restaurant, at the adjacent table, a fintech AI startup founder was pitching to a VC, while the people at the table behind me were talking about data engineering. It feels so good to be in SF, where at the airport’s security check-in a big Anthropic advertisement greets you! 😊 The last 3 days were amazing, filled with inspiration and thought-provoking new tech. I can't wait to get back to building DiffuseDrive and working with our customers!

22

One of the best things about being in San Francisco 🌉 is the ability to drop in on great talks. Last week I had a chance to see NVIDIA's CEO Jensen Huang (JH) being interviewed by Stripe's CEO Patrick Collison. Jensen is among the longest-serving founder-CEOs in Silicon Valley at the moment (30+ years). With that comes a special clarity of mind and some hot takes. 🌶👇 🗣 Jensen doesn't believe in 'praise in public and criticism in private'. Instead, he reasons that most information in the organization should not be privy to certain people. By giving feedback in public, you allow everyone to learn. 'What is better than learning from your own mistakes? It's learning from someone else's mistakes!!' 🔥 JH also doesn't like to fire people. He would rather "torture them into greatness". He invests in people and he doesn't like to write off his investments. 👨‍💻 Asked whether he likes his job every day, he answers "no". The perfect job doesn't give you happiness 100% of the time. "No great things were done, that were easy all the way." ... "Even if I come home from a day where I did things I didn't enjoy, it could still be have been a great day." 👩‍👧‍👦 Jensen famously has 60 direct reports. "This allowed us to cut out probably 7 levels of management". His subordinates don't necessarily have such large teams. "It doesn't scale down, less senior people need more guidance". Looking at the numbers, this approach allowed NVidia to reach ~2T market cap with ~28.000 people vs. Microsoft's 3T with ~220.000 people and Apple's 2.6T with 160.000 people. ❔ Jensen answers almost every question in the same way: "Let's break it down, let's reason about it". This is a great response on so many levels. First, it allows him to not be put on the spot. Second, it allows him to teach his workforce how he reasons through a problem so that they can emulate it and create alignment. Third, it gives people a chance to point out flaws in his reasoning. He admits he is not right all the time. ON AI 👨‍🔧 "You will not lose your job to AI, your company will go out of business because another company used AI." "If you're not using AI you're doing it wrong." ⏱ The extent to which AI will enter many processes is largely underestimated. Right now we are focused on language - sounds in a time sequence. "Any time sequence can be tokenized." This will include robotic movement. 🥛 JH also expects that we won't be satisfied with foundation models "You're going to want to tune it into perfection because you care so deeply. Say if it's already at 99% you'll want to get it to say 99.3%" This last piece is also why we created Query Vary (YC W22). AI will never feel truly intelligent until it completely understands your data and context. It needs to be taught, regardless of how powerful the foundation model is. Finally, besides insights, it was remarkable how funny Jensen was, cracking a joke at roughly a 1 per 3 min rate. It was a very entertaining talk. #ai #sf

61

8 Comments

The math is simple. When you mix text-to-video AI magic with high-performance Cloud GPUs, you get faster video content generation, better image quality, and more sophisticated AI models. Ready for this incredible customer story? Hour One, whose AI video platform is now powered by Cloud GPUs, saw some incredible results from migrating to Google Cloud : - 20x business growth - Improved inference speed by 1.8x with Cloud GPUs - Reduced inference costs by 28% - More realistic and expressive AI-generated avatars with increased processing power Hour One’s mission is one to believe in: To democratize video content creation. And, Google Cloud is on board to make that happen! Find out how, Hour One is leveraging Google Cloud to empower generative AI video creation with our powerful, cost-efficient infrastructure >> https://lnkd.in/eK2w2r3r #CloudForIsrael #GenerativeAI

1

A colleague referred me to this. Great use case in how using humans to perfect, train and scale an AI system. The difference between self driving mode of Tesla vs. Waymo. Waymo is way ahead because of using humans to train the models faster for edge cases. In driving, as we now there are too many edge cases. Similarly, a colleagued complained to me why Speechify does not work well to read legal briefs (for example the solution reads outs headers and footers along with text, it does not disregard table of contents). I told her the solution will become better. Right now we are in first or second grade. Key lesson: solutions will get better and faster for now with humans in the loop.

4

4 models made available today: - Meta Chameleon: 7B & 34B language models - Meta Multi-Token Prediction LLM - Meta JASCO: text-to-music models - Meta AudioSeal: audio watermarking model

31

Learn how to build fast LLM apps with Groq, Langflow, and LangChain! Discover cutting-edge Groq LLM model inference, LangFlow's no-code integration, and LangChain's simplified code approach. Register here for the Live Stream on June 12th --> https://lnkd.in/gwUVpZTM #Langchain #Groq #Langflow #LLM #Developers

14

The most robust, reliable, and productive uses of AI are when many models are used in coordination with each other, with "regular" programming mixed in. This leads to more capable, more reliable, and more interpretable AI systems. Most people building with AI already know this. But (unless you work for Google) the main barrier to realizing multi-step AI workloads is an infrastructure one. For the rest of us, we’re left creating an unwieldy mess of chained API calls to multiple providers, with many roundtrips in between. This state of affairs is stifling progress. Nobody wants to deal with more tooling and infrastructure… but everyone would benefit from simple, intuitive interfaces that abstract away a powerful system underneath. Substrate is the first inference API optimized for multi-step AI workloads. With Substrate, you can write less code, run more inference, and build high performance AI applications with zero infrastructure to manage. No tooling, no infrastructure – just elegant abstractions. Excited to support Ben Guo, Rob Cheung and the amazing Substrate team on their mission to democratize how AI applications are built.

40

4 Comments