Fenye Bao

I am excited to share the latest blog, co-authored by myself, Junhwi Kim and Vyas Sekar, detailing how we managed to successfully localized and improved LLM by combining Fine-Tuning and RAG. https://lnkd.in/gKDkgGPq

69

3 Comments

At Instrumental Inc. we are typically very secretive about the clients we work with. During onboarding 3 years ago, I knew I made the right decision to join, especially after seeing the list of major logos we support. Today we are doing something a bit out of character and publicizing our partnership with @Meta to assist in the quality control of their new hardware. I personally believe this is the sort of edge hardware companies need to outperform competitors in a rapidly evolving market. Don't just take my word for it though, check out the video! Really exciting chapter in tech and at both companies. #AR #Engineering #Innovation

14

Hallucinations in LLMs are inevitable, no seriously! Josh Fritzsche and I have worked hard at Voze to find a way to tune, prompt and even plead 🙏 with LLMs to prevent them from the occasional 🍄 hallucination. It seemed like the better we have done the harder the edge cases have been to solve and eventually we realized that they tend to frequently happen when LLMs predict or make anticipations on "nothing". I remember at one point nearly going insane with frustration because when LLMs had a hallucination, they were EPIC! 😱 Then we pushed through it and benefited for the effort! How did we solve this common problem? We didn't (Not completely), we got better at identifying the issues as they happened. Once we identified in the data we were able to start looking for patterns. The same catalysts that cause it to happen once if left unchanged caused it over and over again. Our solution is astoundingly simple. Once you know that it's likely to happen..... let it! Accept it will 🍄, but also be preemptive in not focusing on it and use alternative data sources, generators or providers. Change the conditions that cause it but continue to monitor it as it happens. As Dan Caffee is fond of saying to me on a regular bases. "Dusty we need the system to be adaptive and provide mechanisms to learn from it's past." Well he is right! All we needed was data and the people that contribute to it's quality. Huge props 👏 to strong support from Kathy and Janelles audit teams. We were armed with a knowledge and a desire to learn from past mistakes. Armed with confidence that we could correct these problems we could focus on the data to prevent their impact going forward. When doing millions of LLM (Generative AI) calls regularly Josh and I have learned that hallucinations will happen. We also know exactly how we will handle and prevent it going forward! Failure is not doing anything and accepting defeat.

4

2 Comments

sharpening writing skill is one of killer skill!

2

Interesting article from Bloomberg about life at Google's X in a time of layoffs and cost reduction at https://lnkd.in/gmw3GG7C. X is a component of Alphabet (Google's parent company) that incubates new ideas. They have been responsible for some great successes - getting Waymo and a lot of Google's AI work started - but they've also had projects that didn't survive - like Google's attempt to provide Internet access via weather balloons. When you work at a company that has hundreds of billions of dollars in revenues, it's hard to get a lot of interest in an entirely new business idea that brings in hundreds of millions. If you have a project that improves ads revenue by that amount, or decreases capital expenditures, you can do it, but if you want to, say, build wearable tech, it becomes hard to attract the attention of marketing or product. So, new ideas often die on the vine. For the most part, this is good and proper. Companies may have the financial wherewithal to branch out, but more often than not, they consider it a distraction. And they are probably right - it's hard to argue that Meta was right to divert such a huge chunk of their resources to chase after the metaverse. They've (rightfully) pulled back, and are more focused on their core business again. On the flip side, if you are in the tech sector, and you don't innovate, you are likely to get left behind at some point. So, you need to make and manage these investments somehow. When I joined Google, the CEO had a 70-20-10 rule - 70% of effort went to core businesses, 20% to broader efforts to improve Google, and 10% to stuff that's completely blue-sky. This is probably not completely right, but it is also probably not completely wrong. X is the division of Alphabet that is *specifically* designed to bring blue-sky products to life. Unsurprisingly, they've had to pivot and rethink their strategy many, many times. The article interviews a number of people who have worked at X over the years; it's well worth a perusal, for folks who are interested in the story.

17

2 Comments

An open letter from Meta employees to Mark Zuckerberg and Meta's leadership. Very well articulated, I couldn't have done it myself better! I left Meta end of 2023 because of many things similar to what's mentioned in the letter, so I can definitely relate.

87

Thats the exact explanation I gave to a group of investors last week. (… it’s just Precision/Recall, attributes vectorization as “embedding”, K-Nearest Neighbor, User Search Intent, DAG, ML algorithms, etc all over again …). LOL and they asked what’s new this time around. And I was like not much, ML is still the backbone, and AI is the inference and actions based on the model. It works better when there is a feedback loop in a closed system. But the current state of the art like ChatGPT is based on scraping the Internet (garbage in, garbage out). Kinda like early search engines. What is a better analogy is that NVIDIA/Snowflake etc are selling tools, aka pickaxes to prospectors. The gold is up to you to find. And the opportunity right now is to monetize industry specific (healthcare, travel, financial services) clean data to build models with (aka a treasure map for the prospectors). From an enterprise perspective, what’s out there today aka generative AI isn’t very useful. Too much hype. To build useful engines will require an effort similar to search revolution 2 decades ago. From

14

3 Comments

Happy to be speaking at Arize:Observe 2024 about GenAI Evaluation with a few wonderful colleagues from Google Cloud. 📅 For GenAI apps, answering the question of "How to measure success?" is non-trivial yet key to iterative improvement. If you are a developer, come talk to me. We would love to brainstorm, learn your thoughts and unique approach towards GenAI model and application evaluation. 🧑‍💻 Arize AI #GenAIEvaluation

86

9 Comments

One of the simplest techniques for improving sensitivity in your experiments is eliminating or reducing dilution. Dilution is akin to throwing more hay in the haystack, thus making your ability to find a signal extremely challenging. In this blogpost, David Press and I explain dilution and provide some practical guidance on how to handle it. We also provide some examples showcasing how enormous is the impact from eliminating dilution. For ML applications, having a way to handle dilution can lead to 100x+ sensitivity gain.

59

2 Comments

This blog post comprehensively presents fundamentals of neural sparse search and measurement of its effectiveness. https://lnkd.in/gJ6yUFEB

28

🌟 Reflecting on Two Years at Amazon: A Journey of Growth and Gratitude 🌟 As I mark the completion of two years at Amazon, I find myself in a state of humble reflection. It's hard to believe its already 2 years since the day I nervously stepped into this bustling hive of innovation. Back then, if you had told me I'd be celebrating this milestone, I might have doubted it. Like many, I was unsure if I would even last six months in such a dynamic and fast-paced environment. These two years have been a whirlwind of challenges, opportunities, and growth. I've had the chance to dive headfirst into many different technologies, each one stretching my abilities and expanding my horizons. From cloud computing to machine learning, I've tried to embrace the unknown with open arms, with an eager to learn and evolve with each new project. But perhaps the most rewarding aspect of my journey has been the opportunity to build high-performing teams from the ground up. There's something truly magical about bringing together a group of talented individuals and watching them soar. Together, we've faced obstacles with determination, resilience, and a shared commitment to excellence. I am grateful for my wonderful teams without whom this milestone wouldnt have been possible. Of course, along the way, there have been setbacks and failures. Projects that didn't quite go as planned, initiatives that missed the mark. But in the true spirit of Amazon, we've embraced these moments as opportunities for growth. Each failure has been a valuable lesson, guiding us towards greater success in the future. Looking back, I realize that these past two years have been a turning point in my career. I've had to unlearn old habits and old ways of thinking, making room for the innovative mindset that defines Amazon, and Amazon ways of doing things. It hasn't always been easy, but it has been truly rewarding. At this juncture, I am deeply grateful for the support and guidance I've received from my leaders and peers at Amazon. Their mentorship and encouragement have been instrumental in my growth and development. Ankur Shrivastava, Naveen Belur, Kameshwari Mudgala, Narayan Hegde, Rahul Pamecha, Nandan Dubey, Abhishek Khandelwal, Partha Bera, Dan Brown, Anil Godavarthy, Venki Ravi, Suraj Ray, Vipul Jain, Vardaan Sharma, Nabagat Purohit, Ulrike Wingenter, David Gona, Bibhuprasad SKA, Avi Spivack and Andrew Connolly As I look ahead to the future, I do so with a sense of excitement and anticipation. The journey so far may have had its challenges, but it has also been filled with moments of triumph and growth. And through it all, one thing remains clear: It's always Day 1 in Amazon, and I am grateful for the opportunity to be a part of this incredible journey. #Amazon #CareerJourney #Gratitude #HumbleReflections #amazonanniversary #2years Amazon Amazon Web Services (AWS)

109

46 Comments

Principal Engineer Staff Engineer Distinguished Engineer (my favourite) I’m going to say it and it’s likely a little rhetorical: I dislike all these terms immensely. Please can someone give me the canonical description for what they are supposed to mean? (I know I can use a search engine or AI to do it… I want to know what people think they mean). The industry I work in is in danger of disappearing up its own backside. What’s wrong with: Software engineer Why do we need to differentiate? The only reason I can see is signposting to people with differing levels of experience or different sets of responsibilities. If we need to do that then, I’m not sure our teams are set up correctly in the first place. #software #softwareengineering #softwaredevelopment #productdevelopment

99

105 Comments

I am writing this just before we start the second of three days where my fellow Principal Engineers in Amazon's People eXperience and Technology organization have come together to discuss strategic technical challenges and how we can leverage our collective skills, talents, knowledge, strengths, and collaborative power to inform solutions to these challenges while delivering real incremental value. Remember my post about the people I work with a couple of weeks ago? This is precisely what I meant. I feel so energized and motivated, ready to take on even the most difficult of situations with the support of my peers - an incredible group of crazy smart and kind people with diverse backgrounds who really care about doing good for our customers (current, future, and former Amazonians) and one another, and from whom I learn a great deal every time we meet. ... which reminds me, we are hiring Principal Engineers in People eXperience and Technology! If you are an experienced software engineering individual contributor leader and if improving the lives of one of the largest, most varied, and complex global populations as part of an amazing community like the one I described speaks to you, please message me to learn more! #amazon #softwareengineering #community #hiring

81

Congrats the team on ICML acceptance of the Wukong paper: https://lnkd.in/g-MyahNH, which represents a breakthrough of recommendation model design and efficiency. Recommendation models traditionally suffer from not able to utilize the full computational power of GPU (only 5-10% of peak GPU FLOPS while LLMs can reach >50%), due to their sparsity. Wukong is first of its kind by pushing toward 40% GPU peak FLOPS though model-infra codesign. One of my teams consisting of GPU experts is building a high performance model component libary where we’ll provide highly optimized model components (eg sequence summarization, x-sequence interaction, x-GPU layout scheme) that could be composed - with the goal that models built from this library have high performance out of box to scale the model-infra codesign effort. We are also scaling training beyond the limits that are typical of recommendation models - from hundreds of GPUs to 10s of thousands. If this is the kind of work that excites you, come join us!

264

3 Comments

👩‍💻 When I worked at Google and Meta, I collaborated with thousands of #softwareengineers who wanted to #workonclimate but since Google and Meta aren't primarily climate companies, most of them did it in their 20% time. 🌎 At #Tesla, you work on climate 100% of the time (and to be honest, sometimes 120%, so don't apply if this scares you 😅). ❓ I often get questions from people to mentor them in their transition into working on climate if they don't already know the domain space. Where should they start? 🛣 There are several ways to progress your career, and no linear path works for everyone, but in general, my practical and actionable advice is to leverage your engineering skillsets in a role that doesn't require you to have as much domain expertise to start, which frequently is in the supporting software spaces. 👨‍💻 Brendan Schlosser is #hiring for both frontend and backend Energy Service Data Infrastructure software engineers: 1️⃣ https://lnkd.in/gRjb4skd 2️⃣ https://lnkd.in/gyqFDHsD 👩‍💼 The roles are in-person in Palo Alto, CA (remote employees will not be considered.) ☝ Please apply using the links above if you would like to do meaningful and interesting work in accelerating the world's transition towards renewable energy.

28

1 Comment

⚡ It is rare that Tesla has open #productmanager roles, but Ahmad Hares is hiring for a #SCADA and industrial battery pack controls product manager right now: https://lnkd.in/gnPuYMRK 🎛 While previous knowledge of SCADA is helpful, very strong candidates with high product sense who can learn extremely quickly and are willing to work ten times harder to learn the domain space will be considered. 👇 Relevant post I wrote below about how to break into climate tech if you don't have the domain expertise.

18

1 Comment

From IC engineer to C-Level exec, the most effective people I've worked with have one thing in common: they break down and decouple decisions. Optionality, trade-offs, last responsible moment, one-way and two-way door decisions, slicing, MVPs, proof of concepts etc. are all techniques they use for breaking decisions down and getting them made. The best engineers did this when working on code. Rather than trying to decide on all the functionality at once they'd be able to say "I can break this story down into smaller pieces of value that we can get released to users fast" or "we don't need to deliver this piece of functionality now, we can wait until we see how user's respond". The best architects, product managers and business leaders do the same thing. It's a real skill to decouple decisions. It's a real skill to be able to defer parts of a decision to next or later (or even never) to enable faster movement and lower risk. I have always been impressed when I witness it. Often it's obvious with hindsight and sometimes its completely out-of-the-box and you would have never have realised.

60

3 Comments

There are a lot of tutorials and learning plans for LLMs and GenAI, especially using AWS Bedrock and Azure Machine Learning. These are super valuable for building things quickly. However, I prefer a more hands-on vendor-agnostic, approach, as it allows me to learn concepts I can apply to any platform. To that end, I've been slowly working through Building a Large Language Model (from Scratch) by Sebastian Rashka, which is part of Manning Publications early access program. It's not like anybody's going to pay me to build one from scratch, but I like knowing how things work at a low-level under the hood. On the one hand, this book is very tutorial-ish so it feels like "now this, now that, now this other thing." On the other hand, just the first chapter or so on tokenizing had me experimenting with tokenizing things that aren't text, because it's easy to play around with code I built myself. https://lnkd.in/gC27QTfF

6

4 Comments