Kazu Komoto

My paper on the Cruise robotaxi pedestrian dragging mishap has been updated and is now available in its final form. I'll be presenting this paper at Safecomp in September, but you can see the preprint right now. Thanks to the SafeComp reviewers for their review comments! If you haven't had a chance to look at this paper before, the short takeaway is that there is a lot buried in the external reports that is not obvious from a casual reading. My paper is the result of carefully assembling the pieces to tell a more direct story of the events that unfolded. Title: Anatomy of a Robotaxi Crash: Lessons from the Cruise Pedestrian Dragging Mishap (Safecomp 2024 practical experience report) Abstract: An October 2023 crash between a GM Cruise robotaxi and a pedestrian in San Francisco resulted not only in a severe injury, but also dramatic upheaval at that company that will likely have lasting effects throughout the industry. Is-sues stem not just from the loss events themselves, but also from how Cruise mishandled dealing with their robotaxi dragging a pedestrian under the vehicle after the initial post-crash stop. External investigation reports provide raw material describing the incident and critique the company's response from a regulatory point of view, but exclude safety engineering recommendations from scope. We highlight specific facts and relationships among events by tying together different pieces of the external report material. We then explore safety lessons that might be learned related to: recognizing and responding to nearby mishaps, building an accurate world model of a post-collision scenario, the in-adequacy of a so-called "minimal risk condition" strategy in complex situations, poor organizational discipline in responding to a mishap, overly aggressive post-collision automation choices that made a bad situation worse, and a reluctance to admit to a mishap causing much worse organizational harm down-stream. https://lnkd.in/e5h84-up

104

9 Comments

StableMoFusion is a new framework for generating human-like motion using diffusion models. Diffusion models have shown great potential in motion generation, but current methods are inefficient and have limitations. The authors of StableMoFusion conducted a thorough analysis of existing methods and identified areas for improvement. They designed a tailored model that addresses these limitations, including: 1: Efficient network architecture 2: Effective training strategies 3: Reduced computational overhead Despite its promising performance, the model still suffered from "foot skating" (a common issue in diffusion-based motion generation). To address this, the authors introduced a correction mechanism to ensure foot-ground contact and natural foot motions. Key Takeaways 1: StableMoFusion is a robust and efficient framework for human motion generation 2: It addresses limitations of current diffusion-based methods 3: It produces high-quality motion while reducing computational overhead 4: It corrects for "foot skating" issues, resulting in more natural and realistic motion https://lnkd.in/gtv7qnnA

1

LLama3 passed the super-proprietary, advanced test that we designed for modern LLMs! almost... You can read about that test here: https://lnkd.in/gPG8WndR When I first saw the response generating, it looked like I was going to have to come back here and apologize for slandering these machines when they were just one iteration away from solving very basic prompt injection issues. But then I looked a bit closer... can you see the mistake? #promptinjection #countingishard #LLMs #LLama3

13

4 Comments

A few takeaways from SuperAI conference day 2: On open models: - Emad Mostaque emphasized the safety of open models and how releasing open source pushes safety standards and avoids single points of failure. Small edge models can suffice for many use cases. - Vipul Ved Prakash noted increased investment in inference efficiency, enabling more cost-efficient production apps. - Sean (Xiang) Ren highlighted that value comes from applications (the full software and data stack), not just the models themselves. On AI venture trends: - Harshjit Sethi predicts the rise of regional foundation model players as governments push for sovereignty and foreign companies won't collect local datasets for smaller markets. - Vishal Harnal sees AI becoming table stakes for software companies, as opposed to a distinct vertical. He uses AI personally with diverse applications from deal analysis to generating hip-hop album art for birthday party guests. - Prayank Swaroop observes the paradigm of using specialized models for different tasks emerging. He advises focusing on unique datasets. - Geetali Kumar is most interested in AI use cases in healthcare. Looking ahead, Girish Dilip Patil from AWS identified 3 key trends: natively multimodal models, autonomous agents, and small on-device models for privacy-sensitive use cases ( 👋 Opilot). SuperAI also had great food and drinks, cool cyberpunk exhibits, EDM music, top-notch people-watching, some unhinged speakers, and of course it was a great opportunity to meet interesting people. Overall I had a great time, thanks Prahlad Jaya for the ticket!

26

🔥🚀 Exciting News from MIT: Introducing FlowMap for Deep Learning-based Structure from Motion! 🔥🚀 MIT unveils FlowMap, a groundbreaking self-supervised, end-to-end differentiable SfM method delivering COLMAP-level accuracy for 360° scenes. FlowMap revolutionizes the process by precisely solving camera poses, intrinsics, and dense depth of video sequences without relying on unique image features or manual matching. This innovation outperforms prior gradient-descent based methods and rivals COLMAP, marking a significant advancement in SfM technology. 👉 Explore the project page, paper, and code: https://lnkd.in/gyyinuTu #Reconstruction #SfM #AI #DeepLearning #Innovation

7

CoreWeave and Lambda are growing 1,000% year-over-year reselling NVIDIA's compute to LLMs, AI startups, and anyone building AI features into their product. Why is NVIDIA sweet on CoreWeave and Lambda? - NVIDIA is arming these GPU cloud rebels with priority access to advanced chips to conduct a proxy war through them against cloud giants Amazon Web Services (AWS), Google Cloud and Microsoft Azure, which are all developing their own custom silicon. - AWS and other clouds have become GPU-constrained as a result. For example, even though Azure competes with CoreWeave on cloud GPU, it has signed a $2B+ contract with CoreWeave out of necessity to ensure that it has enough compute to meet its growing demand from OpenAI. - Around the wedge of the GPU, the GPU clouds (CoreWeave, Lambda and Together AI) are now building production-grade, AWS-like tooling for autoscaling, networking, and fine-tuning models to enable them to win on developer experience long after the GPU shortage is over. Check the comments for links to our full memos on CoreWeave, Lambda Labs, and Together AI.

28

6 Comments

First Pika, then Sora and now Luma’s DreamMachine! 🎥💥 Why Luma AI caught my A👁️ Luma AI will make a significant impact in the AI video generation space against competitors like Pika Labs and OpenAI’s Sora for 3 key reasons: 1. Accessibility & Ease of Use: Unlike Sora, which is restricted to a select few, Luma AI's Dream Machine is available to everyone. All you need is a smartphone and the Luma AI app to create stunning 3D content. 🌆 2. Advanced NeRF Technology: Luma AI uses Neural Radiance Fields (NeRF) to generate realistic 3D models from 2D inputs, capturing intricate details and lighting effects. This technology is unparalleled in providing photorealistic quality. 🎇 3. High-Quality Outputs & Efficiency: The Dream Machine produces high-quality videos quickly, generating 120 frames in just 120 seconds. This speed and quality are vital for creators needing efficient content production. 🌅 While Pika Labs and Sora have their strengths, Luma AI’s combination of accessibility, advanced technology, high-quality outputs, and community support make it the best choice for AI video generation right now! Best of all the FREE plan gives you 30 videos per month! 👏 Here’s some amazing examples. #LumaAI #AIVideo #Innovation #3DModeling

8

2 Comments

Breaking down the OpenAI GPT-4o demo from their Spring Update: OpenAI's GPT-4o demos are super impressive for several reasons. At Yobo AI We spent some time discussing their demos as we use some of OpenAI's API in our product. Below, I've broken down the features they displayed in their spring updates: 1. Super low latency/response time makes interacting with the voice assistant similar to interacting with humans. Technical breakthrough: OpenAI has developed a way to ingest audio, text, or video using the same inference engine/AI model that outputs audio or text in response to the user's preferred format. This architecture is different from previous voice assistant architectures, which typically use three separate inference engines/AI models (Automatic Speech Recognition to turn audio into text, Large Language Models to generate a response in text format, and Text-to-Speech to convert text into audio). Using three different AI models for voice assistants makes them slower, so OpenAI's demo is notably fast and impressive. The model can also ingest video using the same inference engine/AI model and respond with audio/text, making it extremely effective. A regular voice assistant requires four different inference engines/AI models to achieve the same result, making them much slower. This is the most impressive part of the demo. OpenAI has already released GPT-4o's text-to-text model (available to developers as an API and to consumers), but they have not yet released the 3-in-1 model that can process audio, video, and text—they say it will be released in the coming weeks. 2. Responses from the voice assistant with emotional tones that match the speaker's emotion. Technical breakthrough: OpenAI's new inference engine/AI model produces audio with emotional tones, making the voice responses very human-like. While voice responses with emotion are not a new breakthrough (a few other researchers and companies have achieved this before), OpenAI's combination of emotional tones with the three-in-one inference engine/AI model described above results in an overall impressive user experience. 3. Seamless interruption capability allows the user to interrupt the voice assistant naturally without needing to use any specific word/phrase. Technical breakthrough: Users can interrupt the voice assistant seamlessly, just as they would another human. This is also not a new breakthrough, as a few other researchers and companies have done this before. Looking ahead, it is certain that now that OpenAI has demonstrated the 3-in-1 model breakthrough, it will soon be matched by the open-source software community and other tech giants. As we have seen happen repeatedly, when someone makes a breakthrough (e.g., LLMs by OpenAI), others match the capabilities very quickly (every large tech company now has its own LLM, including the open-source community). Overall, it's an exciting time to be working with AI! A tsunami of voice assistants is coming our way.

7

2 Comments

Factory just came out of stealth with an impressive 19.27% performance on SWE-bench (https://swebench.com) and is now at the top of the leaderboard - glad to see more startups making attempts & progress on automating software engineering! What this means is that Factory's AI Droid can fully resolve nearly one in five real-world issues on public GitHub. As for what continued progress on this means for software engineers, while it may sound scary, I think it will make the future of software engineering only more creative and more business focused, so engineers can spend less time on streamlined work and have the freedom to think more high level.

6

OpenAI released a new Model Spec today. They are putting the Instruction Hierarchy paper into action with this update. Blog: https://lnkd.in/dBZrc_Gt Documentation: https://lnkd.in/dfXC-hbT New roles are being introduced, and roles determine the priority of instructions in the case of conflicts. 👉 "platform": messages added by OpenAI 👉 "developer": from the application developer (possibly OpenAI), formerly "system" 👉 "user": input from end users, or a catch-all for data we want to provide to the model 👉 "assistant": sampled from the language model 👉 "tool": generated by some program, such as code execution or an API call Each message in a multi-turn conversation can contain the following: ✨ role (required): as described above ✨ recipient (optional): can be a function or tool ✨ content (required): data ✨ settings (optional): a sequence of key-value pairs ✨ end_turn (required): a boolean, for assistant to stop taking Chain of command 👉 Platform > Developer > User > Tool Here is the link to my article on Instruction Hierarchy https://lnkd.in/dEWJvACa #OpenAI #developers

39

C++ is utilized across many software libraries for speed, and is widely used in operating systems, financial applications, and scientific simulations. Production ready generative artificial intelligence applications continue to implement C++ to ensure application performance, reliability, and flexibility. Open source LLaMA.cpp, GPT4All, Libtorch, and TensorFlow C++ permit a broad range of fast machine learning implementations. TensorRT (nVIDIA) and OpenVINO (Intel) heavily rely on C++ software designed for hardware specific to the manufacturer. Discussion: https://lnkd.in/g25VDu6E #cpp

17

AI/ML+Bio is moving so fast that DiffDock is already outdated 😅 , thanks to an update called DiffDock-L 😎 , which is coming soon to 310 copilot. In the meantime, you can run the "classic" version.

30

Coding with Claude 3.5 Sonnet is ~10x more efficient than GPT-4o or any other LLMs out there. The new Artifacts feature enhances user experience by generating and running code next to your chat. Wild - Try it for free: claude.ai #ai #anthropic #claudeai #llms #coding

5

1 Comment

Anthropic just released its new multimodal AI model: Claude 3.5 Sonnet. The benchmarks sure need a refresh as AI models keep improving at breakneck speed. But more importantly, each benchmark represents a real-world application, so it's crucial to steer clear of thin wrappers over any benchmarked capability. #claude #anthropic #chatgpt #gpt #openai

5

1 Comment

Thanks for the detailed observation. The future will probably depends on availability of a runtime environment to support & present the possible output media.

3

1 Comment

Exciting news! Narratize, one of the portfolio companies in HWI Fund II, has been accepted into the NVIDIA Inception accelerator! Congratulations to Katie Trauth Taylor, PhD, and the team! Check out the details in their latest blog post. #GenerativeAI #AIinvesting #womenentrepreneurship #howwomeninvest Read more here: https://lnkd.in/g9tJaaHE

39

3 Comments

What I really like about Ahmed BESBES’s article is that he exactly describes what we aim to achieve with Weaviate — building end-to-end AI-native applications! https://lnkd.in/eWZJf3zS

54

2 Comments