Narayan Desai

Anthropic released another paper on their ongoing work towards interpretability of large foundational models like their Claude 3 family, built on previous work by many. Their work continues to be enlightening and inspiring. The findings in this work are generally similar to previous findings on smaller models but scaled up to Claude 3 Sonnet: 💡 they find large numbers of "features" corresponding to combinations of neuron activations and these features map to concepts that we recognize/understand: for example, features for transit infrastructure, particular people, concepts (like bias and "objects lacking sentience"). This might sound familiar from classical machine learning, but the interesting thing is that there is no single thing/neuron encoding each of these concepts, it's a particular combination of neuron activations that embeds these concepts in "superposition" in the network. 💡they observe a sort of "distance" of these feature activations that arrange themselves "near" semantically similar features. When they use their sparse autoencoder approach to learn more and more features they observe that the level of abstraction in the detected features becomes more granular, in ways that are interpretable and meaningful. 💡when they artificially fire these activation patterns during inference, it affects the output in ways that relate to the "feature". For example, asking Claude "what's the most interesting science" normally it responds Physics, but if you clamp the feature activations for the feature "Brain Science" to 10x their max values then Claude responds: Neuroscience. Ways to interpret _and_ influence the model. Great readable work [1] along with an interactive explorer of some of the "features" themselves [2] 🎂 There's an easter egg too: if you scroll in the feature explorer tool and try and select one of the "Randomly selected features from 1M" you get a message "if you want to see the rest of the features, we're hiring!" with a link to their recruiting page :) [1] https://lnkd.in/gZbparwp [2] https://lnkd.in/gC9eus53

19

I've been testing GPT-4o since its recent release. While I haven't really had a use for the multi-modal (images, audio, and text) capabilities yet, the improvement in speed of response is noticeable. Not just noticeable from a spec sheet perspective either—it's noticeable in how the product works in real use cases. I've been thinking about why that is, and here's my theory: there's a breakpoint somewhere based on latency and response speed. Above this breakpoint, interactions feel sluggish and disruptive. Below it, they become seamless and effortless, akin to a natural conversation. Previously, I posted about how Llama and Gemini models seemed noticeably more usable than GPT-4, with my preference being Llama whenever possible. This new experience with GPT-4o further emphasizes the importance of optimizing both latency and throughput in AI systems. This chart from ArtificialAnalysis.ai shows that GPT-4o has a total response time of 1.5 seconds for 100 tokens, compared to GPT-4's 5.7 seconds. This speed difference isn't just a matter of numbers—it's fundamental to achieving a smooth and efficient user experience. When latency drops below a certain threshold, the interaction becomes fluid and engaging, making the AI more practical and user-friendly. It also makes its mistakes more tolerable. This improvement transforms how we use these models in real-time applications, making AI interactions faster and more intuitive. It underscores that achieving low latency and high throughput is no longer a nice-to-have but a critical requirement for practical AI deployment.

9

3 Comments

I heard a few people are suspecting this live demo showing a call to a virtual agent is not real https://lnkd.in/gnBiGDfd I can tell you that the call was real and we are able to allow the users to talk to the virtual agent with a text UI while still on the call. To find more, search the newly launched feature called 'Call Companion' under Dialogflow

31

One of the challenges of incorporating AI into systems is that in some respects it's very similar to traditional software development, and in other respects it is not. Our software supply chain teams have been working for some time on how to translate software supply chain principles and practices to fit the needs of the AI supply chain. I'm very proud of the work our team and partners have done here. This report describes the key insights they have made. #ai #hardening

20

pgvectorscale: A Game-Changer for Vector Similarity Search 🚀 Timescale's #pgvectorscale extension for PostgreSQL has achieved a remarkable milestone, matching the performance of Pinecone, a dedicated vector database, while being 75% more cost-effective. 💰 In their latest blog post, Timescale showcases the impressive speed and efficiency of pgvectorscale, which leverages the power of PostgreSQL to deliver lightning-fast vector similarity search. This breakthrough opens up new possibilities for applications like semantic search, recommendation systems, and more. 🔍 Key takeaways: 🌟 pgvectorscale now matches Pinecone's performance in vector similarity search 💸 pgvectorscale is 75% more cost-effective compared to Pinecone 🔧 Easy integration with existing PostgreSQL databases 🌐 Scalable solution for handling billions of vectors ⚡ Optimized for fast and efficient search results pgvectorscale's seamless integration with PostgreSQL allows developers to incorporate vector similarity search into their applications with ease, while benefiting from the robustness and scalability of a proven database system. 🗄️ If you're looking for a high-performance, cost-effective solution for vector similarity search, pgvectorscale is definitely worth exploring. 🔍💡 #pgvectorscale #postgresql #vectorsimilaritysearch #semanticsearch #recommendationsystems #timescale https://lnkd.in/gsfgCK32

5

Now we have two first class data catalogs integrated with MS Fabric now available in Azure. Of course DataBricks is the heavy heavy favorite in the enterprise lakehouses as well as AI (as of the conference that just finished 🤓)

6

<rant> Yesterday morning I wrote a post about the phrase "scaling independently" while waiting for my KIND cluster to spin up, and I promised that I would give an example of where you really DO want things to "scale independently" in a separate post. Well, that post is here. Let's talk about #apachekafka and its new Raft-based metadata solution, known as KRaft (yes, it evokes Mac-and-Cheese, which is partially why I like it so much). KRaft is a replacement for ZooKeeper's former role in Kafka. In a KRaft-based cluster, the Metadata Quorum is a group of Kafka Servers that have the "controller" role. The Metadata Quorum stores information such as "what topics exist in my cluster?" and "which Broker is the leader for which Partition?" and "which follower replicas live on which Broker?". This information is CRITICAL for the availability and consistency of a Kafka cluster. In order to ensure consistency of the Metadata Quorum, one specific Controller is chosen as the leader. All write requests go through that leader. In order to ensure availability of the Metadata Quorum, there can be "follower" Controllers in the Metadata Quorum which also store the metadata updates (synchronous replication). In case of failure of the leader, one of the followers can become the leader. In this specific case, it DOES make sense to scale the Brokers and Controllers independently. Why? First, the Controllers don't actually scale. One can be a leader at a time, so the others are just providing backup (note: for reasons beyond the scope of this post, it's best to have an odd number). Additionally, before KIP-853 is implemented, it's actually really hard to add or remove Controllers to/from the Metadata Quorum without downtime. Secondly, it's possible for a misbehaving client to take down a Broker, for example by sending way too much data to a specific partition. If it's just a single Broker that's lost, most of the cluster will continue on and live to fight another day. However, losing a Controller is a Very Bad Thing. Thus, by separating the Controllers from the Brokers, we can improve the availability of the cluster. It is indeed possible to run Kafka with some servers that share the responsibility of Controller and Broker. This is great for development (especially local dev) and also in *SOME* highly resource-constrained production environments. However, I would suggest as a rule of thumb that you should probably: - Separate out your Controllers on their own isolated machines - Start with a Metadata Quorum of size 3, which allows losing one Controller and continuing on alive - Put your Brokers on another set of nodes. Now, there's another good example of how Kafka scales X and Y independently: Compute and Storage. Watch out for the next "Colt Rant" about this, coming later this week! </rant> PS—here's the link to the previous post: https://lnkd.in/gXU77rKW

12

6 Comments

Large Language Models (#LLMs) are evolving rapidly, and the demand for running them at scale is skyrocketing. #Ollama is stepping up to meet this need with its official container image on Docker Hub, excelling in local #deployments. However, there’s a catch for containerized environments: downloading #models requires the ollama serve command to be running within the #container. This extra step complicates automated deployments via #CI #CD #pipelines, making seamless model access challenging. To take your #model to production, it needs to be #container-ready—a feature Ollama has yet to fully support. The gap in #AIOps for smooth deployments gets blurred with the inference services available as an easy way to take it to production. Goal is to overcome obstacles and deliver solutions 😉 so here is to get you started on having your (O)#llama #production ready in a private #deployment from your #local to #cloud all in #oneshot . Enjoy Medium: https://lnkd.in/g9jqBQ8d dev.to: https://lnkd.in/gi_gGwam #AI #LLM #Ollama #llama3 #phi3 #mistral #S6overlay #AIOps #CI #CD #oneshot #container #docker #podmanEnjoy

11

With the announcement of S3-native-streams (Freight clusters), here is a commentary on Confluent strategy regarding object storage, streaming and an open data architecture. https://lnkd.in/dTg6DKVy

48

6 Comments

Can you trust AI? 🤔 For most companies the answer is no - at least using current foundational models out of the box. ❌ Problems like sensitive data leakage and toxic content are major barriers to implementing production genAI solutions. The route towards trust lies in system architecture and the good news is the field is advancing rapidly, including increasingly easily accessible toolkits of solutions to increase trust in AI systems. 🔥 This is the hot area for new releases from major AI platform players (along with AI security), including AWS (https://lnkd.in/e45S-ehS), DataRobot (https://lnkd.in/eCWvppWR), and Databricks (https://lnkd.in/e6A7cTcH). 👨‍⚖️ Guardrail or judge models - leveraging a second AI model to police the content coming out of an original - are probably the most promising scalable solution. For many the best route is likely to be a new class of foundational models specifically designed for this task such as Llama Guard and Databrick’s new judge LLM. There is also the ability to fine tune a model (think a discriminator model from a GAN architecture) or use a general purpose foundational model. 👮‍♂️ Classic filtering techniques for sensitive data can also be applied to the result of an LLM, whether for PII, company IP, or inappropriate content. This can include regex, classic PII models (such as Amazon Comprehend), etc. 🚻 Human in the loop techniques can also be used, but are far more scalable when used to calibrate automated solutions such as those described above rather than manually verifying outputs in realtime. What’s still missing? 🔎 🎯 While techniques to improve accuracy and reduce hallucination such as RAG and AI agents have made great strides, there remains a big gap in conveying a model’s confidence in its output to end users. 💡One idea is to leverage the probability distributions used for top-p (nucleus sampling) at the token level instead at greater aggregations to convey confidence levels for entire model outputs or significant segments (e.g. each step in a chain of thought structure or AI agents framework). #ai, #artificialintelligence, #GenAI, #GenerativeAI, #llm, #datascience, #machinelearning, #mlops, #bigdata

4

1 Comment

TL;dr - RIP Terraform. Time to say goodbye. I just ran into this obituary for Terraform and Hashicorp. The most probable future seems to be the maximization of sales motion and the minimization of engineering investment - just adding another product to the price list and unleashing the sales force at it. The worst possible outcome would be if IBM decided to make Terraform part of their still-born cloud offering. Or maybe we’ll have Terraform for mainframe? #terraform #hashicorp #ibm #upgrade

32

5 Comments

I have a love hate relationship with #observability space. The hate is mostly all self-inflicted, e.g. misreading docs, misreading configurations, debugging polyglot services and dealing with idiosyncrasies of otel sdks in multiple languages. It’s mostly love though! It’s powerful when it works! I think community is coming along way when it comes to documentation and resources. It still feels early. You have to be very self sufficient and be comfortable ditching/trying/demoing/hacking on many different tools in various languages and various environments. All in all, despite its warts and problems, it’s mostly all gravy! #opentelemetry is one of the best things to have happened in tech that gets under appreciated! Embracing it allows you to be a lot more freer than just to be tied to a single vendor. That being said…love and proud customer (across multiple companies) of honeycomb.io ! Not only are they great vanguards of #opentelemetry, their product (at least for tracing) is till my favorite across all oss/commercial products..and I’ve tried most all of them!

81

6 Comments

🌶🌶🌶 Hot take… Old problem: if we put more devs on the project, we can deliver it sooner. Same thinking, new manifestation: we can hire more overseas devs for less total money to produce the same result as our US staff. Flaw in that thinking: product development is not an assembly line. Devs are not “interchangeable”. Software is as much an art as it is science. It’s a team sport. The people on your team matters. #tech #staffing #management #hiring #jobs #layoffs #software #startup #teamleadership

25

6 Comments

I chew the cud on the magic and mechanics of idea diffusion. Beware, I fanboy hard on this book. https://lnkd.in/gaCQKztq

12

2 Comments

This is a fun blog post about first steps with Rust -- and what looks like a legitimately useful tool!

22

Do you think you need to hire developers as w2 employees to build a highly-skilled, devoted and culturally aligned team? Let Blueprint show you otherwise. Here's how we're changing the game: 1️⃣ Unmatched Quality: Quality is our creed. Our vetting process includes interviewing with former CTOs and senior software engineers, ensuring only top-tier talent makes it to your team. 2️⃣ Long-Term Commitment: Forget the turnover and churn worries! Our developers stay longer than industry norms for W2 developers (average tenure of 2.2 years), providing stability and continued expertise to your projects. 3️⃣ Cultural Integration: Culture isn't just an add-on for us. We dive deep to understand your core values and ensure every developer we match with your team enhances and embraces your company culture. 4️⃣ Equity is Optional: Save your cap table for your growing startup! We ensure that our developers are passionate believers in your product right from the start - without incentivizing them with equity. We rigorously test for craftsmanship, ensuring they value not just a quality end result but also the journey of creation. 🌟 Shoot us a DM to connect and see how we can improve your staffing strategy. #StaffAugmentation #InnovationInStaffing #TechTalent #CultureFit #LongTermGrowth

20

1 Comment

Object storage direct-write Kafka topics are coming to Confluent. In data system design, cost is a kind of unstoppable force that rewrites architectures every decade or so, and that is happening right now for high-volume streams. https://lnkd.in/dHRFnrMB

39

1 Comment

Humanity's approach in pushing forward developing AGI continues to sorta puzzle me. As I see it, there are a few possible outcomes (in the short and long terms), depending on how successful the technology and application are: - We create really sophisticated but ultimately equally unhelpful chat bots, which permeate society to the increased frustration of literally everyone - We automate away manual labor jobs, increasing corporate profits at the expense of the lower class, and shift even more tax burden onto the middle class to compensate - We get productivity multiplying AGI, which is accessible mainly to the wealthy, and magnifies the wealth disparities in society - We get full AGI, which quickly evolves to displace humanity as the dominant life form, in whatever form that takes Now, I get how the people with wealth can think there are some "good" outcomes there, but I kinda still fail to see if/how normal people think this might be "good" for society (if they do at all). Academic points of course (since we as a society can't really stop it), but interesting in a "course of history" sense.

1

🚨 I took down a major piece of infrastructure as a tech lead at Yahoo! and didn’t get fired…. I think this is the nightmare scenario for engineers - to not only break your own product, but to do broader carnage to systems at the company. I think we all fear what the repercussions are if we do this. However, when intent is good, recovery is swift and learnings documented, these things are actually quite formative. When it happened, I was able to remain calm and walk through a rollback in 2 minutes, which allowed the downstream infrastructure to recover in about 10 minutes. This was also the start to my reputation as a level-headed technical leader. Good relationships with my peers in the infrastructure team allowed us to quickly diagnose that my code push was the cause of the incident and take action. I was leading a team building a very powerful piece of software. We had a client library that was installed on every server and allowed us to push out front-end code to every web property simultaneously. Code changes would propagate in about 2 minutes from push. It allowed us to provide a consistent branding, search and navigation system.  Because we understood the power of the system, we also had plans in place if something went wrong and had practiced rollback and forwards. Typically, when things went wrong, they were minor UX bugs in the interface. Medium problems cost us some dollars in search revenues. However, this time, my change had a massive impact downstream. I had added a new polling feature that worked against the caching strategy of the platform I was calling. Within 2 minutes of diagnosis I executed the rollback strategy. The bad news was the product manager had to wait a few more days for the new feature to roll again respecting the caching strategy of the platform. The good news is that we recovered quickly. I learned a few lessons from this outage: 🚨 Outages happen all the time and you will likely not lose your job if you cause one. 📝 Your team should have plans should an outage happen how you will approach debugging and recovery. ⏰ Practice for urgent situations before they occur. 🏋‍♀️ Building and maintaining strong connections with leads of systems where you have dependency is key. 🧞‍♀️ I have a special ability to appear completely calm while feeling absolute panic inside, which is a good capability to have. 

59

9 Comments