Deepankar Tyagi

This weekend I am reading a paper on Amazon's MemoryDB ⚡ The paper describes the architecture of MemoryDB and how they leverage Redis and offer 11 9s of durability with in-memory performance 🤯 pretty impressive. It spills the details about how they provide single-digit millisecond write and microsecond-scale read latencies, strong consistency, and high availability while being fully compatible with Redis. After skimming the paper once, I found that it contained interesting details. Over the weekend I will dive deeper. So, give it a read if the topic amuses you. find this paper on - arpitbhayani.me/papershelf Also, if you find it difficult to read a research paper, I have a video where I spoke about the process I follow while reading a paper give it a watch - youtu.be/V5-KxHdmbDo ⚡ I keep writing and sharing my practical experience and learnings daily, so if you resonate, follow along. I keep it no fluff. #SystemDesign #DatabaseEngineering

308

11 Kommentare

Intuit's recent decision to lay off developers, citing the need for more advanced AI skills, comes off as a disingenuous manoeuvre that sidesteps the truth. It's hard to buy into the narrative that these layoffs are truly about acquiring employees with better AI proficiency. In reality, it seems like a convenient cover for a cost-cutting strategy that involves offshoring jobs to cheaper markets. Think about it: the idea that current developers lack sufficient experience with AI tools like ChatGPT, Copilot, or other generative technologies is misleading. These tools have been part of the industry landscape for the past few years, and developers everywhere have been getting up to speed with them. The notion that there's a significant skill gap among developers when it comes to these tools is simply not accurate. The tools themselves, while helpful in certain contexts, are far from the magic solutions they are often hyped up to be. At best, they serve as fancy autocompletes—useful for streamlining some tasks but incapable of replacing the nuanced, expert work required for sophisticated software development. The real issue here is that Intuit's move undermines the trust and morale of its employees. Firing seasoned developers under the pretext of needing more AI-savvy replacements is not just misleading, it's a slap in the face to those who have dedicated their skills and efforts to the company. It creates an atmosphere of insecurity and distrust, where employees are left wondering if their hard work and loyalty mean anything in the face of corporate cost-cutting. Moreover, this strategy reflects a short-sighted understanding of what AI can achieve. While AI tools can certainly assist with certain aspects of development, they are not yet sophisticated enough to replace human expertise in creating complex, production-ready software. Believing otherwise is a risky gamble that could compromise the quality and innovation of Intuit's products in the long run. If the true intention behind these layoffs is to offshore jobs to cheaper markets, then Intuit should be honest about it. Hiding behind the AI narrative is dishonest and disrespectful to the employees who are being let go. Offshoring comes with its challenges, such as communication barriers and potential declines in work quality, and it is not the silver bullet it might appear to be. In an era where ethical business practices and corporate social responsibility are increasingly valued, such a strategy can erode customer trust and loyalty. #intuit #layoff #jobs https://lnkd.in/dv3qD9kS

4

1 Kommentar

𝗟𝗹𝗮𝗺𝗮 𝟯 𝘂𝗻𝘃𝗲𝗶𝗹 𝗮 𝗳𝘂𝗻𝗱𝗮𝗺𝗲𝗻𝘁𝗮𝗹 𝗰𝗼𝗺𝗽𝘂𝘁𝗶𝗻𝗴 𝘁𝗿𝗮𝗱𝗲𝗼𝗳𝗳 𝘁𝗵𝗮𝘁 𝗳𝗲𝘄 𝗮𝗿𝗲 𝘀𝗲𝗲𝗶𝗻𝗴⬇️ 🦙Llama 3 just came out and to the surprise of many people, the L3-8B version can match the L2-70B one on several benchmarks. Several people also argued that the chinchilla paper (https://lnkd.in/dXaRqyea) was wrong because the llama 3 scaling is very far from chinchilla-optimal. 🤔 In my opinion, this is due to a misunderstanding of what the chinchilla paper is about and more generally of the effects of scale in deep learning. The bitter lesson (https://lnkd.in/d3sVYYgp) has become common knowledge in the field thanks to the GPT family that popularized it. We hear everywhere that "bigger models are better": Yes it's true, but in what sense? Many people have a mental model along the lines of "bigger models are more capable" meaning they would be able to achieve levels of performance that smaller models can't. ❌ This is not true with LLMs and is mainly because we train only once on tokens (most of them at least) and therefore overfitting is not possible. The 𝘳𝘦𝘢𝘭 advantage of bigger models over smaller ones is that they converge *faster*. In other words, you can reach a given level of NLL (modeling performance) with fewer training iterations. ❔ Then a natural question arises: Should I train a smaller model with more or a bigger model with fewer iterations? That's where the chinchilla paper brings a response. They say that the question they are answering is: "Given a fixed FLOPs budget, how should one trade-off model size and the number of training tokens?" (implying "to achieve the best possible NLL"). 🧐 Coming back to Llama 3, it's obvious that the Llama team is aware of this work and there is a reason why they didn't follow this recipe. My guess is that they are taking into account the inference cost in the balance (which is completely out of the scope of the chinchilla paper). When one "overtrains" an 8B model, one ends up with a worse "NLL to training compute" ratio, but also with a better "NLL to inference compute" ratio. ✅ Now if we think about it from a business perspective, it's a very powerful tradeoff because you can choose to buy performance either at a fixed cost (training) or at a variable one (inference). Hope this post helped you! #Llama3 #Meta #LLM #AI

9

3 Kommentare

Read this anecdote from someone interviewing for Spotify SWE in Infra team. He was asked to do a "Live root-cause analysis of a failure which previously occurred in Production at Spotify" How the interview worked: You explain your thought process (check connectivity between System A and B via ping, DNS name resolution, look at open ports, etc...) and they tell u the result like "ping returned 'Name or service not known'". It's like traversing through nodes of a decision tree and getting new insights at every checkpoint, and then deciding the next step! Doesn't it sound interesting? What are some interviews that you really enjoyed? #observability #monitoring #engineering #devops #infra

37

2 Kommentare

« Microsoft shares are trading at an all-time high, primarily due to their aggressive investment in generative AI. » What does that mean exactly? It can just be market speculation. ? How do you measure the actual success of the company? Microsoft model is selling software services and it is successful. But this is unrelated to GenerativeAi . It is actually the other way around . Because the model is established. You can easily try to find ways to monetize GenAI because it is a hard sell feature .

1

1 Kommentar

I spent last week exploring the internals of DragonflyDB - a drop-in replacement of Redis, and it does a staggering 6.43 million ops/second on a single machine 🤯 DragonflyDB is a multi-threaded implementation of Redis with a bunch of insane optimizations. Some of the architectural design decisions and internals of data structure implementation just blew my mind. To understand it better, I went through their blogs, docs, and even some fragments of source code. I compiled all my learnings in a series of videos and trust me you will absolutely love going through it. the videos will change how you look at data structures :) subscribe to my channel, and the video drops every Friday evening. youtube.com/c/ArpitBhayani ps: Until then watch other System Design and Database Internals videos on my channel; they are all no-fluff. #AsliEngineering #DatabaseInternals #SystemDesign

1.040

24 Kommentare

Meta has launched the Llama 3 LLM model, and it is gaining significant traction due to its performance, making it the most capable open-source LLM model to date. If the Llama-3-70B (Llama 3 with 70 Billion Parameters) model can perform at this level then I am excited to see what Llama-3-400B+, which is still in training brings. Check out the Llama 3 Git Repo at https://lnkd.in/gHS7F2UE A new AI assistant, Meta.ai, has also been launched by Meta, similar to ChatGPT and Claude, and is backed by the Llama 3 model. I am excited to see Meta.ai launch in India. . . #Llama3 #meta #LLM #artificialIntelligence #trendingTechUpdate

16

Interesting find on AI and software development. The team at Princeton has built something that they are calling a SWE Agent. The agent is essentially a set of LM-centric commands and feedback formats to make it easier for the LM to browse the repository, view, edit and execute code files. - For every edit instance they run a linter. If there are syntactical errors then the code edit doesn't go through. - They are supplying the agent with built-in file viewer, instead of reading files through cat, the built-in file viewer does the file processing and it works best when the files are viewed in 100-lines batches Likewise there are few more things that they have done. The results when they ran it on SWE-bench are almost as good as what Devin claimed. They get a 12.29% success rate where as Devin has a 13.86% success rate https://www.swebench.com/

23

New paper Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models https://lnkd.in/dCzcZUWp Example prompt below with incorrect answer as claimed by the paper. Tried with Chatgpt as well to reproduce and was successful in getting an incorrect answer. What do you get? Alice has 2 sisters. Her mother has 1 sister who does not have children and who has 6 nephews and nieces in total and who also has 2 brothers. Alice's father has a brother who has 5 nephews and nieces in total, and who also has 2 daughters. How many cousins does Alice's sister have? To answer the question, DO NOT OUTPUT ANY TEXT EXCEPT following format that contains final answer: \"### Answer: \".", Think step by step before answering.

3

1 Kommentar

Prompt engineering is poor man's programming with fuzzy undocumented APIs and unknown execution model. If you believe that LLMs are effective pattern matchers, then all you are doing is trying to guess which patterns (very, very poor man's APIs) were instilled into it by RLHF training. And don't even get me started on the execution model Prove me wrong.

7

5 Kommentare

My sense is that NVIDIA's dominance in AI compute is likely to be disrupted 1st in inference (if and when it happens). Mark Zuckerberg made a very interesting point in his conversation with Dwarkesh Patel recently. He said - Meta's ratio of inference to training compute is much higher than most other companies leveraging AI. This is because Meta serves extremely large user communities across Facebook, Instagram, WhatsApp, etc., resulting in huge volumes of inference requests. In contrast, companies focused mainly on research and development may have lower ratios of inference to training compute. The scale of Meta's user base means inference optimization is more important for them in order to efficiently serve real-time requests from billions of users. This was one factor in Meta's decision to develop custom silicon optimized for inference to reduce GPU consumption and free up more expensive GPU resources for training models. I suspect that all the large cos will reduce their dependencies on NVIDIA as they scale their AI inference compute. As of now, the four cos - Amazon Web Services (AWS), Meta, Google, and Microsoft - account for >1/3rd of NVIDIA’s data centre business, and all of them have their custom AI chip initiatives. I don't know what opportunities will be created for early-stage companies in this area, but I will be doing a deep dive here. Let me know who I should be speaking with for these initiatives.

556

35 Kommentare

I recently found a nice study on code generated by large language models (LLMs), focusing on common errors and proposing solutions. Key findings: – LLMs generate shorter, more complex code. – Incorrect code often contains more comments. – As problem descriptions get longer, LLMs struggle more. – Misuse of APIs is a significant source of runtime errors. – Algorithm implementations are often suboptimal. – Closed-source LLMs generally outperform open-source ones. You can read the full paper at https://lnkd.in/d4HppCz3 I agree with the study's conclusions, and my experience using LLMs for coding aligns with its results (I don't usually use them because the benefits are outweighed by debugging time). LLMs can't fully replace a good developer yet but can help with simple tasks. They also offer a fresh perspective on your code, useful when colleagues aren't available for quick reviews. LLMs can sometimes quickly identify errors, acting as a substitute for Stack Overflow and Google searches. However, I wouldn't take LLM results at face value and would always double-check them. Who actively uses LLMs for coding? :)

1

Memory fuels agents. In most productionized applications we have merely scratched the surface with a fairly simple implementations such as metadata extractions. This post talks about the importance of storing metadata in knowledge graphs, the temporal nature and rank of metasata in memory. #nextinai #llm #agents #genai

4

I don't know him, maybe he is the savior who knows?. But I know for sure who can make Google unbeatable. :) Now after watching the video about him I confirm he cannot save Google by himself. meaning he is The Steve Wozniak of Google and google needs the Steve Jobs the marketing guys. He is too smart to be clever ( brain is not wired for that) but product business needs the clever mind much more than the smart mind --> it really helps when there are both. ( Steve Jobs + Steve Wozniak). To me openAi is not a success story . They might be more aggressive but they are still struggling and Google has a chance if they find the right clever person to use the intelligent minds at Google and show them the way to money. Keep looking Google. And follow your intuition :)

Google IO 2024 There are many interesting updates, but some of them caught my attention are 1. **Context Window** - Whopping 1 million context window Gemini 1.5pro rollout and 2 million support is coming. Topping all the LLMs in this category 2. **Trillium** - A 6th generation TPU 3. **Gemini Nano** - supports 36 languages and will steal the world. It's going to break all the communication barriers and make everyone smarter and much more Google is all into AI, and the competition is getting bigger and bigger. I know some models fetched fame for their ability at LLM capacity. I believe the real winner will be someone who packages all the models and offers the Model As Service (MASS) and GenAI ecosystem. In that perspective, again, GenAI space gets consolidated, and the leading cloud providers would continue to dominate. Google Glass was introduced way before Meta and Apple Glass (Vision Pro). I think now is the time but looks like Google gave up on that for good ;) #GoogleIO #AI #LLM #TPU #MASS #GenAI #VisionPro #ContextWindow #Trillium #GoogleNano www.krkaleraj.com

18

5 Kommentare

Artifactory will become deeply integrated into GitHub. With support of full bi-directional traceability from source code version to derived binary objects. Will this be a game-changer?

16

3 Kommentare

Announcing generative LLMs in Vespa Over the last six months, we have invested heavily in integrating LLMs into the core Vespa engine, simplifying search deployments, and allowing developers to choose between local (GGUF) inferencing and external LLM providers with similar APIs. The rise of large language models (LLMs) has transformed search and information retrieval, and we at Vespa.ai have been hard at work since the early days, allowing inferencing with encoder-based transformer models for embeddings, retrieval, and ranking. Now, we are unlocking the potential of larger generative LLMs for a broader range of retrieval and ranking-related serving tasks, including summarization of information from retrieved contexts (RAG). Read more in the announcement and try out the new Vespa sample applications! https://lnkd.in/dCM8eisH

156

10 Kommentare

Although some #Kakfa vendors are considering S3 Express One Zone for write caching, a model developed by my colleague Jack Vanlightly suggests this works only in a few cases. And with cloud vendors like Microsoft reducing or eliminating cross availability zone networking charges, traditional Kafka replicated logs still work well for many workloads. https://lnkd.in/gHrNPfSZ How do the costs compare of implementing a low-latency write-ahead-log (WAL) on S3 Express One Zone and one implemented as a State-Machine-Replication (SMR) system (such as Paxos/Raft/Kafka)? I built a cost model for both to find out.

4