Bhushan Khanal

Today I hosted #llama3 for the first time using TWO A100-80GB with 32 cpu cores and more than 150G RAM and it was pretty fast. It costed me $5/hr though. I hope to run a quantized version on one A100-80G (for 16fp one should be enough). I also hosted 8B version on something like A6000 RTX with with only 6 cpu core and around 55G RAM which was quite fast. One question is that is it better to run a 8B-32fp or 70B-8bit! #ai #selfhosting #gpu #llm Some comments: For running llama3, first you need to take permission. Second, The whole gpu on-demand allocation and starting and running takes somewhere between 30-60 min, so depending on your machine cost, you are paying for some of that preparation time.

19

4 Comments

Ok. I have done the #llama3-70b with 8bit precision by #ollama which requires half the RAM, ie, ~71GB . I ran on a system with 2 A6000RTX on FluidStack with 12 core cpu and 110 GB of RAM for hourly rate of $1.6/hr. I did not notice any degradation in accuracy, however, I have not seen any useful benchmark to say for sure. I realized that the best pragmatic metric is to compare price of generating 1M tokens. Here is one example: System with 2 A6000RTX generate 8 token/sec so it takes 3-4 sec to generate a single line of 20 words or so. In an hour you can generate at most 3600*8 ~ 29000 token so the cost of generating 1M tokens is like 35*$1.6/hr= $56/hr which is the twice as expensive of #gpt4-tubro. With that $5/hr 2 A100-80G with 60 cpus and 240GB of RAM, the speed was 50 token per second. So, the price for 1M token was around $27.5 which is much cheaper than the previous system. One last word: System comprised of 2 A100-80G ($20K each) with 60 cpus and 240GB of RAM costs at least $60K with one year operation costs no less than $80-$100K and you can rent it for $35K/yr which is pretty good deal IMO.

3

How do the costs compare of implementing a low-latency write-ahead-log (WAL) on S3 Express One Zone and one implemented as a State-Machine-Replication (SMR) system (such as Paxos/Raft/Kafka)? I built a cost model for both to find out. https://lnkd.in/dwczVutW

43

5 Comments

Cool day. Llama 3.1 405B released this AM. CentML has it hosted live this afternoon (try it out below 👇). CentML also optimized it for higher throughput and lower latency based on its preferred compute stack. Planning to build with Llama 3.1 and beyond? CentML might save you millions (on top of what Meta is saving you), and help you deliver best-in-class experiences for your users.

33

Large Language Models (#LLMs) are evolving rapidly, and the demand for running them at scale is skyrocketing. #Ollama is stepping up to meet this need with its official container image on Docker Hub, excelling in local #deployments. However, there’s a catch for containerized environments: downloading #models requires the ollama serve command to be running within the #container. This extra step complicates automated deployments via #CI #CD #pipelines, making seamless model access challenging. To take your #model to production, it needs to be #container-ready—a feature Ollama has yet to fully support. The gap in #AIOps for smooth deployments gets blurred with the inference services available as an easy way to take it to production. Goal is to overcome obstacles and deliver solutions 😉 so here is to get you started on having your (O)#llama #production ready in a private #deployment from your #local to #cloud all in #oneshot . Enjoy Medium: https://lnkd.in/g9jqBQ8d dev.to: https://lnkd.in/gi_gGwam #AI #LLM #Ollama #llama3 #phi3 #mistral #S6overlay #AIOps #CI #CD #oneshot #container #docker #podmanEnjoy

11

Have you ever wondered whether the order of fields within a Go struct affects memory consumption or application performance? I did, and it turns out that it might indeed have an impact. The reason? Memory alignment requirements in modern CPU architectures. I documented my understanding of the topic at https://lnkd.in/ddDsG8MU . Take a look if you're interested, feedback is appreciated!

16

2 Comments

The 6502 CPU rocks. Still here? Great. Greetings fellow 8-bit friends. Just saw the 6o6 “6502 on 6502 VM”. This is sweet - a clever approach that uses the underlying hardware to execute a “VM’s” code so each VM gets an accurate 6502 implementation. To get more memory each VM, the developer uses GeoRAM 512 from Berkeley Softworks (of GEOS fame) for the C64 and used bank switching to assign private memory spaces. Tons of detail in the write-up. One of those “read in pieces between powerpoints” kind of documents. :-) Enjoy your weekend. https://lnkd.in/gDNc_RrT

2

4 Comments

pgvectorscale: A Game-Changer for Vector Similarity Search 🚀 Timescale's #pgvectorscale extension for PostgreSQL has achieved a remarkable milestone, matching the performance of Pinecone, a dedicated vector database, while being 75% more cost-effective. 💰 In their latest blog post, Timescale showcases the impressive speed and efficiency of pgvectorscale, which leverages the power of PostgreSQL to deliver lightning-fast vector similarity search. This breakthrough opens up new possibilities for applications like semantic search, recommendation systems, and more. 🔍 Key takeaways: 🌟 pgvectorscale now matches Pinecone's performance in vector similarity search 💸 pgvectorscale is 75% more cost-effective compared to Pinecone 🔧 Easy integration with existing PostgreSQL databases 🌐 Scalable solution for handling billions of vectors ⚡ Optimized for fast and efficient search results pgvectorscale's seamless integration with PostgreSQL allows developers to incorporate vector similarity search into their applications with ease, while benefiting from the robustness and scalability of a proven database system. 🗄️ If you're looking for a high-performance, cost-effective solution for vector similarity search, pgvectorscale is definitely worth exploring. 🔍💡 #pgvectorscale #postgresql #vectorsimilaritysearch #semanticsearch #recommendationsystems #timescale https://lnkd.in/gsfgCK32

5

Performance is key! As we cannot share the detailed work with (former) customers of Rockset (acquired by OpenAI), we decided to implement the Rockbench benchmark to check how streaming ingest between CrateDB and Rockset (acquired by OpenAI) compares. The results are very insightful. We see: �� CrateDB achieves 6-9x lower latencies than Rockset for streaming ingest. ✅ CrateDB outperforms Rockset on the same hardware while saving about 20% costs. ✅ When volumes increase, the latency increased linearly in Rockset, while remaining mostly flat in CrateDB. As a plus: 1️⃣ Achieve full high availability due to the distributed nature of CrateDB (benchmark results are achieved including replication in CrateDB) 2️⃣ Achieve higher throughput on the same amount of CPUs as Rockset (at 50% of memory needs) 3️⃣ Full open-source solution that prevents you from market shifts On side note: If you are thinking to switch to Elastic/OpenSearch, this benchmark has been created to compare Rockset vs. Elastic, so CrateDB is also faster than Elastic for streaming ingest cases! Read the full comparison here: https://lnkd.in/d_nEMDkx

29

1 Comment

At the core of effective engineering leadership is the ability to stay connected with your team. This is perhaps the most challenging aspect because retaining the respect of your people is difficult unless you remain technically involved yourself. In my journey as an engineering leader, staying true to the individual contributor role that originally earned their respect has kept me close. This approach extends beyond technical expertise; it's about maintaining a deep connection with the core activities that propel product development forward at Gearflow. One of my biggest professional fears is becoming irrelevant or disconnected from my team and our work. This fear motivates me to continually reinvent and retool in response to market demands. Next month, I'll be sharing about my adventures generating synthetic data recently. And while I'm no expert when I start preparing for these talks, the process is incredibly valuable. It's less about delivering a perfect talk and more about how the preparation expands my thinking and exposes me to new capabilities. This helps me connect the dots to opportunities at Gearflow that bring real innovation to our customers. Speaking regularly holds me accountable and pushes me to learn ahead of the curve, ensuring I remain relevant and impactful day-to-day. Every talk is a journey of discovery, helping me to lead with humility, communicate clearly, and inspire a culture of innovation through a deep, technical understanding of our work.

19

New post on what happens after you add a control plane and local storage for observability: you get 1000x the telemetry when it matters. https://lnkd.in/gAvkk-qP

58

1 Comment

This overview of Generative AI and Retrieval-Augmented Generation by Gwen (Chen) Shapira of Nile Database is fantastic. Highly recommend it for other infrastructure nerds like me who want to know what's going on in the AI world. #generativeai #rag https://lnkd.in/gJum7_qa

10

It was hard, but we now have a primitive at Gadget that lets us do absolutely 0 downtime atomic changes to an application's structure or data under the hood, with 0 requests dropped. I call it "request catching" but it was inspired by this Braintree eng post I read long ago For stateless workloads like serverless functions, 0-downtime deploys are pretty easy. You stand up the new one next to the old one, use a load balancer to start sending requests to the new one, and slowly stop sending requests to the old one. But for stateful workloads, it's much harder. You can't just start a new Postgres and send some queries to the new one and some to the old one without lying to somebody about state. If you want to be consistent, there has to be one source of truth. Fancy DBs have quorums for this, but PG doesn't. If you want to be consistent, you're effectively forced to have some tiny-but-real window of downtime in between draining the old one and starting the new one. That said, there's nothing forcing you to return errors to clients while that is happening. If your infrastructure supports it, you can pause requests while this changeover is happening under the hood, and then resume them once it is over. Clients will notice a longer request that normal, but it will be served like normal and they'll be none the wiser! What's required to do this though is a deep deep integration between the load balancing infrastructure and the thing doing the database deploy under the hood. We've built this up using a distributed locking tool, where all requests start by taking out a shared lock. When we're ready to do the maintenance under the hood, the maintenance workflow takes out an exclusive lock, blocking all requests from proceeding. The workflow does the atomic change it needs to make knowing it has exclusive access which means nothing will change under its feet, and then releases the exclusive lock as fast as it can so those requests can proceed. As an optimization, we only check the lock on the request path if an app is marked as near a maintenance window. This is only possible because of Gadget's architecture where we bundle together the load balancer, the request processing layer, and the database access tier such that we can reliably intercept all requests and all database transactions and do this lock checking. Its these nasty, in-the-weeds details that make me so proud of Gadget -- we can do world-class engineering stuff under the hood on behalf of all our apps without them ever noticing. If you're interested in doing this on your stack some OSS tools have the primitives you need! We use nginx for our load balancer, Temporal for the reliable workflow execution and we're currently using postgres advisory locks for hte locking. I think the Braintree post has been lost to time after they got acquired, but @simonw has a great post on the subject here: https://lnkd.in/eZmiktGT

40

12 techniques to reduce your LLM API bill and launch blazingly fast products Cost-efficiency and minimal latency are critical for deploying production-ready LLM applications.   Sahar Mor's article sheds light on 12 innovative techniques to slash your LLM API bills, with gems like Semantic Caching and Summarizing Conversations leading the charge. This isn't just about saving tokens; it's a strategic playbook for scaling smartly in an AI-dominated landscape.  #MachineLearning #AI #NLP #CostOptimization #Innovation #TechTrends https://lnkd.in/ec3qst-M

19

1 Comment

Introducing rindexer, an open source, no-code blazing fast EVM indexing tool built in rust 🦀. rindexer supports any EVM chain out of the box. rindexer is also a framework for building more complex indexers using rust. https://lnkd.in/e8PAs87n Docs - rindexer.xyz

25