Bobby Nakamoto

A successful systems design and delivery strategy begins by assuming that change is likely the only constant. Be adaptable, or fail.

2

We need to move past the idea that knowing everything is the best; asking good questions is a much more critical and underrated skill that we need to embrace. I’ll demonstrate with a case study on how the exact problem to solve can dramatically shift based on answers to basic questions. But first, why does this matter? This is because good questions are the cornerstone to truly understanding the underlying requirements. So many issues would simply not exist if the right questions were asked at the right time - Many products with little to no adoption would not be built - Numerous designs would not be scrapped very late in the game - Countless incidents would be caught before they make it to production Let’s use my favorite leveling question shared previously https://lnkd.in/gxCHTWSu. "Imagine you join our team/company X and discover that our SLAs (Service Level Agreements) are consistently in the red. What steps would you take to address this issue?" Fundamental questions to ask: Q1: Are the customers actually unhappy? If there are no customer complaints, the problem may boil down to a simple SLA update. If there are complaints, changes to the system are likely necessary Q2: What specific SLAs are in red? How long is “consistent”? If multiple SLAs (latency, availability, consistency) have been red for 12 months, it's a deeper systemic problem. If a single SLA is red for 3 months, it’s a simpler problem. Q3: What is the SLA and what is the system performance? If the SLA and system performance don’t diverge that much, it can be solved with incremental steps. However, if the SLA is 99.99% and performance is 99%, then deeper issues must be addressed. To recap… Asking the right questions at the right time can entirely transform the approach and solution, saving a lot of time, effort and grief. It is a critical skill for senior engineers. I’ll leave you with this iconic clip from Margin Call… “Explain, as you might to a young child. Or a golden retriever” ~~~ ✍️ Do you agree? Share if you have examples where the right question changed project’s direction. ♻️ Repost if this is helpful.

30

2 Comments

I am excited to share the latest blog, co-authored by myself, Junhwi Kim and Vyas Sekar, detailing how we managed to successfully localized and improved LLM by combining Fine-Tuning and RAG. https://lnkd.in/gKDkgGPq

69

3 Comments

The Site Reliability Engineering: How Google Runs Production System book is arguably one of the most prevalent and well-known SRE resources out there, but there are many other great options for folks looking to expand on their learning. We recently spoke to Google's Reliability Advocate, Steve McGhee, in our Humans of Reliability interview series. In addition to his interesting anecdotes on the early days of SRE at Google, and his journey to becoming a Reliability Advocate, he also shared a handful of his favorite SRE resources, which we compiled here into a list: https://lnkd.in/gpUVhyRa

16

A new post for my blog this morning, on the great new paper from the Amazon MemoryDB team: https://lnkd.in/eeKYfzBW This paper is a great reminder that distributed systems aren't just about scalability. Availability, durability, consistency, performance, and consistent performance are all equally important. Distribution makes them easier to achieve. Read the team's paper here: https://lnkd.in/eWxuuSTA

415

2 Comments

Part of what inspires me at my job is the ability to identify gaps and having the autonomy to create and build. Andy talks about builders as inventors. He then talks about primitives. I have not heard this before so as I read on I became obsessed with the idea and how I can relate it to my work. To sum it up, the idea is to create every process as a separate entity so that they can be mixed and matched as needed to create a new and better process. The criteria is that a primitive can't be divisible. The idea that a new invention, service or process doesn't need to be perfect it just needs to be better is novel. One can take the product then, gain feedback and improve upon it as often as necessary until it becomes the gold standard. Well, that sums it up for me but feel free to read... https://lnkd.in/gZSuhf7T

9

At the core of effective engineering leadership is the ability to stay connected with your team. This is perhaps the most challenging aspect because retaining the respect of your people is difficult unless you remain technically involved yourself. In my journey as an engineering leader, staying true to the individual contributor role that originally earned their respect has kept me close. This approach extends beyond technical expertise; it's about maintaining a deep connection with the core activities that propel product development forward at Gearflow. One of my biggest professional fears is becoming irrelevant or disconnected from my team and our work. This fear motivates me to continually reinvent and retool in response to market demands. Next month, I'll be sharing about my adventures generating synthetic data recently. And while I'm no expert when I start preparing for these talks, the process is incredibly valuable. It's less about delivering a perfect talk and more about how the preparation expands my thinking and exposes me to new capabilities. This helps me connect the dots to opportunities at Gearflow that bring real innovation to our customers. Speaking regularly holds me accountable and pushes me to learn ahead of the curve, ensuring I remain relevant and impactful day-to-day. Every talk is a journey of discovery, helping me to lead with humility, communicate clearly, and inspire a culture of innovation through a deep, technical understanding of our work.

19

🚨 I took down a major piece of infrastructure as a tech lead at Yahoo! and didn’t get fired…. I think this is the nightmare scenario for engineers - to not only break your own product, but to do broader carnage to systems at the company. I think we all fear what the repercussions are if we do this. However, when intent is good, recovery is swift and learnings documented, these things are actually quite formative. When it happened, I was able to remain calm and walk through a rollback in 2 minutes, which allowed the downstream infrastructure to recover in about 10 minutes. This was also the start to my reputation as a level-headed technical leader. Good relationships with my peers in the infrastructure team allowed us to quickly diagnose that my code push was the cause of the incident and take action. I was leading a team building a very powerful piece of software. We had a client library that was installed on every server and allowed us to push out front-end code to every web property simultaneously. Code changes would propagate in about 2 minutes from push. It allowed us to provide a consistent branding, search and navigation system.  Because we understood the power of the system, we also had plans in place if something went wrong and had practiced rollback and forwards. Typically, when things went wrong, they were minor UX bugs in the interface. Medium problems cost us some dollars in search revenues. However, this time, my change had a massive impact downstream. I had added a new polling feature that worked against the caching strategy of the platform I was calling. Within 2 minutes of diagnosis I executed the rollback strategy. The bad news was the product manager had to wait a few more days for the new feature to roll again respecting the caching strategy of the platform. The good news is that we recovered quickly. I learned a few lessons from this outage: 🚨 Outages happen all the time and you will likely not lose your job if you cause one. 📝 Your team should have plans should an outage happen how you will approach debugging and recovery. ⏰ Practice for urgent situations before they occur. 🏋‍♀️ Building and maintaining strong connections with leads of systems where you have dependency is key. 🧞‍♀️ I have a special ability to appear completely calm while feeling absolute panic inside, which is a good capability to have. 

59

9 Comments

Large Language Models (#LLMs) are evolving rapidly, and the demand for running them at scale is skyrocketing. #Ollama is stepping up to meet this need with its official container image on Docker Hub, excelling in local #deployments. However, there’s a catch for containerized environments: downloading #models requires the ollama serve command to be running within the #container. This extra step complicates automated deployments via #CI #CD #pipelines, making seamless model access challenging. To take your #model to production, it needs to be #container-ready—a feature Ollama has yet to fully support. The gap in #AIOps for smooth deployments gets blurred with the inference services available as an easy way to take it to production. Goal is to overcome obstacles and deliver solutions 😉 so here is to get you started on having your (O)#llama #production ready in a private #deployment from your #local to #cloud all in #oneshot . Enjoy Medium: https://lnkd.in/g9jqBQ8d dev.to: https://lnkd.in/gi_gGwam #AI #LLM #Ollama #llama3 #phi3 #mistral #S6overlay #AIOps #CI #CD #oneshot #container #docker #podmanEnjoy

11

Ever wonder why some companies can ship software at lightning speeds while others seem stuck in the mud? 🚀⏳ . . . . . . . . . Daksh Gupta of Greptile collected real-world insights from engineers at many of the world's largest tech companies, unpacking the intricacies that either propel or impede the speed of shipping code. The big reveal? Older and larger companies tend to move more slowly because the costs of missteps become greater. In some cases, those costs are fairly non-negotiable (e.g. ensuring software for customers in production remains secure) while in other cases, the costs are more optional and cultural (e.g. wanting a lot of accurate estimates, forcing a manager's vs. maker's schedule, etc). #SoftwareEngineering #Tech #DevelopmentVelocity #Innovation #Productivity

1

In my podcast interview with Patrick Gallagher at ELC, we discussed several key topics: - How to identify product-minded engineer and tech-minded engineers and the value they add to various development streams - Communication framework for engineering leaders - Strategies for engineering leaders to accelerate AI adoption This is a podcast I find myself revisiting, as many of the insights we discussed are timeless. Podcast in Apple: https://lnkd.in/gn2VE6z8 Podcast in Spotify: https://lnkd.in/gy3fEPJ4

89

7 Comments

Oso makes this look easy. Historically you either put all of your data into your auth system which is easy, but creates availability and privacy issues, or else you distribute your data which was scalable and secure, but leads to spaghetti authorization logic and complexity for developers. Such a nice balance here. The CAP theorem of authorization?!

4

Reach: Who & How The first system or feature most people own is often built from scratch. My initial foray was developing a ranking model for Messenger-only users. Having built the component, I was naturally inclined and actively advocated for its application in broader contexts, such as Messenger contact recommendations. As an owner, you become the system’s champion—its impact reflects on you. Understanding the system's reach, whether it's advertisers or users segmented by Gross Merchandise Volume (GMV), is crucial. For internal systems, your primary users might be engineers and teams, but their users are your audience too. It’s vital to review user journeys. Ensuring intended users can utilize the system consistently without disruption is a baseline for sustained impact. Monitoring tools for metrics like user drop-off rates, retention, and adoption speed are essential. For internal systems with smaller user counts, supplementing data with qualitative research like surveys is invaluable. One of my first tasks when taking over new systems is a thorough review of these metrics. Regular monitoring aids early bug detection and sets development priorities. For example, while enhancing the Google Maps Notifications Infrastructure, I initiated end-to-end tracking for notification delivery. This revealed a higher than expected rate of dropped notifications, prompting immediate improvements. Once the system is stable and the user experience is effectively managed, expanding reach becomes more manageable. Discovering new user groups often requires collaboration with cross-functional partners. Occasionally, a system may experience rapid adoption. Skilled owners should capitalize on this momentum, making strategic trade-offs when necessary. For instance, implementing optimistic caching for certain user cohorts could be a solution when costs spike, with plans to optimize later. I faced this when Messenger-only user adoption of a contact recommendation service skyrocketed from 10 million to over 700 million users. Navigating this scale required several hardcoded optimizations and commitments to infrastructure engineers. Finally, increasing and maintaining reach isn't feasible without setting a minimum standard for reliability and cost. Higher usage often brings more bug reports and requires managing varied levels of user expertise, which can hinder wider adoption and engagement.

17

If you have built and operated an application or service that relies on an operational database in production, you will immediately understand the insane amount of value that these features deliver.

10

Insanely fast and free GPT-4 (nearly) equivalent? It’s here ⚡️ Go to https://lnkd.in/gF2AUcaq (Grok playground) and choose Llama3-70b I saw someone post about #Llama3 on #Groq and got confused, because I thought Llama3 was Meta and Groq was X -- Grok, with a “k,” is X, but Groq is a free multi-LLM tool. Ok, got it. You can just go to groq.com, but the playground is faster Here’s more about Groq https://lnkd.in/gjmbd9dd Note: Not the same offering as ChatGPT with file uploads, GPTs, etc. unless there’s something I’m missing

2

If you're running mission critical software in kernel space on your clients' machines, which updates itself remotely and can brick those servers, how can you not do health mediated, staged rollouts of remote updates? Seriously, why should we believe you're competent in other, much more complex aspects of your software? https://lnkd.in/gaEcQ4fh

25

2 Comments

Stop playing Tetris with your teams and projects. Investing heavily in meticulous planning to try and achieve the fantasy outcome of multiple teams, schedules, budgets, people, and projects aligning perfectly to meet your overly ambitious and tightly packed agenda is doomed to fail. The odds of that succeeding are slim-to-none yet, it's what we default to when there's so much we want to do. Do more by doing less! 1) Reduce WIP: Work on less things in parallel, not more. 2) Prioritize: Work on only the thing(s) that will deliver the most value the soonest. Will this guarantee success? No. But, the odds of success are orders of magnitude higher than trying to plan for a perfect game of Tetris. John Cutler's latest installment of The Beautiful Mess, "The Failure Threshold" explains this beautifully. How many of you have been in the following situation (pulled from his post): "If George and Ina each allocate 50% time to Project Thunder, and then Lenny does 10% time combined with moving the two open reqs to Team Apollo and hiring those roles in the next two months, we should—if we sort of combine Project Thunder and Project Inferno—be able to finish Thunder, Inferno, and chip away at Homestead so that when The Hex Pistols are done with the event streaming re-architecture and Project Kafka-esque, assuming Xenia approves the design document, we should be able to hit the deadline for Project Sycamore as promised in the 2023 annual plan."

12

New blog post: The little things that got me into stream processing (https://lnkd.in/gU7Ba5Ym)

60