You're dealing with service disruptions in multiple cloud platforms. How do you decide what to tackle first?

When dealing with service disruptions across multiple cloud platforms, prioritizing what to tackle first requires a methodical approach to minimize downtime and impact. Start by assessing the severity and scope of each disruption. Prioritize issues based on their impact on critical business operations and the number of users affected. Implement an incident response plan that includes clear communication with stakeholders about the status and expected resolution time. Utilize automated monitoring tools to quickly identify and diagnose the root causes. This strategic prioritization ensures that resources are allocated efficiently, addressing the most critical disruptions first to restore services swiftly and effectively.

As a Site Reliability Engineer, I'm expected to maintain 99% uptime on a monthly basis. Every disruption is one too many as customers can be very unforgiving and churn to show their distaste. To assess the impact, we have a pro-active plan where we already configured cloud trail to inspect logs and once there is a failure, I get predefined codes via pagerduty alerts if I'm on-call. These codes already exist in our internal documentation and makes troubleshooting faster as I can easily assess the cause, estimated time of resolution, impact as well as methods of resolution. Metrics instantly lead me to quantifying data which primary determines the magnitude of the impact of the disruption.

At a tech startup, we faced a critical disruption in our payment processing system, impacting a large portion of our user base. We swiftly evaluated the impact based on user feedback, financial implications, and service criticality. Recognizing the revenue impact, we prioritized restoring the payment system over less critical issues. This strategic assessment enabled us to mitigate customer dissatisfaction and financial losses effectively, highlighting the importance of prioritizing disruptions based on their operational impact.

Whichever service is the most 'disruptive' = costs the most money. Get back online as fast as you can. Review why it happened. Consider alternative vendors if disruptions are recurring and short term migration pain enables long term uptime.

Impact assessment should always be the first step. If you don't know which system is most critical to the organization, you won't know which one to fix first.

As part of our subscription agreements, we share Service Level Agreements with our customers. Since we do not have control of our cloud vendor(AWS) we have to the same SLA from them. We then make sure we respond to incidents or support tickets in compliance with these SLAs. When we fail, we compensate our customers accordingly. I rely primarily on Solarwinds and Kingdom to measure fulfilments of SLAs but there are other alternatives like ServiceNow and Nagios

In addition to assessing the impact, the SLA with your customers and between your cloud provider and your organization is another variable to consider. I will prioritize resolving the service that generates more penalties with my customers.

I update our status page with Status.io to make sure that our customers are aware of downtimes as soon as they occur and are informed according once service is restored. In status.io, I already have a set of templates indicating the kind of disruption, estimated time to resolve and apologies for the inconvenience. This saves a ton of time especially when time is most crucial. There are also broadcasts that are sent during scheduled maintenances and after post-mortems.

The communication plan is something you need to define, understand, and communicate within your organization, it is critical to know who is the communication channel with the cloud provider and who is the communication channel with our internal and external customers.

Many resolutions are not straightforward. As a result, we sometimes roll-back to a previous update while we fix the underlying issue. This implies that we already have backups in place and several mitigation steps. Things that help fix highly complex disruptions: Teamwork: Two Site reliability engineers are better than one. Proper Documentation: This is proactive documentation of already known issues an how to fix them Subject matter Expertise of stack: Knowing the stack helps a ton and you're more likely to understand the cause and fix it on time.

It's better to have at least one person on-call. I believe two is even better and have a proper way of alerting them. I rely on pager duty and this generates slack alerts, phone calls, emails and a notification on the mobile app. Even if you're not at alert, thanks to Pagerduty, you will be. We also have an on-call rotation and everyone already knows what time they have to be available to avoid prolonged disruptions.

The title of this article is about "service disruptions in multiple cloud platforms." The chances of multiple cloud platforms (like AWS, Azure, OCI, and GCP) simultaneously experiencing service disruptions is very low. The chance of multiple cloud platforms being used by your workload (in a multi-cloud environment) all having disruptions in the regions and services that simultaneously impact your workload is astronomically small.