Once every six months or so, the internet seems to shut down. It might be a network service provider going down, like Rogers or Verizon. Or you might remember when, in the summer of 2024, it seemed like half the internet was down because a security platform called CrowdStrike had an outage. Suddenly people couldn’t work. This week, it was Microsoft Outlook, and entire firms struggled to operate without that basic tool.
The best ability is availability.
Sports fans know this line very well. If your star player gets injured all the time, there’s a cost to that, and it might make sense for the team to pick up a player who’s only 80 per cent as good but never gets hurt. We can think of software and computer systems the same way. We expect our phones and tools to work all the time, but in reality, things are constantly breaking. It’s often not noticeable because it happens in small ways that are hard to detect. One of the most common is when a web page doesn’t load, so you reload it, and all of a sudden it works. Turning it off and on again is often the fix, but something went wrong under the hood.
In technology, this is referred to as availability. While in sports a basketball star should be playing nine out of 10 games (90 per cent), technology operates on a different order of magnitude. Both should attempt to be available as close to 100 per cent of the time as possible. That means instead of nine out of 10, it needs to be 99.9 out of 100 or better.
People who work in software and computer engineering call this “chasing 9s.” An organization like Microsoft is going to be trying to get 99.999 % or better. It seems like a small difference, but to put that in perspective, 99.9% uptime means about 43 minutes of downtime a month. 99.999% means just 26 seconds. A one-day outage, like the one Microsoft had, brings monthly metrics down to about 96.67% availability on its own. A great score on your math test, but not a business metric a key financial service like a payment processor would ever be proud of.
Site Reliability Engineers
To get that next “9”, technology orgs employ a whole discipline of software engineering called site reliability engineering. Site reliability engineers (SREs) are the ones tasked with availability. You can think of them sort of like tech firefighters or IT paramedics. The specific discipline of putting out the fires is referred to as major or critical incident management.
These people’s job is essentially to investigate what is actually happening, identify the root cause, and then develop or coordinate the software fixes that will mitigate and ultimately resolve the incident.
Step one in determining the root cause is typically to ask what has changed. It could be a code deployment, a networking switch, or even a heavier-than-normal load on a company’s website — imagine a rare and valuable product going on sale or a concert being announced. Oftentimes, systems built years ago can’t support the weight of traffic coming to them, especially in the age of LLMs, AI, and bots. This is magnified because synthetic traffic can hit much harder than natural traffic.
When will it come back online?
If there are entire teams dedicated to fixing this stuff, why is there almost never an ETA for when it will be back? Microsoft has a public status page you can access here, where they will tell you very little about what’s going on. They probably had a hunch early on about the root cause, and internal estimates for when they planned to be back online.
So why wouldn’t they tell the public?
The answer is that while they would love to communicate this to you, there’s no value in Microsoft communicating an ETA to its customers. There is no winning if they do. If they know what the problem is, they’ll be criticized for not fixing it faster. If they get it wrong, that’s actually much worse than if they never said anything at all. Either way, Microsoft looks bad. This is why it’s extremely rare for organizations to say, “Yes, we think we’ll be back online in an hour.” The most you’ll usually get is “we’re working on it.”
The key is the time it takes for a fix. Some outages can be fixed in seconds or minutes, oftentimes before anyone notices. Others, like certain networking outages, can take hours or days. Without getting too in-depth, we can think of it like the difference between turning around or backing up a canoe versus a battlecruiser. The canoe is simple and nimble, and you can pivot quickly. Turning around a battlecruiser is not a simple feat. You might think the fix is to turn the cruiser left, but you won’t know if that’s correct until you’ve finished turning. You might be wrong and have to double all the way back and then try to turn it right.
AI and the workforce
AI may or may not be a contributing factor in this instance. It probably is being leveraged by Microsoft, but not to the extent you might think. Where AI might matter more is the way it can encourage, or provide cover for, organizations like Microsoft to reduce the size and capacity of their incident response teams.
When orgs are concerned about their bottom line, these teams are at risk because they’re strictly a cost centre. In other words, they don’t bring in any revenue, and it’s difficult for them to demonstrate value (until Microsoft Outlook goes down globally).
AI can do some pretty fantastic work in this field. Companies will have internal LLMs that can crawl their knowledge bases and help engineers who aren’t experts in incident response and disaster recovery do the job. The problem is that in some instances, the AI won’t be able to identify what’s going wrong, or won’t be able to implement the fix.
You can think of it like a firefighter replaced by a robot, but in this instance, the robot can’t climb a certain set of stairs because they’re too steep. So what happens when the house at the top of those stairs catches fire? This is a calculation companies make all the time now.
So what can you actually do about it?
If you or your organization leverage technology as part of your day-to-day life, this risk exists for you. It exists in practically every software or internet-adjacent tool you use, if it’s not stored on your computer or phone. That includes social media, your internet service provider, even the ability to send emails.
The first tangible step is limiting exposure with diversification. This is difficult when it’s something like Microsoft Outlook going down, but diversifying and having backup switches where possible is the best way to prevent this. Think of it like having a backup generator at your house. An easy-to-understand example is to have a different phone provider than your internet provider. That way, if Rogers goes down, Bell will likely still be up.
Save backups. Back up what you can to your personal devices, hard drives, etc. This seems laborious in the age of cloud software, but it could be the difference between losing your family photos and not.
Sophisticated organizations will have explicit disaster recovery plans. The same way you’ll have a fire drill protocol on the wall of the office, teams will have whole repositories of disaster recovery plans for what to do if a certain service goes down. If it’s email that goes down, you might say, “Let’s all use WhatsApp,” or “Let’s all use Google Drive.” Developing, maintaining, and testing these plans is another key function of site reliability engineers.
At the end of the day, this is about how you choose to manage risk, individually and for your organization. You might be okay if you can’t send an email for 24 hours. Many got a free afternoon off. But some people were inevitably in some pretty high stakes situations: maybe they were approaching a big deadline for school or their job, or trying to send a vital communication about closing on a mortgage or a major purchase. That’s what’s ultimately at risk.