Microsoft's Exchange Online PIR: the five runbook changes I'm making

November 22, 2026 · 17 min read

Microsoft's Exchange Online PIR: the five runbook changes I'm making

TL;DR - Microsoft's preliminary post-incident report for the November Exchange Online outage describes 23 hours 50 minutes of customer impact, from 16:20 GMT on Thursday 12 November to 11:10 ET on Friday 13 November, under incidents EX1550543 and MO1550821. Three conditions lined up: a latent lock-contention defect from a July 2026 update to the Microsoft 365 networking gateway, Teams failing onto Exchange Online's gateway about 15 hours earlier, and nearly 14% of that gateway's capacity already out of service. More than 80% of the gateway servers went unresponsive, failover was not possible for scale reasons, and rollback was not possible because the change was four months old. The final report was due 20 November. What you need to do: write down your own blind-window figure, map the layers your services share, test your failover assumptions against a shared dependency, and store a rollback state for your last significant change.

By The Numbers

FigureValue
Outage length, first customer impact to service restoration23 hours 50 minutes
Start of impact16:20 GMT Thursday 12 November (02:50 Friday, Adelaide)
End of active customer impact11:10 ET Friday 13 November (02:40 Saturday, Adelaide)
Gateway servers that stopped respondingmore than 80%
Capacity on the affected gateway already out of service beforehandnearly 14%
How long Teams had been failed onto that same gatewayabout 15 hours
Age of the defective code at the point it failedfour months (July 2026 update)
Email timeout window Microsoft extended to48 hours

I read Microsoft's post-incident report for the November Exchange Online outage the way I read a coroner's note, which is to say I go straight to the timeline. The cause of death is Microsoft's problem. The timeline is mine, because it says how long my organisation would have been blind before anyone noticed.

My tenants came through it. Nothing of mine fell over permanently, and nobody lost mail once the queues flushed. What I did lose was a working assumption I had been carrying for years, the one that says a Microsoft cloud failure is a single service falling over while everything else carries on. That assumption was wrong for nearly a day, and the report says so in the politest possible language.

I am not going to recap the news, because the news is everywhere and the r/sysadmin thread on the report hit 2,871 points and 568 comments this week with people doing that job already. What I want to do is narrower and more useful. I want to write down what I am changing in my own incident runbook, in the order I would hit it at 3am, and let you argue with the list.

Some of it is uncomfortable. Two of the five changes cost effort that my business continuity plan had quietly assumed I would not need.


Let me walk you through the timeline first, because the arithmetic is where the lesson hides, then the five changes and what each one replaces.

What actually happened, in the order it happened

The two incident numbers are EX1550543 for Exchange Online and MO1550821 for the wider Microsoft 365 impact. Same window, same root cause, two views of one event.

The preliminary report lists three conditions that had to line up, and each one on its own would have been survivable.

The first was a latent defect introduced in a July 2026 update to Microsoft's own Microsoft 365 networking gateway servers. The defect could cause lock contention under certain traffic patterns. It sat there for four months, doing nothing, which is what latent defects do until the conditions around them change.

The second was that Teams failed over onto the gateway Exchange Online was using, roughly fifteen hours before the outage started. Teams added load to a system that was already carrying Exchange traffic with a defect in it.

The third was that nearly 14% of that gateway's capacity was already out of service.

Put the three together and more than 80% of the gateway servers became unresponsive. Exchange Online could not fail over to other infrastructure because of scale, not because nobody thought of it. Restarted servers stopped responding again as soon as traffic came back to them, which is the detail that turns a restart into a loop. And the rollback that would normally be the first move was not available, because the change was four months old and the previous state had long since stopped being the previous state.

So engineers deployed an update that reverted the defective code path. Impact ended at 11:10 ET on Friday.

Microsoft also extended the email timeout window to 48 hours so that mail sitting in queues still arrived once things came back. And the company stated plainly that this was a technical issue, not related to cyber or to AI and agents. I want to come back to that sentence, because it is the most useful thing in the report for anyone running triage.

One honesty note. The preliminary report is what was published in the middle of November. The final report was due on 20 November and I have not been able to confirm its contents at the time of writing, so everything above is the preliminary version, and anything the final report revises takes precedence over this piece.

The arithmetic nobody did at the time

Here is where the ANZ timezone becomes an asset rather than an annoyance.

Impact began at 16:20 GMT on Thursday 12 November. Adelaide is UTC+10:30, so that is 02:50 on Friday morning. Impact ended at 16:10 GMT on Friday, which is 02:40 on Saturday morning.

Read those two numbers together. Every business hour of Friday 13 November in Adelaide, Melbourne, Sydney, Brisbane and Perth happened inside the outage window. The event started while the country was asleep and finished after the weekend had begun.

That matters because of who was awake. At 02:50 no one was watching a status page except the on-call rota, and the on-call rota in a small business is usually one person who is also the escalation path. By the time anyone opened a laptop at 7am, the outage was already four hours old, the queue depth was already substantial and the vendor had already published something. The team walking in that morning did not get a fresh incident. They inherited an incident with a four-hour head start.

That is the blind window, and it is not the same number as the outage duration. My blind window was about four hours. The outage was 23 hours 50 minutes. Those are different figures and they need different entries in the runbook, which is the first change on my list.

For anyone outside ANZ, run the same sum against your own timezone. The outage started at 11:20 in New York and 16:20 in London, both inside a working day, and both of those audiences got to watch it happen in real time. ANZ got the overnight version, which is the harder version to staff.

What broke in the reasoning, not just the platform

Before the list, the part I found genuinely arresting.

The normal mental model for a hyperscaler outage is one service failing. Email is down, Teams still works, you tell people to message instead. That model held for years and it is why "we'll just use the other tool" is such a common line in business continuity plans, including one I have written myself.

The report describes the opposite. Teams failed onto the Exchange gateway. The gateway was the shared dependency. When the gateway went, the fallback channel went with the primary, and the shared dependency was Microsoft's own network layer rather than a product that appears on a service health dashboard.

That is a dependency question, not a product question, and it is exactly the question I have been lazy about. My vendor outage plan lists Microsoft 365 as one line item, as though Exchange, Teams, SharePoint, Copilot, Defender for Office 365 and Intune were six separate risks that could be handled one at a time. The outage took all six down in the same 23 hours because they shared something below the product layer.

So when someone asks me now why the fallback channel died at the same moment as the primary, I have a real answer: because they were never separate. That answer applies to my own estate too, and it is the uncomfortable part of this whole exercise.

The five runbook changes

1. Put a blind-window number in the runbook, next to the outage number. My runbook had a line about acceptable downtime. It did not have a line about time to first useful signal. Those are different. The question to answer on paper, before the next one, is: if Exchange stopped accepting mail at 02:50, at what minute would my organisation know it was a platform failure rather than our own mail flow rule, connector or certificate? For most small tenants the honest answer is "when someone arrives and notices", which is measured in hours. Once you know the number, you can shrink it. A second channel that is genuinely independent, a status page subscription that goes to a phone rather than an inbox, and a rule that says any report of "email is weird" gets checked against the tenant health page before anything else. My target is under 30 minutes, and I am not there yet.

2. Rebuild the dependency map around shared infrastructure, not products. List every service you depend on, then draw the lines between them that your vendor does not publish. In this event the line was Teams and Exchange sitting on the same networking gateway. In your own estate the equivalent lines are shared identity providers, shared DNS, shared certificate authorities, shared internet breakouts, shared on-premises hybrid connectors and shared backup infrastructure. If two "independent" services share one of those, they are one service for the purposes of an outage plan, and the plan should say so. Mine now does, on one page, with the shared things circled.

3. Set your own capacity headroom rule, and pick a number you can defend. The report says nearly 14% of the gateway capacity was already out of service and more than 80% of the servers became unresponsive under additional load. I am not going to second-guess Microsoft's capacity planning from the outside, because I do not have the traffic data. What I can do is apply the shape of it to my own kit. If 14% of my capacity went away and demand rose by the amount that Teams added, would my remaining capacity survive? For most SMB estates the answer involves a single appliance, a single firewall pair or a single internet circuit, all of which fail the test. Write down the number you are running at, so that "we have headroom" becomes a percentage rather than a feeling.

4. Separate change rollback from change approval. The rollback was impossible because the change was months old, and that is not a Microsoft-only problem. It is a problem of how long you keep the thing you would roll back to. In my own environment, the July configuration that I would need in November is often gone: the previous firewall rule set overwritten, the old container image untagged, the previous template exported and then edited in place. So the rule I am adding is that any change which could break something months later needs a stored, testable previous state and a note on where it is. That is more work per change. It also would have saved Microsoft, by their own account, from a bad night.

5. Write down what "not a cyber incident" means for triage. Microsoft stated this was a technical issue not related to cyber or AI and agents, and that sentence is worth ten minutes of the next incident, not because vendors are always right, but because it tells you which of the two triage trees you are climbing. My runbook now has an explicit fork: before minute fifteen, decide whether you are in a threat-hunting incident or a platform availability incident, and write down the evidence that decided it. The failure mode this prevents is a small team spending the first hour looking for a compromise when the actual answer is that the vendor is having a bad day. During a genuine platform outage, hunting for attackers is not diligence. It is delay.

Why this report is better than the ones we used to get

I want to give credit where it is due, because most post-incident reports from hyperscalers read like they were written by a committee whose first priority was not being quoted.

This one names three specific conditions and states that all three had to line up. It says the rollback was not possible and explains why. It admits that an earlier failover event contributed to the failure, which is the kind of admission that costs a company something internally. It gives a duration with both timestamps rather than an approximate window. And it states the 48-hour email timeout extension, which is a customer-facing detail that actually changes what a mail admin should do in the aftermath.

For an IT manager at an SMB, that level of detail is the difference between a report you can act on and a report you read to find out whose fault it was. This is the former.

The r/sysadmin thread at 2,871 points tells you the same thing. When a report is honest, the audience does something more useful than complain. It starts quoting the timeline back at each other and comparing it to their own.

Where I am still uneasy

The part that nags at me is the four months. A defect introduced in July activated in November because two unrelated conditions happened to land on it. No amount of testing at the time of the change would reliably have caught that, and I do not think "test harder" is a useful lesson.

What it does mean is that the risk of any large platform accumulates quietly in the gap between when a change ships and when it meets the traffic that breaks it. I have the same exposure in my own estate, in the form of the change I made in August that has been fine ever since and might have a defect that only one specific workload can trigger. The difference is that when mine goes wrong it takes down one business's mail flow rather than a continent's, and nobody writes a report about it.

The other thing I am uneasy about is how quickly "not related to cyber" becomes a reason to skip verification. Microsoft's statement is credible in this case. It is not a rule. The correct handling is to record the vendor's attribution, act on the availability incident, and revisit the security question once service is restored. Two of my clients have asked me this week whether the outage was an attack, which tells me the statement did not travel as far as the timeline did.

What to do right now

  1. Find the exact timestamps from your own tenant, not the news summary. Microsoft published the tenant-specific incident records under EX1550543 and MO1550821. Note when your impact started and ended, and compare it with the global window. Tenants did not all experience the same 23 hours 50 minutes.
  2. Write your blind-window number down. How long before somebody in your organisation would have known this was a platform failure? Be honest. If the answer is "when staff arrive", say so, and then fix the channel.
  3. Check your mail queues and your timeout assumptions. Microsoft extended its email timeout window to 48 hours for this event. That means some mail arrived much later than your users expected. Tell staff the delay window explicitly next time rather than letting them resend things manually and double up.
  4. Update the read of your own dependency map, one page only. Circle every shared thing that two or more services sit on. Identity, DNS, certificates, internet breakout, hybrid connectors, backups. Anything circled is one failure, not two.
  5. Test the fallback channel that was already affected. Teams failed onto the same gateway. Whatever you use as the emergency channel, ask what it shares with the primary, and if the answer is "the network stack", write down a third option that shares nothing.
  6. Store a rollback state for your last significant change. Pick the most recent change that touches a shared dependency and check whether you could actually revert it tonight. If you could not, that is this week's job.
  7. Give the final report a calendar entry. It was due on 20 November. Read it when it lands and revise anything above that it contradicts, because the preliminary report is a first account, not a verdict.

Key Takeaways

  • The outage ran 23 hours 50 minutes from 16:20 GMT on Thursday 12 November to 11:10 ET on Friday 13 November, which covered every ANZ business hour that Friday.
  • Three conditions had to line up: a latent lock-contention defect from a July 2026 gateway update, Teams failing onto the same gateway about 15 hours earlier, and nearly 14% of that gateway's capacity already out of service.
  • More than 80% of the gateway servers went unresponsive, Exchange Online could not fail over for scale reasons, and restarted servers stopped responding again when traffic returned.
  • Rollback was not available because the change was four months old, so the fix was an update that reverted the defective code path.
  • Your blind window is not your outage duration. Mine was about four hours, and that four-hour figure is the one the runbook needs to attack.
  • The fallback channel failed with the primary because both sat on the same shared layer. That is a dependency problem, not a product problem, and your own estate has the same lines in it.
  • Microsoft called it a technical issue not related to cyber or AI, which is useful triage information and not a reason to skip verification.

FAQ

How long did the November 2026 Exchange Online outage last? 23 hours 50 minutes. Customer impact began at 16:20 GMT on Thursday 12 November and ended at 11:10 ET on Friday 13 November. The tenant-level incident numbers were EX1550543 and MO1550821.

Which Microsoft services were affected? Outlook and Exchange Online, Teams, Copilot, SharePoint, Defender for Office 365 and Intune, worldwide. The shared element was Microsoft's Microsoft 365 networking gateway layer rather than any one product.

Was it a cyber attack? Microsoft stated it was a technical issue not related to cyber or AI and agents. The preliminary report describes a latent software defect activated by a combination of failover load and reduced capacity, with no security component.

What was the actual root cause? Three conditions together. A latent defect from a July 2026 update that could cause lock contention, Teams failing onto Exchange Online's gateway about 15 hours before impact began, and nearly 14% of that gateway's capacity already out of service. The combination left more than 80% of gateway servers unresponsive.

Why could Microsoft not just roll back the change? Because the change was four months old. Rolling back a gateway update that had been live since July was not a safe operation, so engineers deployed a new update that reverted the defective code path instead.

Was any email lost? Microsoft extended the email timeout window to 48 hours so delayed mail was delivered once service returned. Mail sat in queues rather than being discarded, which is why some messages arrived hours late.

Where is the final report? It was due on 20 November 2026. Anything the final report revises takes precedence over the preliminary account this piece is based on.

What is the single change most IT managers should make? Write down your blind-window number. Not your acceptable downtime, the time before you would know it was a platform failure. Most SMB tenants discover it is measured in hours, and that is the number worth shrinking.

My Take

The instinct after an outage like this is to treat it as a Microsoft problem and move on, because there is nothing you can do about a gateway defect four months deep in somebody else's network. That instinct is mostly right and entirely useless as a takeaway, because the part you can act on is not the defect. It is the hour before you knew about it, and the assumption you were carrying about which systems share what.

Here is the assumption I would like to see retired. It is the one where a cloud provider's services are treated as independent risks that fail one at a time. That model is convenient because it lets a business continuity plan fit on a page, and it is wrong for the same reason it is convenient. The things that fail together are the things that are built on the same foundation, and vendor product boundaries have nothing to do with that.

The 23 hours 50 minutes is a number I will remember. What I will actually act on is smaller and duller: a blind-window figure, a one-page map with the shared things circled, a stored rollback state per significant change, and a triage fork written down so that a small team stops hunting for attackers during a platform outage.

None of those would have prevented this. All of them shorten the next one.

Mathew Clark Founder, SecureInSeconds Currently: rewriting one page of a runbook and circling everything that shares a network path Drafted with AI assistance - read, verified, and fixed by a human.

Further Reading

Share:
Buy me a coffee

You might also like