Incident sheet · /infrastructure
- System
- Infrastructure & Cloud
- Filed
- Read
- 3 min · 593 words
- Severity
- Routine
Microsoft 365 outage traced to three-month-old gateway bug
Microsoft traced last Friday’s global Microsoft 365 outage to a gateway lock-contention defect, compounded by a Teams failover and nearly 14% of capacity already offline.

Event log
- More than 80% of gateway servers became unresponsive in less than two hours.
- Nearly 14% of gateway capacity was out of service when Teams failed over about 15 hours before the outage.
- The gateway fix began rolling out in July and ran worldwide for more than three months without problems.
- Engineers restored the previous code in large server blocks, in some cases covering more than half a region.
- A final post-incident review was promised for later in the week.
By Amara Osei
More than 80% of Microsoft’s networking gateway servers became unresponsive in less than two hours during last Friday’s lengthy global Microsoft 365 outage. Microsoft attributed the disruption to a lock-contention defect that had remained hidden for more than three months.
Microsoft executive vice-president of Copilot, agents and platform Charles Lamanna wrote on social media that the defect sat inside an internal process and made servers unresponsive under certain load conditions.
Lock contention occurs when many tasks queue for the same shared resource, so work stalls while they wait. Microsoft introduced the defect through a fix written specifically for the gateway servers.
The fix began rolling out in July and ran worldwide for more than three months without problems. That combination—a long-running change and no immediate symptom—delayed detection before last Friday’s failure.
Takeaway: A gateway-specific fix created the failure condition, while production exposed what testing missed.
What did Microsoft’s testing miss?
Microsoft’s lab load testing missed the issue because the test setup concentrated on weaker hardware with fewer processor cores. The contention proved worse on newer machines with more cores.
Lamanna said: “Neither the underlying defect nor the fix applies to broader Windows deployments.”
The hardware mismatch allowed the defect to remain dormant until production traffic and newer infrastructure exposed conditions the laboratory setup did not represent.
Takeaway: The gateway fix passed lab tests but met newer hardware and production traffic that the test environment did not represent.
How did the outage reach global scale?
Microsoft entered the outage with reduced capacity and then shifted traffic onto the same gateway infrastructure. The sequence unfolded in four steps:
- An unrelated problem on the previous day left nearly 14% of gateway capacity out of service, roughly one-seventh of the available pool.
- About 15 hours before the outage—roughly two-thirds of a day—Teams failed over from a separate gateway system onto the primary gateway, which also serves Exchange Online. The move added load and changed traffic patterns.
- Servers began failing during peak United States traffic. As traffic management steered users away from failing servers, the remaining machines became overloaded.
- In less than two hours, the cascade made more than 80% of gateway servers unresponsive, or more than four in five.
Microsoft said isolation between the gateway’s thousands of capacity units and its regions should have contained the damage. It did not, because of a separate issue that Lamanna did not describe.
Takeaway: Several faults lined up: a dormant defect, reduced capacity, a traffic shift and isolation that did not hold.
Why did recovery take so long?
Teams, OneDrive, SharePoint and Office apps could fail out of the faulty gateway and recovered earlier. Exchange Online could not use that route, leaving it dependent on the impaired infrastructure while other services returned.
Restored servers were immediately swamped by waiting traffic. Microsoft’s automated recovery processes also were not built for a failure across such a broad server base.
Lamanna wrote: “Additionally, since the main code defect was a latent bug from several months previous, our normal rollback, flighting, and feature flags were unable to quickly revert the change.”
Engineers instead built an update that returned the code to its previous state. They applied it to large blocks of servers at a time, in some cases covering more than half of a region. Lamanna said a final post-incident review would provide more detail later that week.
Takeaway: Recovery required a new rollback update applied in large blocks after standard controls failed.
via itnews.com.au (Source)
More from Amara Osei
Show full bio
Correspondent covering regulation, policy and their impact on enterprise information technology and the leaders who run at CIO Monitor.
188 articles
The newsroom digest
One email a day with the stories that matter. No noise.
Cross-references
- Microsoft outage report: latent July bug, mismatched test labs felled 80% of gateway servers
- Microsoft 365 outage traced to July networking gateway bug
- Microsoft 365 outage: 80% of gateway servers down, security gaps under review
- Microsoft's Global Outage: A $7 Billion Lesson in Cloud Concentration Risk
- 24 Hours Down, Zero Defections: Why Microsoft Outages Don't Cost It Customers