Incident sheet · /infrastructure

Filed
Read
3 min · 593 words
Severity
Routine

Microsoft 365 outage traced to three-month-old gateway bug

Microsoft traced last Friday’s global Microsoft 365 outage to a gateway lock-contention defect, compounded by a Teams failover and nearly 14% of capacity already offline.

Microsoft says three-month-old bug behind global M365 outage
FeedMicrosoft says three-month-old bug behind global M365 outage — PIRO4D / pixabay

Event log

  1. More than 80% of gateway servers became unresponsive in less than two hours.
  2. Nearly 14% of gateway capacity was out of service when Teams failed over about 15 hours before the outage.
  3. The gateway fix began rolling out in July and ran worldwide for more than three months without problems.
  4. Engineers restored the previous code in large server blocks, in some cases covering more than half a region.
  5. A final post-incident review was promised for later in the week.

By Amara Osei

More than 80% of Microsoft’s networking gateway servers became unresponsive in less than two hours during last Friday’s lengthy global Microsoft 365 outage. Microsoft attributed the disruption to a lock-contention defect that had remained hidden for more than three months.

Microsoft executive vice-president of Copilot, agents and platform Charles Lamanna wrote on social media that the defect sat inside an internal process and made servers unresponsive under certain load conditions.

Lock contention occurs when many tasks queue for the same shared resource, so work stalls while they wait. Microsoft introduced the defect through a fix written specifically for the gateway servers.

The fix began rolling out in July and ran worldwide for more than three months without problems. That combination—a long-running change and no immediate symptom—delayed detection before last Friday’s failure.

Takeaway: A gateway-specific fix created the failure condition, while production exposed what testing missed.

What did Microsoft’s testing miss?

Microsoft’s lab load testing missed the issue because the test setup concentrated on weaker hardware with fewer processor cores. The contention proved worse on newer machines with more cores.

Lamanna said: “Neither the underlying defect nor the fix applies to broader Windows deployments.”

The hardware mismatch allowed the defect to remain dormant until production traffic and newer infrastructure exposed conditions the laboratory setup did not represent.

Takeaway: The gateway fix passed lab tests but met newer hardware and production traffic that the test environment did not represent.

How did the outage reach global scale?

Microsoft entered the outage with reduced capacity and then shifted traffic onto the same gateway infrastructure. The sequence unfolded in four steps:

  • An unrelated problem on the previous day left nearly 14% of gateway capacity out of service, roughly one-seventh of the available pool.
  • About 15 hours before the outage—roughly two-thirds of a day—Teams failed over from a separate gateway system onto the primary gateway, which also serves Exchange Online. The move added load and changed traffic patterns.
  • Servers began failing during peak United States traffic. As traffic management steered users away from failing servers, the remaining machines became overloaded.
  • In less than two hours, the cascade made more than 80% of gateway servers unresponsive, or more than four in five.

Microsoft said isolation between the gateway’s thousands of capacity units and its regions should have contained the damage. It did not, because of a separate issue that Lamanna did not describe.

Takeaway: Several faults lined up: a dormant defect, reduced capacity, a traffic shift and isolation that did not hold.

Why did recovery take so long?

Teams, OneDrive, SharePoint and Office apps could fail out of the faulty gateway and recovered earlier. Exchange Online could not use that route, leaving it dependent on the impaired infrastructure while other services returned.

Restored servers were immediately swamped by waiting traffic. Microsoft’s automated recovery processes also were not built for a failure across such a broad server base.

Lamanna wrote: “Additionally, since the main code defect was a latent bug from several months previous, our normal rollback, flighting, and feature flags were unable to quickly revert the change.”

Engineers instead built an update that returned the code to its previous state. They applied it to large blocks of servers at a time, in some cases covering more than half of a region. Lamanna said a final post-incident review would provide more detail later that week.

Takeaway: Recovery required a new rollback update applied in large blocks after standard controls failed.

via itnews.com.au (Source)

Filed under

  • microsoft-365
  • microsoft-teams
  • exchange-online
  • gateway-infrastructure
  • lock-contention
Share this article:

More from Amara Osei

Amara Osei

Show full bio

Correspondent covering regulation, policy and their impact on enterprise information technology and the leaders who run at CIO Monitor.

188 articles

The newsroom digest

One email a day with the stories that matter. No noise.

By subscribing you agree to the privacy policy.

Cross-references

‹ Previous articleNext article ›