Incident sheet · /infrastructure

Filed
Read
3 min · 600 words
Severity
Routine

Microsoft 365 outage traced to July networking gateway bug

A July Microsoft 365 gateway update combined with a Teams failover and nearly 14% offline capacity to leave more than 80% of servers unresponsive worldwide.

Microsoft says months-old bug caused global Microsoft 365 outage
FeedMicrosoft says months-old bug caused global Microsoft 365 outage — Tho-Ge / pixabay

By Tom Whitfield

More than 80% of Microsoft 365 networking gateway servers became unresponsive during a nearly 24-hour disruption that began at 11:20 a.m. ET on Thursday, November 12. Microsoft ended active customer impact at 11:10 a.m. ET on Friday, November 13, when it restored the affected services.

In a preliminary post-incident report shared through the Microsoft 365 Admin Center, Microsoft described a worldwide failure affecting Outlook, Exchange Online, Teams and Copilot. Users could not access email, and significant delivery delays left messages in transit.

Microsoft put the scale this way: "In less than two hours, the previously unknown defect was encountered across our fleet and rendered them unresponsive. At its lowest, more than 80% of servers were unresponsive."

A three-part failure

Microsoft assigned the incident to three problems:

  • A previously undetected software bug.
  • Increased traffic after a Microsoft Teams failover.
  • Reduced gateway server capacity.

A July update developed for Microsoft 365 networking gateway servers introduced the defect. It had been deployed for more than three months without incident, but under certain traffic conditions it could overwhelm servers and stop them responding. Production traffic eventually supplied the stress case the lab tests had missed.

About 15 hours before the outage, Microsoft Teams automatically switched to the networking infrastructure used by Exchange Online. That failover increased traffic on the gateway servers. Teams also experienced a separate outage earlier that day; Microsoft said it could not establish whether the incidents were related.

Nearly 14% of the gateway's server capacity was already offline because of an unrelated issue. Microsoft said the infrastructure had sufficient capacity to handle that reduction by itself. Extra traffic and reduced capacity together triggered the defect, turning a capacity reduction into a global service failure.

A cascade the safeguards did not stop

As servers stopped responding, Microsoft's systems redirected users to other available servers. Those servers also became overwhelmed by the additional traffic and failed. Safeguards intended to prevent problems spreading between regions also failed, allowing the disruption to spread worldwide.

Some Microsoft 365 services, including Teams, OneDrive and SharePoint, could switch to alternative infrastructure and recover earlier. Exchange Online could not make the same switch, leaving users unable to access their mailboxes.

Testing missed the production configuration

Microsoft said its tests used less powerful hardware than production servers. The company attributed the missed detection to differences in traffic mix and SKUs between its laboratories and the installed fleet.

The contention issue worsened on machines with more cores and newer hardware, while Microsoft's load tests tended toward weaker hardware with fewer cores. It had tested the update; it had not reproduced the production configuration that failed.

Recovery met the backed-up queue

Restarting affected servers quickly reproduced the overload. They became overwhelmed again by backed-up traffic, which made recovery difficult. After identifying the defect, Microsoft engineers developed a fix and deployed it across affected servers, allowing services to stabilize throughout Friday.

What does this change for Microsoft 365 administrators?

Microsoft has since increased capacity, improved monitoring and added safeguards to prevent traffic redirection from triggering similar cascading failures. For administrators, those changes provide the immediate operational difference; the broader root-cause account remains provisional while Microsoft continues its investigation.

Charles Lamanna, Microsoft's executive vice president of Copilot, Agents, and Platform, apologized in a separate statement on X: "We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations." Microsoft said the investigation continues and expects to publish the final post-incident report later this week.

via bleepingcomputer.com (Source)

Filed under

  • microsoft-365
  • networking-gateway
  • service-outage
  • microsoft-teams
  • cloud-resilience
Share this article:

More from Tom Whitfield

Tom Whitfield

Show full bio

Market editor covering people, launches and day-to-day practice in enterprise information technology and the leaders who run at CIO Monitor.

185 articles

The newsroom digest

One email a day with the stories that matter. No noise.

By subscribing you agree to the privacy policy.

Cross-references

‹ Previous articleNext article ›