Microsoft said a software defect introduced in a July update to its network-gateway servers caused last week's worldwide Microsoft 365 outage, according to a preliminary post-incident review described by The Guardian (opens in new tab), Ars Technica (opens in new tab) and BleepingComputer (opens in new tab). Ars Technica (opens in new tab) said Microsoft's report listed the incident as "a technical issue not related to cyber or AI/agents." Charles Lamanna, Microsoft's executive vice president of Copilot, Agents, and Platform, wrote on X on Monday that a latent July bug caused servers to "become overwhelmed and unresponsive," The Guardian (opens in new tab) reported.

Lamanna said Microsoft typically runs two networking gateways and that a Microsoft Teams failover onto the primary gateway, together with over 14% of gateway server capacity already being out, caused the network to "encounter the latent bug from July." "Taken together, the increased load and the reduced server capacity, caused the network gateway to encounter the latent bug from July," he wrote, according to The Guardian (opens in new tab). Ars Technica (opens in new tab), describing the review, said the defect could exhaust CPU and memory and left the gateways with lock contention. BleepingComputer (opens in new tab) quoted the review as saying "the contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on weaker hardware with fewer cores."

As servers failed, traffic was shifted onto other machines that then failed as well. "At its lowest, more than 80% of servers were unresponsive," the review said, according to BleepingComputer (opens in new tab). Services such as Teams, OneDrive and SharePoint could be moved off the faulty gateway; Exchange Online could not, which prolonged the email disruption. A simple rollback was not possible because the change was months old. Microsoft said it applied an update that restored a previously working code path and that customer impact had ended by 11:10 a.m. ET on Nov. 13, BleepingComputer (opens in new tab) reported. A final post-incident report is expected on Nov. 20, Ars Technica (opens in new tab) and BleepingComputer (opens in new tab) reported.

The outage, which began on Nov. 12, disrupted Outlook, Exchange Online, Teams and other Microsoft 365 services worldwide. Lamanna apologized, saying, "We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations," according to BleepingComputer (opens in new tab).