Microsoft admitted that the global outage in the company’s network infrastructure, which lasted almost five hours, was caused by a mistake by network engineers when making changes to the configuration of WAN routers. Change IP addresses WAN routers was done with a command that was not thoroughly tested and resulted in different behavior on different network devices,” Microsoft explained

The company said network engineers made changes to the configuration of the routers as part of a planned effort to update the IP addresses on the WAN routers. However, due to improperly implemented configuration changes, Microsoft WAN routers began to cascade down and disconnect from other routers in the WAN. Updated routing tables on devices removed Microsoft Autonomous Systems and traffic management systems from forwarding to optimize WAN traffic

On January 25, 2023, Microsoft managed to restore the operation of all its main services 4 hours after a global outage. The incident affected millions of users around the world, as the cause was an error in the company’s network infrastructure.

Customers and users were unable to access multiple Microsoft services during the incident, such as Azure, Microsoft 365, Microsoft Teams, Exchange Online, Outlook, SharePoint Online, OneDrive for Business, Microsoft Graph, Microsoft Store, Xbox, Minecraft, and VS Code.

“Between 07:05 UTC and 12:43 UTC on January 25, 2023, customers experienced network connectivity issues manifesting as long network latency and/or timeouts when trying to connect to resources hosted in Azure regions, as well as other services Microsoft, including Microsoft 365 and Power Platform” – says in an interim report from Microsoft released today.

After a global outage, the company’s network engineers manually rolled back the previously made changes and all services were successfully restored. In connection with the incident, Microsoft has decided to prohibit any changes to the configuration of network devices that do not comply with security recommendations and have not been tested on the entire fleet of network equipment.

Incorrect changes to Microsoft’s network settings caused the company’s autonomous system to lose communications with 47 other ASNs, including AS701 UUNET (Verizon), reported experts from Qrator. The disruption was felt all over the world.