08:00 UTC | 2024-Jul-19
WATS accounts located in Central US may have experienced problems during the hours in the timeline below. We will keep monitoring the service continuously.
From Microsoft:
Service: App Service
Region: Central US
Impact Statement: Starting at 22:16 UTC on 18 Jul 2024, you have been identified as a customer using App Service in Central US who may receive intermittent HTTP 500-level response codes, experience timeouts or high latency when accessing App Service (Web, Mobile, and API Apps), App Service (Linux), or Function deployments hosted in this region.
Current Status: We have determined that a backend cluster management workflow deployed a configuration change causing backend access to be blocked between a subset of Azure Storage clusters and compute resources in the Central US region. This resulted in the compute resources automatically restarting when connectivity was lost to virtual disks. We have confirmed mitigation for Azure Storage and Compute resources. We are continuing to investigate and mitigate a number of downstream impacted services that have not returned to a fully healthy state.
Status Timeline update:
- 21:56 UTC on 18 Jul 2024 – Customer impact began
- 22:13 UTC on 18 Jul 2024 – Storage team started investigating
- 22:41 UTC on 18 Jul 2024 – Additional Teams engaged to assist investigations
- 23:27 UTC on 18 Jul 2024 – All deployments in Central US stopped
- 23:35 UTC on 18 Jul 2024 – All deployments paused for all regions
- 00:45 UTC on 18 Jul 2024 – A configuration change as the underlying cause was confirmed
- 01:10 UTC on 19 Jul 2024 – Mitigation started
- 01:30 UTC on 19 Jul 2024 – Customers started seeing signs of recovery
- 02:51 UTC on 19 Jul 2024 – 99% of all impacted compute resources recovered
- 03:23 UTC on 19 Jul 2024 – all Azure Storage clusters confirmed recovery
- 03:41 UTC on 19 Jul 2024 – Mitigation confirmed for compute resources
- 06:30 UTC on 19 Jul 2024 - Investigation and mitigate process continuing for a number of downstream impacted services
Comments
0 comments
Please sign in to leave a comment.