8:11 PM PDT We are investigating degraded performance for some EBS volumes in a single Availability Zone (USE1-AZ2) in the US-EAST-1 Region. Some new EC2 instance launches, within the affected Availability Zone, are also impacted by this issue. We are working to resolve the issue.
8:41 PM PDT We can confirm degraded performance for some EBS volumes within a single Availability Zone (USE1-AZ2) in the US-EAST-1 Region. Existing EC2 instances within the affected Availability Zone that use EBS volumes may also experience impairment due to stuck IO to the attached EBS volume(s). Newly launched EC2 instances within the affected Availability Zone may fail to launch due to the degraded volume performance. We continue to work toward determining root cause and mitigating impact but recommend that you fail out of the affected Availability Zone (USE1-AZ2) if you are able to do so. Other Availability Zones within the US-EAST-1 Region are not affected by this issue.
9:17 PM PDT We are making progress in determining the root cause and have isolated it to a subsystem within the EBS service. We are working through multiple steps to mitigate the issue and will continue to provide updates as we make progress. Other Availability Zones remain unaffected by this issue and affected EBS volumes and EC2 instances within the affected Availability Zone have plateaued at this stage. We continue to recommend that you fail out of the affected Availability Zone (USE1-AZ2) if you are able to do so.
9:47 PM PDT We continue to make progress in determining the root cause of the issue causing degraded performance for some EBS volumes in a single Availability Zone (USE1-AZ2) in the US-EAST-1 Region. A subsystem within the larger EBS service that is responsible for coordinating storage hosts is currently degraded due to increased resource contention. We continue to work to understand the root cause of the elevated resource contention, but are actively working to mitigate the issue. Once mitigated, we expect performance for the affected EBS volumes to return to normal levels. We will continue to provide you with updates on our progress. For immediate recovery, we continue to recommend that you fail out of the affected Availability Zone (USE1-AZ2) if you are able to do so.
10:23 PM PDT We continue to make progress in determining the root cause of the issue causing degraded performance for some EBS volumes in a single Availability Zone (USE1-AZ2) in the US-EAST-1 Region. We have made several changes to address the increased resource contention within the subsystem responsible for coordinating storage hosts with the EBS service. While these changes have led to some improvement, we have not yet seen full recovery for the affected EBS volumes. We continue to expect full recovery of the affected EBS volumes once the subsystem issue has been addressed. For immediate recovery, we continue to recommend that you fail out of the affected Availability Zone (USE1-AZ2) if you are able to do so.
11:19 PM PDT We continue to work to address the increased resource contention within the EBS service subsystem responsible for coordinating EBS storage hosts. We have applied several mitigations, and while we have seen some improvements, we have not yet seen performance for affected volumes return to normal levels. We are currently rolling out another mitigation that we believe addresses the root cause and has shown promising signs in testing. We will know within the next 30-45 minutes as to whether this restores normal operations for current affected volumes. For immediate recovery, we continue to recommend that you fail out of the affected Availability Zone (USE1-AZ2) if you are able to do so.
11:43 PM PDT We can confirm that the deployed mitigation has worked and we have started to see recovery for some affected EBS volumes within the affected Availability Zone (USE1-AZ2). We are still finishing the deployment of the mitigation, but expect performance of affected EBS volumes in this single Availability Zone to return to normal levels over the next 60 minutes.
Sep 27, 12:30 AM PDT We can confirm that the performance has returned to normal levels for the majority of the affected EBS volumes within the affected Availability Zone (USE1-AZ2). We continue to work towards full recovery for the remaining EBS volumes.
