Global outage incident that affected up to 8.1 million customers appears over.
Amazon says all cloud services are “returned to normal operations” with the global outage incident that affected up to 8.1 million customers over.
AWS has issued this update on the incident which confirms the issues experienced have been identified – with services returning to normal.
The update reads: “Between 11:49 PM PDT on October 19 and 2:24 AM PDT on October 20, we experienced increased error rates and latencies for AWS Services in the US-EAST-1 Region. Additionally, services or features that rely on US-EAST-1 endpoints such as IAM and DynamoDB Global Tables also experienced issues during this time.
“At 12:26 AM on October 20, we identified the trigger of the event as DNS resolution issues for the regional DynamoDB service endpoints. After resolving the DynamoDB DNS issue at 2:24 AM, services began recovering but we had a subsequent impairment in the internal subsystem of EC2 that is responsible for launching EC2 instances due to its dependency on DynamoDB.
“As we continued to work through EC2 instance launch impairments, Network Load Balancer health checks also became impaired, resulting in network connectivity issues in multiple services such as Lambda, DynamoDB, and CloudWatch. We recovered the Network Load Balancer health checks at 9:38 AM. As part of the recovery effort, we temporarily throttled some operations such as EC2 instance launches, processing of SQS queues via Lambda Event Source Mappings and asynchronous Lambda invocations.
“Over time we reduced throttling of operations and worked in parallel to resolve network connectivity issues until the services fully recovered.
“By 3:01 PM, all AWS services returned to normal operations. Some services such as AWS Config, Redshift, and Connect continue to have a backlog of messages that they will finish processing over the next few hours.”
Downdetector says it has received more than 8.1 million reports from users who couldn’t access a web service.
In the first two hours of the outage alone, Downdetector confirmed:
- Over 1 million reports from the United States
- More than 400,000 from the United Kingdom
- Over 200,000 each from Australia, the Netherlands and Germany
- Around 180,000 from Japan
“These figures highlight the global scale of the disruption,” Downdetector says.
At this stage, the outage does not appear to be caused by a cyber-attack, with reports suggesting root cause being significant error rates for requests made to the AWS data storage service DynamoDB in the US-EAST region.
