Amazon.com Inc. said it resolved network device issues that led to a web services outage, disrupting a host of online services—from Disney and Netflix to Coinbase and Robinhood—and wreaking havoc on a massive package-delivery operation.
“With the network device issues resolved, we are now working towards recovery of any impaired services,” Amazon Web Services, the company’s cloud computing arm, said on its online dashboard Tuesday evening at 7:35 p.m. Eastern time.
Earlier on Tuesday, AWS said it identified the cause of “increased error rates” and was working to fix it. Meanwhile, the company directed customers to alternative servers in its western region that weren’t experiencing problems. The increased errors were in the eastern North American region. Multiple Amazon cloud-computing services were affected, including Amazon DynamoDB and Amazon Elastic Compute.
Amazon is No. 1 in the 2021 Digital Commerce 360 Top 1000.
The outage began about 10 a.m. Eastern time, according to Downdetector. At the height of the failure, the web monitoring site reported more than 20,000 complaints for Amazon and more than 11,000 for AWS. By 1:45 p.m., the reported outages had declined by about half for AWS and two-thirds for Amazon.
Multiple popular websites were also affected, including those operated by McDonald’s, Venmo and T. Rowe Price, according to Downdetector. Walt Disney Co. said people were able to get into the company’s parks but had difficulty checking in online and paying for purchases. Webcast presentations from Comcast Corp. and Altice USA at UBS’s Global TMT Conference experienced disruptions Tuesday and Charter Communications Inc. rescheduled a presentation for Wednesday.
Some Amazon services, including music and video streaming, the voice-activated Alexa platform and its security arm, Ring, were affected, too.
Nothing much Amazon sellers could do
Joe Stefani, president of Desert Cactus, an Amazon store that sells licensed products emblazoned with the logos of sports teams, colleges, and other organizations, says the Dec. 7 outage was the second disruption the company faced during the fourth quarter of 2021.
“There’s really not much we can do,” when AWS goes down, Stefani says.
Stefani says the company is “focused on Amazon,” and products shipped from Amazon warehouses represent 90% of its business. He does not believe Desert Cactus lost a lot of business on Tuesday, as customers could place orders and the retailer processed those the next day. However, the outage generated a one-day delay in shipping. He says that’s important because his customers often make time-sensitive orders centered on graduations and other important milestones.
The biggest potential problem, Stefani says, was that the outage delayed some products from arriving and being checked in at Amazon warehouses because logistical software was inoperable. The resulting confusion could have led some customers to find that items they wanted were unnecessarily out of stock, he says.
Stefani says Desert Cactus remains dedicated to Amazon despite the outage because no other marketplace comes close to offering the same level of web traffic. He says the retailer has recently tested selling a limited group of products on Walmart.com, with encouraging results.
Despite what he says is a frustrating user interface, Desert Cactus might list more products on Walmart.com in 2022, Stefani says. But he does not think the Walmart marketplace could rival Amazon.com any time soon.
Chaos ensued for AWS customers
The outage wreaked havoc on the ecommerce giant’s delivery operation, preventing drivers from getting routes or packages and shutting down communication between Amazon.com Inc. and the thousands of drivers it relies on, four people familiar with the situation told Bloomberg News. Three delivery service partners said an Amazon app used to communicate with delivery drivers is down. Vans that were supposed to be on the road delivering packages are sitting idle with no communication from the company.
Amazon Flex drivers, independent delivery people who carry parcels in their own cars, could not log into Amazon’s app to get assignments, said another person. The problems came amid Amazon’s critical holiday shopping season when the ecommerce giant can ill afford delays that could potentially create lasting logjams, the person said.
Amazon, through a spokeswoman, didn’t immediately comment on the problems in its delivery operation.
Video streaming service Netflix experienced a 26% drop in traffic after the AWS problems were reported, showing how quickly outages can ripple outward, said Doug Madory, an analyst at the network monitoring firm Kentik in San Francisco. “It gets more and more complicated with software running these services, so when something goes sideways, it can take a long time to figure out what went wrong and fix it,” he said. “Complexity has risks. You introduce unknown errors.”
At the time, Amazon declined to comment on the outage and directed users to its AWS dashboard.
AWS is the leading cloud-computing provider, selling companies computing power and software services on demand rather than maintaining their own data centers and teams in-house. Its customers include a wide range of industries and the federal government.
What retailers can do during outages
Thibaud Clement, CEO and co-founder of Loomly, a social media management technology provider, says reaching out to customers via social media and email can be effective for keeping customers informed—and calm—when retailer websites go down.
A good example, Clement says, was online eyeglasses retailer Warby Parker, which informed customers on Twitter about the cause of the outage and reassured them the company was working on it.
Our website and app are unexpectedly down due to the widespread Amazon Web Services (AWS) outage, which is impacting a number of companies. We are in constant communication with their team but do not know when the issue will be resolved.
— Warby Parker Help (@WarbyParkerHelp) December 7, 2021
If email capabilities operate on a separate server than the website, Clement says retailers might also consider sending customers emails explaining the cause of an outage. After that, he says, they might want to use the same channel to inform customers when their sites become operable once again.
A third strategy, Clement says, is to create a status page—set up on a separate server—that pops up when the main website goes down. The page could inform customers about the reason for the outage and redirect them, where applicable, to other online or offline sales channels.
James Melton contributed to this report.