Amazon packages pile up as outage spawns delivery havoc

The outage prevented drivers from getting packages or routes, and shut down communications between them and the company. PHOTO: REUTERS

SEATTLE (BLOOMBERG) - An Amazon Web Services (AWS) outage wreaked havoc on the e-commerce giant's delivery operation, preventing drivers from getting routes or packages and shutting down communication between Amazon and the thousands of drivers it relies on, according to four people familiar with the situation.

Three delivery service partners said an Amazon.com app used to communicate with delivery drivers was down as part of Tuesday's (Dec 7) system failure. Vans that were supposed to be on the road delivering packages sat idle with no communication from the company, the person said.

Amazon Flex drivers, independent delivery people who carry parcels in their own cars, could not log into Amazon's app to get assignments, said another person.

Amazon said late on Tuesday afternoon that much of its Web service was working again. "We have mitigated the underlying issue that caused some network devices in the US-EAST-1 Region to be impaired," the company said on its AWS dashboard just before 6pm Eastern time. "We are seeing improvement in availability across most AWS services."

The company did not comment on the issues with its delivery operation.

The problems came during Amazon's critical holiday shopping season when the company can ill afford delays that could potentially create lasting logjams. One delivery business owner on the West Coast said the company halted deliveries on Tuesday and planned to regroup on Wednesday. Two delivery partners in earlier time zones said drivers that already had routes were instructed to put their phones in airplane mode and not log out of the Amazon routing app so they could continue making stops, but drivers who had not already been assigned routes were sidelined.

The outage began at about 10am Eastern time, according to Downdetector. At the height of the outage, the Web monitoring site reported more than 20,000 complaints for Amazon and more than 11,000 for the company's cloud computing arm, AWS.

By 1.45pm, the reported outages had declined by about half for AWS and two-thirds for Amazon. Multiple popular websites were also affected, including those operated by Coinbase Global, Robinhood Markets, Walt Disney and Netflix, according to Downdetector.

Disney said that though people were able to get into its parks, they were having difficulty checking in online and paying for purchases. Webcast presentations from Comcast and Altice USA at UBS's Global TMT Conference experienced disruptions earlier on Tuesday, and the Charter Communications presentation was rescheduled for Wednesday.

Some Amazon services, including music and video streaming, the voice-activated Alexa platform and security arm, Ring, were affected, too. Amazon said it had identified the root cause of the issue and was working to fix it. Meanwhile, the company directed customers to alternative servers in its western region that were not experiencing problems. The increased errors were in the eastern North American region.

Amazon, in an update on its AWS dashboard, said it still had not completely fixed the issue by 6pm, although "many services have already recovered". Multiple Amazon cloud-computing services had been affected, including Amazon DynamoDB and Amazon Elastic Compute.

Video streaming service Netflix experienced a 26 per cent drop in traffic after the AWS problems were reported, showing how quickly outages can ripple outward, said Mr Doug Madory, an analyst at the network monitoring firm Kentik in San Francisco.

"It gets more and more complicated with software running these services, so when something goes sideways it can take a long time to figure out what went wrong and fix it," he said. "Complexity has risks. You introduce unknown errors."

AWS is the leading cloud-computing provider, selling companies computing power and software services on demand rather than maintaining their own data centres and teams in-house. Its customers include a wide range of industries and the federal government.

Join ST's Telegram channel here and get the latest breaking news delivered to you.