Amazon S3 Outage: Our Response

Mar 13, 2017

Back to Veoci BlogAmazon S3 Outage: Our Response

What happened? 

On February 28, 2017 at 12:37 PM EST, Veoci users started experiencing problems accessing files stored in Veoci as well as uploading new files into Veoci.  The issues were triaged and eventually resolved at approximately 2:30 PM EST. 

How did this happen? 

One of Amazon’s system administrators accidentally entered an incorrect command which resulted in the complete shutdown of many Amazon servers, thus disabling Amazon S3 services in the US East Region (Virginia) and affecting thousands of customers. 

Amazon’s summary can be found here: https://aws.amazon.com/message/41926/

 Why this impacted Veoci? 

Amazon Simple Storage Service, Amazon S3, is an internet storage system to store and retrieve any amount of data from anywhere on the web.  It is split up among 8 storage locations around the world. Veoci uses Amazon S3 storage in the US East Region in Northern Virginia and the US West Region in Oregon. All Veoci files are stored across several servers in Virginia and are automatically replicated to Oregon.  

 What Veoci functionality was impacted? 

Most Veoci use was unaffected; however, uploading files, downloading files, importing via Excel, exporting to Excel, emailing attachments to Rooms, and submitting entries into public Forms with attachment fields were inhibited. How did the Veoci team respond? Veoci’s technical development team agreed to wait a short time after the outage to see if Amazon would recover quickly. After about 90 minutes without a resolution from Amazon, the Veoci team decided at 2:00 PM EST to modify our configuration so Veoci’s files would be directly accessed and stored using our Oregon location. This failover process was fully in place at 2:30 PM EST. At that time, the only functionality that was still impacted was accessing some previous file versions. Amazon S3 recovered from the outage at 4:54 PM EST. To ensure that Amazon had fully recovered, Veoci’s failover to Oregon continued for approximately two days, continuing to support almost all normal operations. On March 1, 2017 at 11:50 PM EST, our team reconfigured file access and storage back to the Virginia site, restoring all normal operations. 

How did the Veoci team communicate this issue? 

Veoci’s primary tool for delivering outage notifications, Mailchimp, was also impacted by the Amazon S3 outage. After exploring other communication options, we emailed all Organization Administrators at 2:03PM EST.  An update was also sent to the same audience at 2:52 PM EST. 

Next Steps 

To further improve our recovery method, we are currently working to configure all Veoci files to be automatically stored and accessed via both the East and West Amazon S3 Buckets. This way, should the Virginia location encounter another issue, files will automatically be sent to Oregon with no failover process needed. Veoci is also implementing a capability for our team to broadcast an Incident Alert to all users directly within Veoci, removing our reliance on Mailchimp for distributing timely communications.

Subscribe to the Veoci Blog

Receive all the latest emergency, crisis, and continuity management news, tips, and advice

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Related Posts

Vue, Vuetify, & Veoci: Evolving through Innovation

Veoci’s Geoff Baum recently penned an article about Veoci’s move to Vue and Vuetify on Medium. Learn more about this innovation and Veoci’s future.

Continue reading
Connecticut Does it Right – Investing in Jobs the Right Way

The latest report from Pitchbook, “2014 Annual US VC Valuations & Trends Report” was just published with great news for Connecticut: Connecticut Innovations, the government-sponsored organization for venture investment, broke into the TOP 10 ranks of investors for both the Seed/Angel and Early Stage Investments.

Continue reading
Cloud Computing - It's Smaller than You Think

When asked "Is it always better to own your own data center?" ZDNet and TechRepublic's CIO jury panel of technology decision-makers responded with a resounding "NO" by a margin of 11 to 1.

Continue reading

Connect with us on Social Media

Join us on our journey to improve emergency, operations, and continuity management!

Veoci Facebook PageVeoci Twitter AccountVeoci Linkedin Company Page

Face crisis and continuity challenges with expert solutions designed for you and your teams.

Learn how Veoci puts you in control