Rapid Implementation COVID-19 Solutions Available

Amazon S3 Outage: Our Response

Mar 13, 2017

Back to Veoci BlogAmazon S3 Outage: Our Response

What happened? 

On February 28, 2017 at 12:37 PM EST, Veoci users started experiencing problems accessing files stored in Veoci as well as uploading new files into Veoci.  The issues were triaged and eventually resolved at approximately 2:30 PM EST. 

How did this happen? 

One of Amazon’s system administrators accidentally entered an incorrect command which resulted in the complete shutdown of many Amazon servers, thus disabling Amazon S3 services in the US East Region (Virginia) and affecting thousands of customers. 

Amazon’s summary can be found here: https://aws.amazon.com/message/41926/

 Why this impacted Veoci? 

Amazon Simple Storage Service, Amazon S3, is an internet storage system to store and retrieve any amount of data from anywhere on the web.  It is split up among 8 storage locations around the world. Veoci uses Amazon S3 storage in the US East Region in Northern Virginia and the US West Region in Oregon. All Veoci files are stored across several servers in Virginia and are automatically replicated to Oregon.  

 What Veoci functionality was impacted? 

Most Veoci use was unaffected; however, uploading files, downloading files, importing via Excel, exporting to Excel, emailing attachments to Rooms, and submitting entries into public Forms with attachment fields were inhibited. How did the Veoci team respond? Veoci’s technical development team agreed to wait a short time after the outage to see if Amazon would recover quickly. After about 90 minutes without a resolution from Amazon, the Veoci team decided at 2:00 PM EST to modify our configuration so Veoci’s files would be directly accessed and stored using our Oregon location. This failover process was fully in place at 2:30 PM EST. At that time, the only functionality that was still impacted was accessing some previous file versions. Amazon S3 recovered from the outage at 4:54 PM EST. To ensure that Amazon had fully recovered, Veoci’s failover to Oregon continued for approximately two days, continuing to support almost all normal operations. On March 1, 2017 at 11:50 PM EST, our team reconfigured file access and storage back to the Virginia site, restoring all normal operations. 

How did the Veoci team communicate this issue? 

Veoci’s primary tool for delivering outage notifications, Mailchimp, was also impacted by the Amazon S3 outage. After exploring other communication options, we emailed all Organization Administrators at 2:03PM EST.  An update was also sent to the same audience at 2:52 PM EST. 

Next Steps 

To further improve our recovery method, we are currently working to configure all Veoci files to be automatically stored and accessed via both the East and West Amazon S3 Buckets. This way, should the Virginia location encounter another issue, files will automatically be sent to Oregon with no failover process needed. Veoci is also implementing a capability for our team to broadcast an Incident Alert to all users directly within Veoci, removing our reliance on Mailchimp for distributing timely communications.

Subscribe to the Veoci Blog

Receive all the latest emergency, crisis, and continuity management news, tips, and advice

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Related Posts

Practitioner's Profile: Lifting Off with Brian Barnes

Brian has lived a life full of adventure and discipline, which has instilled in him a strong work ethic and respect for the field of emergency management, and which has ultimately led him to become the Aviation Client Success Manager at Veoci.

Continue reading
Practitioner's Profile: Anthony DeGeorge - First Responder and Healthcare Emergency Manager

At Veoci, we already have an impressive number of skilled practitioners involved in the development and implementation of our software, so we are thrilled to welcome Anthony DeGeorge to the team. Nearly 20 years of experience as a first responder and emergency manager (who used Veoci in the field) has prepared him to create and implement solutions that will improve operations for our healthcare clients.

Continue reading
Thanksgiving Safety Tips for the Holidays

Veoci is all about preparedness, even during the fun and joy of the winter holidays. So, as an extension of our preparedness, take some of these Thanksgiving safety tips back to your kitchen as you get ready to celebrate (and don’t be afraid to use them after Thanksgiving, too).

Continue reading

Connect with us on Social Media

Join us on our journey to improve emergency, operations, and continuity management!

Veoci Facebook PageVeoci Twitter AccountVeoci Linkedin Company Page

Face crisis and continuity challenges with expert solutions designed for you and your teams.

Schedule a Demo Now