Summary

On 3rd April 2019 at around 10:00am (UTC) following the restoration of service from the previous incident, Interact engineers began receiving reports from customers within the US pod that content, including widgets and assets were out of date.

Investigation and Root Cause

During the incident earlier that morning a rotation of the shared asset store took place. Unfortunately, after the rotation of the system the assets available were from approx. 20 days earlier. As customers reported issues, asset data was recovered from a backup taken at 07:00am (UTC) that morning. No data was lost, but the search index was also out of date therefore also rebuilt for affected customers.

Interact believes that data within the asset store should not be lost when rotation of a server occurs and has completed this step many times without any issue. We are working with AWS to understand why this would occur on this occasion.

Resolution and Mitigation Steps

In order to mitigate the re-occurrence of this issue, whenever a rotation of the asset store is required, we will ensure this is close to a backup event and ensure that we’re ready to restore the entire asset store should this be required. We continue to work with AWS to understand how this can happen and identify any other preventative steps we need to ensure are in place.