US Cloud - old content appearing
Incident Report for Interact
Postmortem

Summary

On 3rd April 2019 at around 10:00am (UTC) following the restoration of service from the previous incident, Interact engineers began receiving reports from customers within the US pod that content, including widgets and assets were out of date.

Investigation and Root Cause

During the incident earlier that morning a rotation of the shared asset store took place. Unfortunately, after the rotation of the system the assets available were from approx. 20 days earlier. As customers reported issues, asset data was recovered from a backup taken at 07:00am (UTC) that morning. No data was lost, but the search index was also out of date therefore also rebuilt for affected customers.

Interact believes that data within the asset store should not be lost when rotation of a server occurs and has completed this step many times without any issue. We are working with AWS to understand why this would occur on this occasion.

Resolution and Mitigation Steps

In order to mitigate the re-occurrence of this issue, whenever a rotation of the asset store is required, we will ensure this is close to a backup event and ensure that we’re ready to restore the entire asset store should this be required. We continue to work with AWS to understand how this can happen and identify any other preventative steps we need to ensure are in place.

Posted 15 days ago. Apr 09, 2019 - 18:30 UTC

Resolved
This incident has now been resolved and any affected customers should now be showing relevant content.
A post mortem will be published following investigation and remediation
Posted 21 days ago. Apr 03, 2019 - 16:51 UTC
Investigating
Interact are aware of an issue with content indexes being out of date in the US public cloud. This is related to the asset store earlier today and we are working on resolving
Posted 21 days ago. Apr 03, 2019 - 13:47 UTC
This incident affected: United States Public Cloud.