Summary

On 3rd April 2019 at 6:40am (UTC) Interact engineers began rolling out a small update to the application across the US pod. Following this, between 6:40am (UTC) and 08:10am (UTC) customers hosted in the US Pod experienced a number of failed requests for static assets resulting in broken themes (due to missing CSS, JS and images).

Investigation and Root Cause

During the release process it was identified that a number of servers failed to come into rotation, and those servers that successfully came into rotation had failed to map to the underlying asset store. Assets including CSS, JS and images are stored on a shared file storage system that each web server maps too as part of the initialisation process.

Interact believes that an underlying hardware issue with the asset store itself, caused the failure, and consequently rotated the asset store system too. Once this was completed, all application servers were rotated again and the asset store was correctly mapped, effectively restoring the service.

Resolution and Mitigation Steps

Subsequent to the issue, Interact has updated its deployment run book, adding a check to ensure that connectivity to the asset store is active as expected.