File Handling Service - 29 June 2021
Incident Report for Interact
Postmortem

On 29 June 2021, around 7:14 am ET we observed a degradation of file handling service. Our file handling mechanism is utilised to store file-based content across Interact such as videos, specific attachments and other data files (e.g. themes). In essence the file handling service manages the file uploading, downloading, authentication and virus scanning of the files which are stored within the intranet.

We have identified that the file handling service degradation resulted from a failure of a single storage node across our data storage platform resulting in the intermittent unavailability of some files for a subset of customers. Upon replacing of the node with new hardware the service recovered to within normal operating parameters. During the service recovery, customers may have observed file request failures.

Interact is currently transitioning customers to a new file handling service, which offers increase durability and availability. Interact’s data storage nodes currently offer an annual failure rate (AFR) of 0.1% to 0.2%, whilst the new file handling service operates at 99.999999999 percent durability (Eleven 9’s). We expect to have migrated all customers to the new service throughout Q32021.

Please note the Postmortem has been updated 01 July 2021 06:44am.
Posted Jul 01, 2021 - 06:45 UTC

Resolved
This incident has been resolved.
Posted Jun 29, 2021 - 18:00 UTC
This incident affected: North America / HIPAA Public Cloud.