On 29 June 2021, around 7:14 am ET we observed a degradation of file handling service. Our file handling mechanism is utilised to store file-based content across Interact such as videos, specific attachments and other data files (e.g. themes). In essence the file handling service manages the file uploading, downloading, authentication and virus scanning of the files which are stored within the intranet.
We have identified that the file handling service degradation resulted from a failure of a single storage node across our data storage platform resulting in the intermittent unavailability of some files for a subset of customers. Upon replacing of the node with new hardware the service recovered to within normal operating parameters. During the service recovery, customers may have observed file request failures.
Interact is currently transitioning customers to a new file handling service, which offers increase durability and availability. Interact’s data storage nodes currently offer an annual failure rate (AFR) of 0.1% to 0.2%, whilst the new file handling service operates at 99.999999999 percent durability (Eleven 9’s). We expect to have migrated all customers to the new service throughout Q32021.