Increased Error Rates and Latency
Incident Report for Interact
Postmortem

SUMMARY OF IMPACT: On 13 July 2019 customers on both US and EU Cloud experienced high latency and issues with themes not correctly loading.

ROOT CAUSE AND MITIGATION: On the 13 July 2019 an update to 7.9.47 of Interact caused Customers using Internet Explorer to make abnormal search requests to the search web endpoint, this in turn caused high latency. Our Engineers quickly diagnosed the issue and reverted to the previous version of Interact (7.9.46). During this rollback period customers experienced loss of themes for a short time when both 7.9.47 and 7.9.46 were in rotation within the load balancers.

NEXT STEPS: We sincerely apologize for the impact this had to customers. We have identified and fixed the search issue with Interact when using Internet Explorer and development are currently working on a fix for the themes not correctly rendering during release/rollback of a release.

Posted 3 months ago. Jun 18, 2019 - 10:25 UTC

Resolved
We are currently experiencing issues with high latency across both EU and US Pods cause by a abnormal increase in searches.
Posted 3 months ago. Jun 13, 2019 - 10:00 UTC