Service disruption - US/HIPAA
Incident Report for Interact
Postmortem

At approximately 10:40am EDT / 7:40am PDT an update was applied to an internal monitoring system used to record and monitor usage of the interact application. An issue with this deployment process inadvertently resulted in an authentication issue when recording this information. This had the unexpected side effect of causing pages within Interact to automatically refresh every page. This caused a significant increase in load and resultant degradation of performance

As part of standard procedures engineers were notified. To recover the service as soon as possible engineers restored to a previous version of the monitoring system. The Interact service became fully available at 11:10am EDT / 8:10am PDT.

We are reviewing processes for the deployment of this system along with making modifications to the application code in order to ensure that future updates for this system will not have the same impact.

Posted Jul 27, 2018 - 06:58 UTC

Resolved
This issue has been marked has resolved. We will update with a postmortem soon.
Posted Jul 26, 2018 - 15:39 UTC
Update
We are now monitoring the infrastructure to ensure all affected sites are functional and working as expected.
Posted Jul 26, 2018 - 15:06 UTC
Monitoring
We are currently experiencing higher then normal latency on our systems. We have identified the issue and have reverted a change to an internal system
Posted Jul 26, 2018 - 14:50 UTC
This incident affected: North America / HIPAA Public Cloud.