Error 500s occurring
Incident Report for Interact
Postmortem

As part of routine maintenance on core infrastructure, an Engineer has made a manual error and applied an erroneous configuration change which impacted the Connection String store making sites inaccessible from around 5:15pm GMT by causing connection issues. The mistake was immediately spotted, recognised and senior team got involved to help resolve the issues. The fix involved restoring to last know working configuration and busting the cache on web servers to force propagation of changes. At 5:40pm all sites were restored to full working order. This was followed by full review of configuration and actions taken alongside standard customer site health validation process. The issues was closed at 5:50pm when all processes and services were confirmed to be in full working order. Interact is looking into the steps and circumstances which led to this issue, and appropriate changes will be implemented to reduce the risk of manual error by increasing automation, reviewing change controls, and reviewing/improving change screening process for any manual changes.

Posted Nov 03, 2020 - 18:16 UTC

Resolved
The incident has been resolved.
Posted Nov 03, 2020 - 18:15 UTC
Update
A fix has been implemented and we are monitoring the system.
Posted Nov 03, 2020 - 17:46 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Nov 03, 2020 - 17:39 UTC
Identified
The issues has been identified and a fix is being implemented.
Posted Nov 03, 2020 - 17:28 UTC
Investigating
We are currently investigating an issue with Error 500s on client Intranets. We will update once we have more information.
Posted Nov 03, 2020 - 17:27 UTC
This incident affected: Europe Public Cloud 1 and Europe Public Cloud 2.