Latency and high errors impacting a subset of EU Customers
Incident Report for Interact
Postmortem

On 24 August 2020 around 8:00:00 UTC, our Cloud team started investigating a potential issue with our EU Interact services that handles Web functions for rendering Interact content (such as homepages, etc.).  Issues were reported by customers starting around 8:10:00 UTC including blank homepages and styling issues on pages.  After investigating a number of potential causes of the issue, the core issue appeared to be caused by an inability for the web servers to establish a consistent connection with a single database server – affecting a subset of the EU customers.  The DB server was restarted and the Web Servers were rotated causing fresh hardware to be used and connections to be reset.  After restarting the web servers around 9:27:00 UTC, web functions returned to a normal state.  A spot-check of client environments was performed.  We verified that everything was running successfully and marked the incident as full resolved at 10:58:00 UTC. Additional changes were made to monitoring the connectivity with SQL Servers.

Posted Aug 24, 2020 - 14:57 UTC

Resolved
This incident has been resolved.
Posted Aug 24, 2020 - 10:58 UTC
Monitoring
Engineers have increased capacity and are monitoring the service.
Posted Aug 24, 2020 - 09:27 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Aug 24, 2020 - 08:20 UTC
Investigating
Interact Engineers are currently investigating issues within the EU of higher than normal latency which is causing slowness and intermittent service disruption across all EU customers.
Posted Aug 24, 2020 - 08:14 UTC
This incident affected: Europe Public Cloud 1 and Europe Public Cloud 2.