Resolved -
We’re pleased to confirm that the recent service disruption has been fully resolved. Our teams, in collaboration with our cloud hosting partner, have implemented corrective measures and performance has returned to normal across all regions, including the US Pod.
We appreciate your patience and understanding while we worked to restore full functionality. If you experience any further issues, please reach out to our support team.
Oct 20, 20:58 UTC
Monitoring -
We're seeing large improvements in site performance and stability, with many sites fully back to normal. The team continues to monitor and verify the various components.
Oct 20, 19:49 UTC
Update -
We have started to see AWS allow provisioning of new instances, and our environment is scaling back up to its natural size. We have increased the desired size to ensure that once scaled up, the environment has enough capacity to support the system without further scaling, in case of other unexpected issues with provisioning new hardware. We expect the performance to improve soon and the environments to stabilise. We continue to monitor and will update soon.
Oct 20, 19:12 UTC
Update -
We are continuing to work closely with our cloud hosting partner to resolve the ongoing issue. While challenges persist on the US Pod, we have observed an improvement in performance over the past 60 minutes. Our teams remain fully engaged and committed to restoring full service as quickly as possible.
Oct 20, 17:53 UTC
Identified -
We continue to work with our cloud provider on the issues experienced across US EAST 1 region. Currently the issues stem from AWS's inability to process new requests for scaling up new infrastructure, preventing more capacity being added to the server pools. To alleviate some of the pressure, our team is temporarily rewiring the already running virtual machines to help with the extra load, to get around the issue with scaling new hardware. Our team continues to work on the issue, and will provide updates as progress is made.
Oct 20, 14:44 UTC
Investigating -
We’re currently experiencing increased latency and intermittent 5xx errors when accessing the platform on the US pod. Most customers are not impacted. Our team has identified the root cause and is actively working to migrate these cells.
We’ll continue to monitor the situation and provide updates as progress is made. Thank you for your patience while we work to restore normal performance.
Oct 20, 13:36 UTC