This incident is now resolved, our Elasticsearch Cluster has now been migrated into a managed AWS service and a full reindex has taken place.
A full incident report will be made available in due course.
Posted Jul 28, 2022 - 16:25 BST
Update
We have now resolved an issue relating to media uploaded via EIS/Tark not appearing in the Wordpress Media library. This was an unforeseen issue related to the ongoing migration from our self managed elastic search cluster to a managed AWS solution.
If you have attempted to upload an image, and it's still not available in Wordpress - please attempt to re upload it.
Posted Jul 28, 2022 - 15:40 BST
Identified
At 15:17 our elasticsearch cluster began to suffer from some service degradation, resulting in slow search results and some reduces performance when loading pages not already present within the redis cache.
At 16:00 this escalated and the elasticsearch cluster experienced a complete outage. This caused all search queries to run against the database, this caused slow downs accross the site and a steady increase in 5XX errors - at 16:10, search was disabled to prevent further disruption.
The self managed Elasticsearch cluster we currently have held in Linode is pending migration to a managed service within AWS. In order to prevent further disruption we have elected to carry out this migration beginning at 09:00 on the 28th July 2022 with search functionality remaining disabled until this work is complete.
Posted Jul 27, 2022 - 17:19 BST
Investigating
Our Elasticsearch cluster is currently down. We are investigating the cause of this now. While we're investigating, search will not be available across our sites.
Posted Jul 27, 2022 - 16:19 BST
This incident affected: WordPress Infrastructure (Elasticsearch Cluster) and Websites.