The live master node of the db04 cluster failed this morning at about 06:20 BST. The backup node was switched live within about a minute and the cluster was again accessible.
The problem occurred due to memory exhaustion on the live master node. We had identified the situation yesterday afternoon and were planning a maintenance period to increase the resources on the cluster but some usage pattern tipped the balance early this morning before we could act. The new master node has more headroom so we’re not expecting this to reoccur immediately, but we will be arranging some emergency maintenance to upgrade this cluster, which we’ll announce via this status blog as usual.
