We’re rolling out new docker runners. As instances roll one by one, there should be no service interruption. (08:00 UTC — Sep 27)
First docker runner checked up and running. (09:09 UTC — Sep 27)
Second runner up and running. (09:49 UTC — Sep 27)
Shutting down third runner node. Containers does not balance to the second node as it should be. (09:50 UTC — Sep 27)
Restarting third runner node. Runtime is back normal. (09:55 UTC — Sep 27)
Rolling again second docker runner. Docker runner cluster state is now broken. As an emergency measure, starting to deploy the cluster from scratch to recover as fast as possible. (10:15 UTC — Sep 27)
Cluster is back up and running. (11:08 UTC — Sep 27)
Team is investigating on why the second node did not correctly join the cluster and why the cluster state broke at some point.
Last updated: October 17, 2024 at 5:27 PM UTC