Major outage on runtime

• We’re rolling out new docker runners. As instances roll one by one, there should be no service interruption. (08:00 — Sep 27)

• First docker runner checked up and running. (09:09 — Sep 27)

• Second runner up and running. (09:49 — Sep 27)

• Shutting down third runner node. Containers does not balance to the second node as it should be. (09:50 — Sep 27)

• Restarting third runner node. Runtime is back normal. (09:55 — Sep 27)

• Rolling again second docker runner. Docker runner cluster state is now broken. As an emergency measure, starting to deploy the cluster from scratch to recover as fast as possible. (10:15 — Sep 27)

• Cluster is back up and running. (11:08 — Sep 27)

Team is investigating on why the second node did not correctly join the cluster and why the cluster state broke at some point.