<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>UBIKA Service Status</title><link>https://status.ubika.io/</link><description>Incident history</description><generator>github.com/cstate</generator><language>en</language><lastBuildDate>Tue, 28 Apr 2026 08:50:00 +0100</lastBuildDate><atom:link href="https://status.ubika.io/index.xml" rel="self" type="application/rss+xml"/><item><title>[Resolved] Disruption Detected</title><link>https://status.ubika.io/issues/2026-04-28-incident_ddos/</link><pubDate>Tue, 28 Apr 2026 08:50:00 +0100</pubDate><guid>https://status.ubika.io/issues/2026-04-28-incident_ddos/</guid><category>2026-04-28T11:10:00+01:00</category><description>&lt;p>Dear Customer, Dear Partner,&lt;/p>
&lt;p>There are currently disruptions on the Cloud Protector platform.&lt;/p>
&lt;p>All our teams are mobilized to get it back to 100% operational as soon as possible.&lt;/p>
&lt;p>This ticket will be updated as the investigation progresses.&lt;/p>
&lt;p>The Ubika Support Team&lt;/p>
&lt;hr>
&lt;p>We experienced a DDoS attack from 8:50 a.m. to 11:10 a.m. The Cloud Protector platform is now fully operational again.
Traffic was impacted, resulting in slowdowns, and some application logs were also lost.
The console was not impacted.
Actions have been taken to improve the anti-DDoS system as well as the scalability of the filtering environment.
&lt;span class="faded">(11:10 — Apr 28)&lt;/span>
&lt;/p></description></item><item><title>[Resolved] Disruption Detected</title><link>https://status.ubika.io/issues/2026-04-13-console/</link><pubDate>Mon, 13 Apr 2026 11:10:00 +0200</pubDate><guid>https://status.ubika.io/issues/2026-04-13-console/</guid><category>2026-04-13T11:30:00+02:00</category><description>&lt;p>Dear Customer, Dear Partner,&lt;/p>
&lt;p>The Cloud Protector management console is currently unavailable, and our team is investigating the issue.&lt;/p>
&lt;p>&lt;strong>No impact is to be expected on traffic.&lt;/strong>&lt;/p>
&lt;hr>
&lt;p>Console and API are back online at 11h30&lt;/p>
&lt;p>This issue is caused by a OVH network incident. You can find more detail here : &lt;a href="https://public-cloud.status-ovhcloud.com/incidents/4fqsbwrzpf5m">https://public-cloud.status-ovhcloud.com/incidents/4fqsbwrzpf5m&lt;/a>&lt;/p>
&lt;p>Impacts of the incident between 10:53 AM and 11:31 AM:&lt;/p>
&lt;ul>
&lt;li>No impact on your asset traffic&lt;/li>
&lt;li>Interrupted access to console.ubika.io and APIs&lt;/li>
&lt;li>Log ingestion was impacted. Depending on the amount of traffic, logs may have been lost during the incident period.&lt;/li>
&lt;/ul></description></item><item><title>[Resolved] Disruption Detected</title><link>https://status.ubika.io/issues/2026-03-26-incident/</link><pubDate>Thu, 26 Mar 2026 11:40:00 +0100</pubDate><guid>https://status.ubika.io/issues/2026-03-26-incident/</guid><category>2026-03-26T14:15:00+01:00</category><description>&lt;p>Dear Customer, Dear Partner,&lt;/p>
&lt;p>Our filtering cluster hosted in Gravelines is facing an OVH incident impacting all of our gateways. The service remains fully operational via our Strasbourg cluster, but since 50% of the traffic is still routed towards the impacted cluster, half of the requests get a 503 HTTP error.&lt;/p>
&lt;p>We decided to deactivate Gravelines from production. The &lt;strong>service is again fully operational&lt;/strong>.
&lt;span class="faded">(11:59 — Mar 26)&lt;/span>
&lt;/p>
&lt;hr>
&lt;p>The Gravelines cluster is back up and running. We are checking everything before reintroducing it in production.
&lt;span class="faded">(13:40 — Mar 26)&lt;/span>
&lt;/p>
&lt;hr>
&lt;p>The Gravelines cluster is back in production and now handling traffic again.
&lt;span class="faded">(14:15 — Mar 26)&lt;/span>
&lt;/p></description></item><item><title>[Resolved] Logs and dashboard disrupted</title><link>https://status.ubika.io/issues/2026-02-17-incident-logs-dashboards/</link><pubDate>Tue, 17 Feb 2026 17:47:39 +0200</pubDate><guid>https://status.ubika.io/issues/2026-02-17-incident-logs-dashboards/</guid><category>2026-02-17T18:30:00+02:00</category><description>&lt;p>Dear Customer, Dear Partner,
There are currently disruptions on logs and dashboard access.
All our teams are mobilized to get it back to 100% operational as soon as possible.
This ticket will be updated as the investigation progresses.
The Ubika Support Team&lt;/p>
&lt;hr>
&lt;p>Access to logs and dashboards has been fully restored.
During the disruption, all logs were ingested; only viewing them was affected.&lt;/p></description></item><item><title>[Resolved] Maintenance on Dec 15</title><link>https://status.ubika.io/issues/2025-12-10-maintenance-etcd/</link><pubDate>Mon, 15 Dec 2025 06:00:00 +0100</pubDate><guid>https://status.ubika.io/issues/2025-12-10-maintenance-etcd/</guid><category>2025-12-15T09:00:00+01:00</category><description>&lt;p>Dear Customer, Dear Partner,&lt;/p>
&lt;p>Maintenance is scheduled for Monday, December 15 at 6:00 a.m. (Europe/Paris time), and should last no longer than 3 hours.
No interruptions are planned.&lt;/p>
&lt;p>The Ubika Support Team&lt;/p></description></item><item><title>[Resolved] Disruption Detected</title><link>https://status.ubika.io/issues/2025-12-10-incident/</link><pubDate>Wed, 10 Dec 2025 12:00:00 +0100</pubDate><guid>https://status.ubika.io/issues/2025-12-10-incident/</guid><category>2025-12-10T20:00:00+01:00</category><description>&lt;p>Dear Customer, Dear Partner,&lt;/p>
&lt;p>Our filtering cluster hosted in Gravelines faced troubles to deploy new configuration updates. Hence, we decided to isolate it from production.&lt;/p>
&lt;p>Please note that &lt;strong>the service remains fully operational via our Strasbourg cluster&lt;/strong>, ensuring continued protection and availability. In case of configuration modification, the deployment status in the console will remain in “pending” state (orange). This is a consequence of Gravelines isolation and has no impact on the service availibility.&lt;/p>
&lt;p>A case is ongoing with our hosting provider.&lt;/p>
&lt;h2 id="the-ubika-support-team">The Ubika Support Team&lt;/h2>
&lt;p>The Gravelines cluster in back on production.
&lt;span class="faded">(20:00 — Dec 10)&lt;/span>
&lt;/p></description></item><item><title>[Resolved] Maintenance on Dec 8</title><link>https://status.ubika.io/issues/2025-12-08_console_update/</link><pubDate>Mon, 08 Dec 2025 06:00:00 +0000</pubDate><guid>https://status.ubika.io/issues/2025-12-08_console_update/</guid><category>2025-12-08 08:00:00</category><description>&lt;ul>
&lt;li>
&lt;p>&lt;strong>Description:&lt;/strong> Software Upgrade&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>When:&lt;/strong> Expected Upgrade Start Date and Time: 8 December 2025 from 6:00am CET to 12:00pm CET&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Business Impact:&lt;/strong> No impact on traffic side. This duration defines the Maintenance Window only and does not reflect a downtime duration. Interruption on management console should not last longer than 10 minutes.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Expected action(s) from customer:&lt;/strong> None.&lt;/p>
&lt;/li>
&lt;/ul>
&lt;p>We will be performing maintenance on the &lt;a href="https://console.ubika.io/">https://console.ubika.io/&lt;/a> website on Monday 8 December.&lt;/p>
&lt;p>We apologize for the inconvenience caused by this upgrade. Please do not hesitate to contact our support team if you encounter any problems following this upgrade.&lt;/p></description></item><item><title>[Resolved] Availability issue AFTER CONFIGURATION UPGRADE</title><link>https://status.ubika.io/issues/2025-11-07/</link><pubDate>Fri, 07 Nov 2025 12:00:00 +0100</pubDate><guid>https://status.ubika.io/issues/2025-11-07/</guid><category>2025-11-07T20:55:00+01:00</category><description>&lt;p>Dear Customer, Dear Partner,&lt;/p>
&lt;p>Some asset are facing traffic logs loss on our Gravelines cluster. Cause is under investigation.&lt;/p>
&lt;p>This have no impact on filtering efficiency and traffic passing through Cloud Protector.&lt;/p>
&lt;p>The Ubika Support Team&lt;/p>
&lt;hr>
&lt;p>We do not reproduce impact, but it seems that there are issues reported by customers that it has impact on traffic.&lt;/p>
&lt;p>Gravelines cluster is now out of production to avoid side effects
&lt;span class="faded">(15:50 — Nov 7)&lt;/span>
&lt;/p>
&lt;hr>
&lt;p>Our second cluster now exhibits the same behaviour. You may encounter 504 HTTP error in particular if WAF configurations are frequently reloaded.&lt;/p>
&lt;p>We enabled back the Gravelines cluster as there is no benefit of keeping it off. Investigations are ongoing with our hoster.
&lt;span class="faded">(16:40 — Nov 7)&lt;/span>
&lt;/p>
&lt;hr>
&lt;p>If possible, avoid to update configurations of assets to avoid assets issues. We are on the process of applying a workaround that mitigate the issue.&lt;/p>
&lt;p>Our hoster is investigating on the managed kubernetes infrastructures that seems to be the cause of the issue.
&lt;span class="faded">(18:10 — Nov 7)&lt;/span>
&lt;/p>
&lt;hr>
&lt;p>Root cause has been identified. A workaround has been deployed and situation should go back to normal. A proper fix will be deployed on 12/11/2025.
&lt;span class="faded">(20:55 — Nov 7)&lt;/span>
&lt;/p></description></item><item><title>[Resolved] K8s cluster outage</title><link>https://status.ubika.io/issues/2025-10-31-1/</link><pubDate>Fri, 31 Oct 2025 17:30:00 +0100</pubDate><guid>https://status.ubika.io/issues/2025-10-31-1/</guid><category>2025-10-31T20:05:00+01:00</category><description>&lt;p>Dear Customer, Dear Partner,&lt;/p>
&lt;p>Our filtering cluster hosted in Strasbourg is currently having Control Plane issue and does not deploy anymore configuration updates. We decided to remove it from our load balancing to avoid outdated configurations running.&lt;/p>
&lt;p>A case is ongoing with our hoster. Service remains operational in Gravelines cluster.&lt;/p>
&lt;p>The Ubika Support Team&lt;/p>
&lt;hr>
&lt;p>The Strasbourg cluster in back on production.
&lt;span class="faded">(20:05 — Oct 31)&lt;/span>
&lt;/p></description></item><item><title>[Resolved] K8s cluster outage</title><link>https://status.ubika.io/issues/2025-10-20-1/</link><pubDate>Mon, 20 Oct 2025 16:00:00 +0200</pubDate><guid>https://status.ubika.io/issues/2025-10-20-1/</guid><category>2025-10-20T17:40:00+02:00</category><description>&lt;p>Dear Customer, Dear Partner,&lt;/p>
&lt;p>Our filtering cluster hosted in Gravelines is currently having Control Plane issue and does not deploy anymore configuration updates. We decided to remove it from our load balancing to avoid outdated configurations running.&lt;/p>
&lt;p>A case is ongoing with our hoster. Service remains operational in Strasbourg cluster.&lt;/p>
&lt;p>The Ubika Support Team&lt;/p>
&lt;hr>
&lt;p>The Gravelines cluster in back on production.
&lt;span class="faded">(17:40 — Oct 20)&lt;/span>
&lt;/p></description></item><item><title>[Resolved] Maintenance on Sep 18</title><link>https://status.ubika.io/issues/2025-09-18-08-00/</link><pubDate>Thu, 18 Sep 2025 08:00:00 +0200</pubDate><guid>https://status.ubika.io/issues/2025-09-18-08-00/</guid><category>2025-09-18T09:15:00+02:00</category><description>&lt;ul>
&lt;li>
&lt;p>&lt;strong>Description:&lt;/strong> Software Upgrade&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>When:&lt;/strong> Expected Upgrade Start Date and Time: 18 September 2025 at 8:00am CET to 10:00am CET&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Business Impact:&lt;/strong> Users that use GeoIp or IpRep in blocking mode may experience a short perturbation. It should last less than 1 minute.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Expected action(s) from customer:&lt;/strong> None.&lt;/p>
&lt;/li>
&lt;/ul>
&lt;p>We will be performing maintenance on our filtering cluster on Thursday 18 September 2025.&lt;/p>
&lt;p>We apologize for the inconvenience caused by this upgrade. Please do not hesitate to contact our support team if you encounter any problems following this upgrade.&lt;/p>
&lt;p>&lt;strong>Update:&lt;/strong> Maintenance is over at 09:15am CET - everything went well.&lt;/p></description></item><item><title>[Resolved] Maintenance on Sep 17 - Postponed</title><link>https://status.ubika.io/issues/2025-09-17-08-00/</link><pubDate>Wed, 17 Sep 2025 08:00:00 +0200</pubDate><guid>https://status.ubika.io/issues/2025-09-17-08-00/</guid><category>2025-09-16T18:30:00+02:00</category><description>&lt;ul>
&lt;li>
&lt;p>&lt;strong>Description:&lt;/strong> Software Upgrade&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>When:&lt;/strong> Expected Upgrade Start Date and Time: 17 September 2025 at 8:00am CET to 10:00am CET&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Business Impact:&lt;/strong> Users that use GeoIp or IpRep in blocking mode may experience a short perturbation. It should last less than 1 minute.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Expected action(s) from customer:&lt;/strong> None.&lt;/p>
&lt;/li>
&lt;/ul>
&lt;p>We will be performing maintenance on our filtering cluster on Thursday 17 September 2025.&lt;/p>
&lt;p>We apologize for the inconvenience caused by this upgrade. Please do not hesitate to contact our support team if you encounter any problems following this upgrade.&lt;/p>
&lt;ul>
&lt;li>&lt;strong>UPDATE:&lt;/strong> Maintenance planned on Sep 17 is postponed&lt;/li>
&lt;/ul></description></item><item><title>[Resolved] Maintenance on Sep 09</title><link>https://status.ubika.io/issues/2025-09-09-08-00/</link><pubDate>Tue, 09 Sep 2025 08:00:00 +0100</pubDate><guid>https://status.ubika.io/issues/2025-09-09-08-00/</guid><category>2025-09-09T09:30:00+01:00</category><description>&lt;ul>
&lt;li>
&lt;p>&lt;strong>Description:&lt;/strong> Software Upgrade&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>When:&lt;/strong> Expected Upgrade Start Date and Time: 09 September 2025 at 8:00am CET to 10:00am CET&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Business Impact:&lt;/strong> No impact on traffic side. Access to the management console may be disrupted. This duration defines the Maintenance Window only and does not reflect a downtime duration. Interruption on management console should not last longer than 10 minutes.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Expected action(s) from customer:&lt;/strong> None.&lt;/p>
&lt;/li>
&lt;/ul>
&lt;p>We will be performing maintenance on the &lt;a href="https://console.ubika.io/">https://console.ubika.io/&lt;/a> website on Thursday 09 September 2025.&lt;/p>
&lt;p>We apologize for the inconvenience caused by this upgrade. Please do not hesitate to contact our support team if you encounter any problems following this upgrade.&lt;/p></description></item><item><title>[Resolved] DDoS attack</title><link>https://status.ubika.io/issues/2025-08-01/</link><pubDate>Sun, 03 Aug 2025 13:20:00 +0200</pubDate><guid>https://status.ubika.io/issues/2025-08-01/</guid><category>2025-08-04T08:20:00+02:00</category><description>&lt;p>Dear Customer, Dear Partner,&lt;/p>
&lt;p>A DDoS attack is ongoing since Friday, 01/08/2025, starting at 13:20 PM CEST.&lt;/p>
&lt;p>The service remains operational. For now, the impact is very low. The disruptions are barely visible to the most sensitive probes.
If the attack escalates, you can redirect your DNS to our new CNAME. You can find the CNAME in the asset configuration page.&lt;/p>
&lt;p>The Ubika Support Team&lt;/p>
&lt;hr>
&lt;p>Attack ended.
&lt;span class="faded">(08:20 — Aug 4)&lt;/span>
&lt;/p>
&lt;p>Attack remained under control with no visible impact.&lt;/p></description></item><item><title>[Resolved] Release causing latency</title><link>https://status.ubika.io/issues/2025-07-22-release-latence/</link><pubDate>Tue, 22 Jul 2025 10:30:00 +0100</pubDate><guid>https://status.ubika.io/issues/2025-07-22-release-latence/</guid><category>2025-07-22T11:30:00+01:00</category><description>&lt;p>Dear Customer, Dear Partner,&lt;/p>
&lt;p>Following a production release on our filtering clusters this morning, we observed a temporary increase in latency between 10:30 AM and 11:30 AM.
The situation was fully resolved by 11:30 AM, and our teams have confirmed that all services are now stable.&lt;/p>
&lt;p>This was an isolated and exceptional event.&lt;/p>
&lt;p>Thank you for your understanding.&lt;/p>
&lt;p>The Ubika Support Team&lt;/p></description></item><item><title>[Resolved] DDoS attack</title><link>https://status.ubika.io/issues/2025-07-03/</link><pubDate>Thu, 03 Jul 2025 08:20:00 +0200</pubDate><guid>https://status.ubika.io/issues/2025-07-03/</guid><category>2025-07-04T08:20:00+02:00</category><description>&lt;p>Dear Customer, Dear Partner,&lt;/p>
&lt;p>A DDoS attack is ongoing since Wednesday, 03/07/2025, starting at 8:20 AM CEST.&lt;/p>
&lt;p>The service remains operational. Clients may encounter sporadic packet loss during TLS handshake. Browser usually recover well for such network errors at the cost of a small delay. API clients that do not retry on TCP error and monitoring probes may return errors.&lt;/p>
&lt;p>We are currently investigating on a working workaround to this typology of attacks.&lt;/p>
&lt;p>The Ubika Support Team&lt;/p>
&lt;hr>
&lt;p>Attack ended.
&lt;span class="faded">(08:20 — Jul 4)&lt;/span>
&lt;/p>
&lt;p>New information have been collected and provided to OVHCloud to narrow down the root cause of the perturbations. Conclusions will be shared later after further analysis.&lt;/p></description></item><item><title>[Resolved] Network pertubations</title><link>https://status.ubika.io/issues/2025-07-02-2/</link><pubDate>Wed, 02 Jul 2025 14:23:00 +0200</pubDate><guid>https://status.ubika.io/issues/2025-07-02-2/</guid><category>2025-07-03T09:30:00+02:00</category><description>&lt;p>Dear Customer, Dear Partner,&lt;/p>
&lt;p>Network pertubations are ongoing on our cloud provider since Wednesday, 02/07/2025, starting at 14:23 CEST.&lt;/p>
&lt;p>The service remains operational. Clients may have encountered sporadic 503 http response code. We decided to update our zoning configuration of Strasbourg zone at 15:45 CEST. There is no perturbations anymore.&lt;/p>
&lt;p>We closly monitoring the incident and will update the status&lt;/p>
&lt;p>The Ubika Support Team&lt;/p>
&lt;hr>
&lt;p>End of network perturbations, there is no new impact to report. Zoning configuration of Strasbourg has been reset to original one.&lt;/p></description></item><item><title>[Resolved] DDoS attack</title><link>https://status.ubika.io/issues/2025-07-02/</link><pubDate>Wed, 02 Jul 2025 08:20:00 +0200</pubDate><guid>https://status.ubika.io/issues/2025-07-02/</guid><category>2025-07-02T11:30:00+02:00</category><description>&lt;p>Dear Customer, Dear Partner,&lt;/p>
&lt;p>A DDoS attack is ongoing since Wednesday, 02/07/2025, starting at 8:20 AM CEST.&lt;/p>
&lt;p>The service remains operational. Clients may encounter sporadic packet loss during TLS handshake. Browser usually recover well for such network errors at the cost of a small delay. API clients that do not retry on TCP error and monitoring probes may return errors.&lt;/p>
&lt;p>We are currently investigating on a working workaround to this typology of attacks.&lt;/p>
&lt;p>The Ubika Support Team&lt;/p>
&lt;hr>
&lt;p>Attack has reduced. We do not notice disruptions anymore.
&lt;span class="faded">(11:30 — Jul 2)&lt;/span>
&lt;/p>
&lt;p>Further tests have been performed and shared with our OVHCloud taskforce to exclude various root causes of the perturbations. Conclusions will be shared later after further analysis.&lt;/p></description></item><item><title>[Resolved] DDoS attack</title><link>https://status.ubika.io/issues/2025-05-22/</link><pubDate>Thu, 22 May 2025 08:20:00 +0200</pubDate><guid>https://status.ubika.io/issues/2025-05-22/</guid><category>2025-05-23T08:20:00+02:00</category><description>&lt;p>Dear Customer, Dear Partner,&lt;/p>
&lt;p>A DDoS attack is ongoing since Thursday, 22/05/2025, starting at 8:20 AM CEST.&lt;/p>
&lt;p>The service remains operationnal with minimal impact. We are currently investigating on the attack typology before applying a workaround planned in collaboration with our hoster OVHCloud to completely cut down the attack.&lt;/p>
&lt;p>The Ubika Support Team&lt;/p>
&lt;hr>
&lt;p>Analyze is ongoing. The workaround has been applied by OVHCloud. Service should improve for those that are the most impacted (API consumers).
&lt;span class="faded">(10:21 — May 22)&lt;/span>
&lt;/p>
&lt;hr>
&lt;p>Attack has ended.
&lt;span class="faded">(08:20 — May 23)&lt;/span>
&lt;/p>
&lt;p>Network dumps have been captured before and after the workaround was applied by OVHCloud. Analyze will continue in collaboration with OVHCloud to understand the root cause of the sporadic packet loss and retransmission that was observed during the attack and address it.&lt;/p></description></item><item><title>[Resolved] Disruption Detected</title><link>https://status.ubika.io/issues/2025-05-12-15-38-1370041/</link><pubDate>Mon, 12 May 2025 15:38:39 +0200</pubDate><guid>https://status.ubika.io/issues/2025-05-12-15-38-1370041/</guid><category>2025-05-12T15:41:39.497775419+02:00</category><description>&lt;p>&lt;em>Investigating&lt;/em> - We are investigating a potential issue that might affect the uptime of one our of services. We are sorry for any inconvenience this may cause you. This incident post will be updated once we have more information.
This is an automatic post by a monitor bot.&lt;/p></description></item><item><title>[Resolved] DDoS attack</title><link>https://status.ubika.io/issues/2025-03-12/</link><pubDate>Wed, 12 Mar 2025 07:15:00 +0100</pubDate><guid>https://status.ubika.io/issues/2025-03-12/</guid><category>2025-03-13T07:15:00+01:00</category><description>&lt;p>Dear Customer, Dear Partner,&lt;/p>
&lt;p>A DDoS attack is ongoing since Wednesday, 12/03/2025, starting at 7:15 AM CET, targeting one of our customer.&lt;/p>
&lt;p>IMPACT:&lt;/p>
&lt;ul>
&lt;li>The service remain operational with a minor impact on the platform&amp;rsquo;s response time for browser clients.&lt;/li>
&lt;li>However, probes and API clients may have issues contacting the service due to packet loss (without aggressive retries like web browser).&lt;/li>
&lt;/ul>
&lt;p>Update
&lt;span class="faded">(11:00 — Mar 12)&lt;/span>
&lt;/p>
&lt;p>Attack is still ongoing, but at very low rate. Impact should be nearly invisible for every workload.&lt;br>
We may have identified the source of the sporadic traffic loss, related to the attack typology. A fix will be deployed soon to address the issue.&lt;/p>
&lt;p>Update
&lt;span class="faded">(07:20 — Mar 13)&lt;/span>
··&lt;/p>
&lt;p>Attack ended at 7:15 AM CET on 13/03/2025.
Traffic capture have been done both by UBIKA and OVH, various mitigations are studied to be more efficient at reducing the impact of those attacks.&lt;/p>
&lt;p>The Ubika Support Team&lt;/p></description></item><item><title>[Resolved] Logs database maintenance</title><link>https://status.ubika.io/issues/2025-02-27-08-00-00/</link><pubDate>Thu, 27 Feb 2025 08:00:00 +0100</pubDate><guid>https://status.ubika.io/issues/2025-02-27-08-00-00/</guid><category>2025-02-27T11:35:00+01:00</category><description>&lt;ul>
&lt;li>&lt;strong>Description:&lt;/strong> Database Upgrade&lt;/li>
&lt;li>&lt;strong>When:&lt;/strong> Expected Upgrade Start Date and Time: 27 February 2025 at 8:00am CET to 14:00am CET&lt;/li>
&lt;li>&lt;strong>Business Impact:&lt;/strong> This duration defines the Maintenance Window only. No traffic interruption planned.&lt;/li>
&lt;/ul>
&lt;p>A corrective maintenance is planned to improve queries performance on business logs (traffic logs and security events).&lt;br>
&lt;strong>No interruption of the filtering service is expected.&lt;/strong>&lt;/p>
&lt;p>No loss of logs is expected during the update. However, the history of business logs will be unavailable for some hours, while they are asynchronously rewritten.&lt;/p>
&lt;hr>
&lt;p>Migration of traffic logs is finished, all history data are available again. Migration of security events is ongoing with an ETA of 30 minutes.
&lt;span class="faded">(11:05 — Feb 27)&lt;/span>
&lt;/p>
&lt;hr>
&lt;p>Migration logs history is finished.
&lt;span class="faded">(11:35 — Feb 27)&lt;/span>
&lt;/p></description></item><item><title>[Resolved] Maintenance on Feb 06</title><link>https://status.ubika.io/issues/2025-02-06-09-00-1738828/</link><pubDate>Thu, 06 Feb 2025 09:00:00 +0100</pubDate><guid>https://status.ubika.io/issues/2025-02-06-09-00-1738828/</guid><category>2025-02-06T09:30:00+01:00</category><description>&lt;ul>
&lt;li>
&lt;p>&lt;strong>Description:&lt;/strong> Software Upgrade&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>When:&lt;/strong> Expected Upgrade Start Date and Time: 06 February 2025 at 9:00am CET to 10:00am CET&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Business Impact:&lt;/strong> No impact on traffic side. Access to the management console may be disrupted. This duration defines the Maintenance Window only and does not reflect a downtime duration. Interruption on management console should not last longer than 10 minutes.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Expected action(s) from customer:&lt;/strong> None.&lt;/p>
&lt;/li>
&lt;/ul>
&lt;p>We will be performing maintenance on the &lt;a href="https://console.ubika.io/">https://console.ubika.io/&lt;/a> website on Thursday 06 February 2025.&lt;/p>
&lt;p>We apologize for the inconvenience caused by this upgrade. Please do not hesitate to contact our support team if you encounter any problems following this upgrade.&lt;/p></description></item><item><title>[Resolved] Logs dataloss</title><link>https://status.ubika.io/issues/2025-02-06-06-50-00/</link><pubDate>Thu, 06 Feb 2025 06:52:00 +0100</pubDate><guid>https://status.ubika.io/issues/2025-02-06-06-50-00/</guid><category>2025-02-06T07:30:00+01:00</category><description>&lt;p>A routine user operation stuck the logs server in a unreachable state.&lt;br>
Logs ingestion only recover after complete cluster reboot operation.&lt;/p>
&lt;p>We apologize for the inconvenience&lt;/p></description></item><item><title>[Resolved] SYN flood attack</title><link>https://status.ubika.io/issues/2025-02-04-10-25-1738661/</link><pubDate>Tue, 04 Feb 2025 07:15:00 +0100</pubDate><guid>https://status.ubika.io/issues/2025-02-04-10-25-1738661/</guid><category>2025-02-05T07:15:00+01:00</category><description>&lt;p>Dear Customer, Dear Partner,&lt;/p>
&lt;p>A SYN Flood attack is ongoing since Tuesday, 04/02/2025, starting at 7:15 AM CET, targeting the load balancer upstream of the Cloud Protector infrastructure.
The service remain operational with a minor impact on the platform&amp;rsquo;s response time. However, some probes turned red due to packet loss (without retries).&lt;/p>
&lt;p>The Ubika Support Team&lt;/p>
&lt;p>Update
&lt;span class="faded">(08:00 — Feb 5)&lt;/span>
&lt;/p>
&lt;p>The SYN Flood attack ended at 7:15 AM CET on Wednesday 05/02/2025.&lt;br>
Investigations to explain the packet loss are still ongoing with our hoster.&lt;/p></description></item><item><title>[Resolved] Maintenance on Jan 30</title><link>https://status.ubika.io/issues/2025-01-30/</link><pubDate>Thu, 30 Jan 2025 05:00:00 +0000</pubDate><guid>https://status.ubika.io/issues/2025-01-30/</guid><category>2025-01-30 06:40:00</category><description>&lt;ul>
&lt;li>
&lt;p>&lt;strong>Description:&lt;/strong> Software Upgrade&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>When:&lt;/strong> Expected Upgrade Start Date and Time: 30 January 2025 at 5:00am CET to 6:00am CET&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Business Impact:&lt;/strong> Risk of missing traffic logs during a 10 minutes time frame. Interruption on management console should not last longer than 10 minutes.
This duration defines the Maintenance Window only and does not reflect a downtime duration.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Expected action(s) from customer:&lt;/strong> None.&lt;/p>
&lt;/li>
&lt;/ul>
&lt;p>We will be performing maintenance on the &lt;a href="https://console.ubika.io/">https://console.ubika.io/&lt;/a> website on Thursday 30 January.&lt;/p>
&lt;p>We apologize for the inconvenience caused by this upgrade. Please do not hesitate to contact our support team if you encounter any problems following this upgrade.&lt;/p></description></item><item><title>[Resolved] Maintenance on Jan 14</title><link>https://status.ubika.io/issues/2025-01-13/</link><pubDate>Tue, 14 Jan 2025 08:30:00 +0000</pubDate><guid>https://status.ubika.io/issues/2025-01-13/</guid><category>2025-01-14 09:30:00</category><description>&lt;ul>
&lt;li>
&lt;p>&lt;strong>Description:&lt;/strong> Software Upgrade&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>When:&lt;/strong> Expected Upgrade Start Date and Time: 14 January 2025 at 8:30am CET to 9:00am CET&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Business Impact:&lt;/strong> No impact on traffic side. This duration defines the Maintenance Window only and does not reflect a downtime duration. Interruption on management console should not last longer than 10 minutes.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Expected action(s) from customer:&lt;/strong> None.&lt;/p>
&lt;/li>
&lt;/ul>
&lt;p>We will be performing maintenance on the &lt;a href="https://console.ubika.io/">https://console.ubika.io/&lt;/a> website on Tuesday 14 January.&lt;/p>
&lt;p>We apologize for the inconvenience caused by this upgrade. Please do not hesitate to contact our support team if you encounter any problems following this upgrade.&lt;/p></description></item><item><title>[Resolved] SYN flood attack</title><link>https://status.ubika.io/issues/2024-12-31-10-50-1309220/</link><pubDate>Tue, 31 Dec 2024 07:23:00 +0100</pubDate><guid>https://status.ubika.io/issues/2024-12-31-10-50-1309220/</guid><category>2025-01-01T07:20:00+01:00</category><description>&lt;p>Dear Customer, Dear Partner,&lt;/p>
&lt;p>A SYN Flood attack occurred on Tuesday, 12/31/2024, starting at 7:00 AM, targeting the load balancer upstream of the Cloud Protector infrastructure.
The service remained operational with a minor impact on the platform&amp;rsquo;s response time. However, some probes turned red due to packet loss (without retries).
The attack was moderate and ended on Wednesday, 01/01/2025, at 7:00 AM.&lt;/p>
&lt;p>The Ubika Support Team&lt;/p></description></item><item><title>[Resolved] Decommissioning of the CP NG intermediate platform (178.33.22.59) by the end of 2024.</title><link>https://status.ubika.io/issues/2024-12-16-00-00-00/</link><pubDate>Tue, 31 Dec 2024 00:00:00 +0000</pubDate><guid>https://status.ubika.io/issues/2024-12-16-00-00-00/</guid><category/><description>&lt;p>As announced several times since the beginning of September, the “intermediate” CP NG platform will be decommissioned by the end of 2024. Please read this carefully.&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Affected customers:&lt;/strong> You are using CP NG, and your asset is &lt;strong>NOT&lt;/strong> showing up as “DNS redirected” in the console. Your DNS is redirected to &lt;code>178.33.22.59&lt;/code>.&lt;/li>
&lt;li>&lt;strong>Action required:&lt;/strong> We kindly remind you that you need to &lt;strong>update your DNS records&lt;/strong>.&lt;/li>
&lt;/ul>
&lt;p>Information on how to proceed is available in the release note: &lt;a href="https://documentation.ubikasec.com/waap-saas-en/2024-version-history.html#infrastructure-resilience-upgrade">https://documentation.ubikasec.com/waap-saas-en/2024-version-history.html#infrastructure-resilience-upgrade&lt;/a>&lt;/p></description></item><item><title>[Resolved] Disruption Detected</title><link>https://status.ubika.io/issues/2024-12-10-04-41-1309220/</link><pubDate>Tue, 10 Dec 2024 04:41:13 +0100</pubDate><guid>https://status.ubika.io/issues/2024-12-10-04-41-1309220/</guid><category>2024-12-10T04:43:06.430543119+01:00</category><description>&lt;p>&lt;em>Investigating&lt;/em> - We are investigating a potential issue that might affect the uptime of one our of services. We are sorry for any inconvenience this may cause you. This incident post will be updated once we have more information.
This is an automatic post by a monitor bot.&lt;/p></description></item><item><title>[Resolved] Disruption Detected</title><link>https://status.ubika.io/issues/2024-11-29-18-06-1305710/</link><pubDate>Fri, 29 Nov 2024 18:06:43 +0100</pubDate><guid>https://status.ubika.io/issues/2024-11-29-18-06-1305710/</guid><category>2024-11-29T18:07:40.713132392+01:00</category><description>&lt;p>&lt;em>Investigating&lt;/em> - We are investigating a potential issue that might affect the uptime of one our of services. We are sorry for any inconvenience this may cause you. This incident post will be updated once we have more information.
This is an automatic post by a monitor bot.&lt;/p></description></item><item><title>[Resolved] Urgent logs database maintenance</title><link>https://status.ubika.io/issues/2024-11-14-00-00-00/</link><pubDate>Thu, 14 Nov 2024 08:00:00 +0100</pubDate><guid>https://status.ubika.io/issues/2024-11-14-00-00-00/</guid><category>2024-11-14T12:51:00+01:00</category><description>&lt;ul>
&lt;li>&lt;strong>Description:&lt;/strong> Database Upgrade&lt;/li>
&lt;li>&lt;strong>When:&lt;/strong> Expected Upgrade Start Date and Time: 14 November 2024 at 8:00am CET to 11:00am CET&lt;/li>
&lt;li>&lt;strong>Business Impact:&lt;/strong> This duration defines the Maintenance Window only. No traffic interruption planned.&lt;/li>
&lt;li>&lt;strong>Expected action(s) from customer:&lt;/strong> None.&lt;/li>
&lt;/ul>
&lt;p>A performance problem is currently affecting the business log ingestion platform (traffic logs and security events).&lt;/p>
&lt;p>Corrective maintenance is scheduled for November 14, 2024 from 8am CET.&lt;br>
&lt;strong>No interruption of the filtering service is expected.&lt;/strong>&lt;/p>
&lt;p>No loss of logs is expected during the update. However, the history of business logs will be unavailable for 1 to 3 hours, while they are reimported asynchronously.&lt;/p>
&lt;p>Database upgrade is finished. New logs are correctly processed. Reimport of history data will now be processed.
&lt;span class="faded">(08:33 — Nov 14)&lt;/span>
&lt;/p>
&lt;p>Import of history data is still ongoing without error.
&lt;span class="faded">(11:10 — Nov 14)&lt;/span>
&lt;/p>
&lt;p>Import is finished, all history data are available again.
&lt;span class="faded">(12:51 — Nov 14)&lt;/span>
&lt;/p></description></item><item><title>[Resolved] Disruption Detected</title><link>https://status.ubika.io/issues/2024-11-06-18-03-1294928/</link><pubDate>Wed, 06 Nov 2024 18:03:40 +0100</pubDate><guid>https://status.ubika.io/issues/2024-11-06-18-03-1294928/</guid><category>2024-11-06T18:47:39.372571989+01:00</category><description>&lt;p>&lt;em>Investigating&lt;/em> - We are investigating a potential issue that might affect the uptime of one our of services. We are sorry for any inconvenience this may cause you. This incident post will be updated once we have more information.
This is an automatic post by a monitor bot.&lt;/p></description></item><item><title>[Resolved] Disruption Detected</title><link>https://status.ubika.io/issues/2024-11-06-12-07-1294532/</link><pubDate>Wed, 06 Nov 2024 12:07:39 +0100</pubDate><guid>https://status.ubika.io/issues/2024-11-06-12-07-1294532/</guid><category>2024-11-06T12:21:38.964201757+01:00</category><description>&lt;p>&lt;em>Investigating&lt;/em> - We are investigating a potential issue that might affect the uptime of one our of services. We are sorry for any inconvenience this may cause you. This incident post will be updated once we have more information.
This is an automatic post by a monitor bot.&lt;/p></description></item><item><title>[Resolved] Configuration deployment issues</title><link>https://status.ubika.io/issues/2024-11-04-13-51-1293693/</link><pubDate>Mon, 04 Nov 2024 13:51:41 +0100</pubDate><guid>https://status.ubika.io/issues/2024-11-04-13-51-1293693/</guid><category>2024-11-05T09:40:00+01:00</category><description>&lt;p>The deprecated filtering service is facing deployment issues. New configuration deployments will not occur anymore, and traffic may face some instabilities until it’s fixed.
&lt;span class="faded">(13:51 — Nov 4)&lt;/span>
&lt;/p>
&lt;p>&lt;strong>If you urgently need to deploy configuration changes&lt;/strong>, please redirect your DNS to the new platform as advised in previous communications. Information on how to proceed is available in the release note: &lt;a href="https://documentation.ubikasec.com/waap-saas-en/2024-version-history.html#infrastructure-resilience-upgrade">https://documentation.ubikasec.com/waap-saas-en/2024-version-history.html#infrastructure-resilience-upgrade&lt;/a>&lt;br>
More generally, it’s strongly advised to switch to the new platform in any cases.&lt;/p>
&lt;p>Deployment issue is solved. An internal network was saturated, forbidding spawning of new containers.
&lt;span class="faded">(09:40 — Nov 5)&lt;/span>
&lt;/p></description></item><item><title>[Resolved] Let’s Encrypt certificate expiration starting on 4 November</title><link>https://status.ubika.io/issues/2024-11-04-00-00-00/</link><pubDate>Mon, 04 Nov 2024 00:00:00 +0000</pubDate><guid>https://status.ubika.io/issues/2024-11-04-00-00-00/</guid><category/><description>&lt;p>We kindly remind you that you need to &lt;strong>update your DNS records to get the resiliency benefits&lt;/strong> from the last releases. Follow the direction that were provided in the previous communications. You can also get those instructions in the release note. Especially &lt;strong>be careful about your firewall rules before updating your DNS records&lt;/strong>.&lt;/p>
&lt;p>Information on how to proceed is available in the release note: &lt;a href="https://documentation.ubikasec.com/waap-saas-en/2024-version-history.html#infrastructure-resilience-upgrade">https://documentation.ubikasec.com/waap-saas-en/2024-version-history.html#infrastructure-resilience-upgrade&lt;/a>&lt;/p>
&lt;p>&lt;strong>You are especially concerned if your protected assets are not anymore reported as “DNS redirected” in the console.&lt;/strong>&lt;/p></description></item><item><title>[Resolved] Disruption Detected</title><link>https://status.ubika.io/issues/2024-11-01-21-56-1292663/</link><pubDate>Fri, 01 Nov 2024 21:56:39 +0100</pubDate><guid>https://status.ubika.io/issues/2024-11-01-21-56-1292663/</guid><category>2024-11-01T22:06:37.043902923+01:00</category><description>&lt;p>&lt;em>Investigating&lt;/em> - We are investigating a potential issue that might affect the uptime of one our of services. We are sorry for any inconvenience this may cause you. This incident post will be updated once we have more information.
This is an automatic post by a monitor bot.&lt;/p></description></item><item><title>[Resolved] Disruption Detected</title><link>https://status.ubika.io/issues/2024-11-01-21-45-1292659/</link><pubDate>Fri, 01 Nov 2024 21:45:42 +0100</pubDate><guid>https://status.ubika.io/issues/2024-11-01-21-45-1292659/</guid><category>2024-11-01T21:52:40.113239306+01:00</category><description>&lt;p>&lt;em>Investigating&lt;/em> - We are investigating a potential issue that might affect the uptime of one our of services. We are sorry for any inconvenience this may cause you. This incident post will be updated once we have more information.
This is an automatic post by a monitor bot.&lt;/p></description></item><item><title>[Resolved] Logs service interruption</title><link>https://status.ubika.io/issues/2024-10-17-16-00/</link><pubDate>Thu, 17 Oct 2024 15:53:00 +0000</pubDate><guid>https://status.ubika.io/issues/2024-10-17-16-00/</guid><category>2024-10-17 16:12:00</category><description>&lt;ul>
&lt;li>
&lt;p>We detected that the log service is currently not accessible. Dashboards and logs view are currently not available. Filtering service is not affected.
&lt;span class="faded">(15:53 — Oct 17)&lt;/span>
&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Log service is up again.
&lt;span class="faded">(16:12 — Oct 17)&lt;/span>
&lt;/p>
&lt;/li>
&lt;/ul>
&lt;p>A wrong ansible log was run. Logs collection has been interrupted between 15:48 CEST and 16:12 CEST. Logs within this timeframe are lost.&lt;/p></description></item><item><title>[Resolved] Disruption Detected</title><link>https://status.ubika.io/issues/2024-08-25-15-10-1261282/</link><pubDate>Sun, 25 Aug 2024 15:10:40 +0200</pubDate><guid>https://status.ubika.io/issues/2024-08-25-15-10-1261282/</guid><category>2024-08-25T15:23:38.984596526+02:00</category><description>&lt;p>&lt;em>Investigating&lt;/em> - We are investigating a potential issue that might affect the uptime of one our of services. We are sorry for any inconvenience this may cause you. This incident post will be updated once we have more information.
This is an automatic post by a monitor bot.&lt;/p></description></item><item><title>[Resolved] Disruption Detected</title><link>https://status.ubika.io/issues/2024-08-24-23-51-1261079/</link><pubDate>Sat, 24 Aug 2024 23:51:42 +0200</pubDate><guid>https://status.ubika.io/issues/2024-08-24-23-51-1261079/</guid><category>2024-08-24T23:55:39.332771181+02:00</category><description>&lt;p>&lt;em>Investigating&lt;/em> - We are investigating a potential issue that might affect the uptime of one our of services. We are sorry for any inconvenience this may cause you. This incident post will be updated once we have more information.
This is an automatic post by a monitor bot.&lt;/p></description></item><item><title>[Resolved] Disruption Detected</title><link>https://status.ubika.io/issues/2024-08-23-22-07-1260762/</link><pubDate>Fri, 23 Aug 2024 22:07:41 +0200</pubDate><guid>https://status.ubika.io/issues/2024-08-23-22-07-1260762/</guid><category>2024-08-23T22:10:39.219006341+02:00</category><description>&lt;p>&lt;em>Investigating&lt;/em> - We are investigating a potential issue that might affect the uptime of one our of services. We are sorry for any inconvenience this may cause you. This incident post will be updated once we have more information.
This is an automatic post by a monitor bot.&lt;/p></description></item><item><title>[Resolved] Disruption Detected</title><link>https://status.ubika.io/issues/2024-02-07-23-02-433512/</link><pubDate>Wed, 07 Feb 2024 23:02:41 +0100</pubDate><guid>https://status.ubika.io/issues/2024-02-07-23-02-433512/</guid><category>2024-02-07T23:03:38.86753019+01:00</category><description>&lt;p>&lt;em>Investigating&lt;/em> - We are investigating a potential issue that might affect the uptime of one our of services. We are sorry for any inconvenience this may cause you. This incident post will be updated once we have more information.
This is an automatic post by a monitor bot.&lt;/p></description></item><item><title>[Resolved] Maintenance on Nov 2</title><link>https://status.ubika.io/issues/2023-11-02-05-30/</link><pubDate>Thu, 02 Nov 2023 05:30:00 +0000</pubDate><guid>https://status.ubika.io/issues/2023-11-02-05-30/</guid><category>2023-11-02 08:26:00</category><description>&lt;p>Maintenance is scheduled for Thursday, November 2 at 5:30 a.m., and should last no longer than 1 hour.&lt;br>
No interruptions are planned, maintenance should be totally transparent.&lt;/p>
&lt;p>Maintenance was longer than expected:&lt;/p>
&lt;ul>
&lt;li>Instead of transparently migrating the compute cluster, we had to reconstruct it asside. Without interruption on running trafic.&lt;/li>
&lt;li>Once this reconstruction was finished, traffic was switched to the new compute cluster. Thus, active connections have been interrupted by the load balancer reconfiguration. Thus some end-user clients may have seen disruption and reconnect around 08:20.&lt;/li>
&lt;/ul></description></item><item><title>[Resolved] Major outage on runtime</title><link>https://status.ubika.io/issues/2023-10-23-17-40/</link><pubDate>Mon, 23 Oct 2023 17:40:00 +0000</pubDate><guid>https://status.ubika.io/issues/2023-10-23-17-40/</guid><category>2023-10-23 18:09:00</category><description>&lt;ul>
&lt;li>
&lt;p>Our monitoring reports protected applications down and runtime cluster going red.
&lt;span class="faded">(17:40 — Oct 23)&lt;/span>
&lt;br>
Runtime cluster reports 100% CPU without significant network load. Investigation is going on.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Our private cloud reports runtime hosts as dead, runtime cluster is not reacheable any more
&lt;span class="faded">(18:05 — Oct 23)&lt;/span>
.&lt;br>
The team decides to hard-reboot the runtime cluster.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Cluster is back and running
&lt;span class="faded">(18:09 — Oct 23)&lt;/span>
.&lt;br>
Team will investigate during working hour on why the cluster went 100% CPU without traffic to avoid this situtaition again.&lt;/p>
&lt;/li>
&lt;/ul></description></item><item><title>[Resolved] Major outage on runtime</title><link>https://status.ubika.io/issues/20230927_maintenance_issue/</link><pubDate>Wed, 27 Sep 2023 09:50:00 +0000</pubDate><guid>https://status.ubika.io/issues/20230927_maintenance_issue/</guid><category>2023-09-27 11:08:00</category><description>&lt;ul>
&lt;li>
&lt;p>We’re rolling out new docker runners. As instances roll one by one, there should be no service interruption.
&lt;span class="faded">(08:00 — Sep 27)&lt;/span>
&lt;/p>
&lt;/li>
&lt;li>
&lt;p>First docker runner checked up and running.
&lt;span class="faded">(09:09 — Sep 27)&lt;/span>
&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Second runner up and running.
&lt;span class="faded">(09:49 — Sep 27)&lt;/span>
&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Shutting down third runner node. Containers does not balance to the second node as it should be.
&lt;span class="faded">(09:50 — Sep 27)&lt;/span>
&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Restarting third runner node. Runtime is back normal.
&lt;span class="faded">(09:55 — Sep 27)&lt;/span>
&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Rolling again second docker runner. Docker runner cluster state is now broken. As an emergency measure, starting to deploy the cluster from scratch to recover as fast as possible.
&lt;span class="faded">(10:15 — Sep 27)&lt;/span>
&lt;/p>
&lt;/li>
&lt;li>
&lt;p>Cluster is back up and running.
&lt;span class="faded">(11:08 — Sep 27)&lt;/span>
&lt;/p>
&lt;/li>
&lt;/ul>
&lt;p>Team is investigating on why the second node did not correctly join the cluster and why the cluster state broke at some point.&lt;/p></description></item></channel></rss>