Major Outage affecting all systems

Incident Summary

On 15 September 2026, between 19:09 and 22:05 CEST, Magicline and PerfectGym Next experienced a major outage lasting approximately three hours for customers across the EU. The root cause was a self-amplifying build-up of database row locks that started on one customer-record write path and, within seconds, spread across multiple customer databases at once. As requests stalled while holding these locks, the request-handling capacity of a central backend service (ml-internal-api) became saturated, so errors appeared across many unrelated parts of the applications rather than being confined to the workflow that first triggered the contention. Using Magicline and PerfectGym Next through the Webclient was the hardest hit: the Internal API that serves it was effectively unavailable from about 19:10 to 20:10, then reachable but slow — pages and actions took several times longer than normal — until about 20:45, and back to normal from about 20:50. Turnstile check-ins and new-member sign-ups via the Open API kept working throughout; only device-management actions initiated through the Webclient failed alongside the rest of the Webclient traffic. No software change was released that day — the condition arose purely from load and timing on an otherwise unchanged system.

Impact on Customers

During the incident, customers may have experienced:

• Slow or failed page loads and actions across many parts of the Magicline and PerfectGym Next web applications
• Errors on actions unrelated to the underlying trigger, such as logging in, checking in members, or opening everyday screens
• Failures spanning many customer databases at the same time, rather than a single isolated feature or a single customer
• Heaviest impact roughly 19:10–20:10; the applications were then reachable but slow to load for a further half hour or so (pages and actions took several times longer than normal), with customer-facing errors and response times back to baseline from about 20:50; the incident was held open through 22:05 for stabilization and monitoring

Our Response

Timeline of the response (CEST):

• 19:09 — Incident begins
  Background database contention crossed a tipping point on one customer-record path, and a self-amplifying lock build-up started and immediately began cascading across customer databases. (Public status pages record the incident start at 19:10.)
• 19:09–19:15 — Escalates
  The build-up spread across eight customer databases simultaneously. As stalled operations accumulated, the central service’s capacity to handle new requests became saturated, producing a broad wave of errors across many unrelated parts of the applications. At the edge, Webclient errors climbed past 40% within the first few minutes and peaked around 64% near 19:50.
• 19:15–19:42 — Response under way
  On-call was already engaged and investigating before the formal declaration. The intervening minutes went into getting online, forming a first picture of the scale, and establishing which systems were affected. Once it was clear this was a major, multi-system event, the incident was declared as quickly as possible – the later formal start does not mean the problem went unnoticed earlier.
• 19:42 — Acknowledged
  The situation was recognised as a complete service disruption and publicly acknowledged, and multiple engineers were engaged.
• 20:10 — Investigation ongoing
  Symptoms were visible across all affected databases, but the underlying cause had not yet been identified. Customer-facing errors were still high at this point (~30%) before dropping into single digits from about 20:15.
• 20:40 — Cause identified, first remediation
  The blocking database locks were identified and the blocking processes were terminated cluster by cluster, producing a first, partial recovery. Clearing individual database sessions proved insufficient on its own — the contention refilled within minutes — so the team shifted to cycling the affected service instances. By this point customer-facing errors had fallen to single digits, though the requests that did succeed were still noticeably slow — several times slower than normal — until performance normalised shortly afterwards.
• ~20:50 — Customer-facing recovery
  Error rates at the edge fell from ~4.5% at 20:45 to under 0.1% from 20:50, and response times returned to normal; customers could use the applications normally again. Database-side clean-up and formal confirmation followed as the pod rollout completed.
• ~21:05 — Locks released
  The remaining database locks were released as the affected service instances were replaced; with customer-facing errors already back to baseline, the response focused on clearing the last wedged sessions.
• 21:11 — Recovery confirmed
  Error rates and response times were confirmed at baseline; the response moved to monitoring.
• 21:35–22:05 — Recovery completes
  Recovery completed once the last remaining affected instances were replaced, followed by a final period of monitoring and verification.
• 22:06 — Resolved
  Error rates and response times returned to normal and the incident was closed. (Public status pages record the resolved window as 19:10–22:04.)

Resolution

The incident was resolved through a staged approach:

• Containment — Terminating individual blocking database sessions was attempted to relieve the contention, but was not sufficient on its own; the contention re-saturated within minutes.
• Remediation — Cycled (restarted) the affected service instances. This discarded the saturated request-handling capacity and the stuck open transactions held inside those instances, and was the reliable fix that resolved the incident.
• Verification — Confirmed error rates, response times, and backend capacity utilisation returned to baseline as the rollout completed.
• Monitoring — Once metrics had recovered, the system was kept under active observation for a final period to confirm stability before the incident was closed.

Lessons Learned

• We identified the root cause of the incident, in one endpoint triggering a row-lock convoy. We will fix this by short term.
• A localised problem was able to affect a broader set of customers than the ones directly involved, because some resources are shared across tenants. We are evaluating measures to contain this “blast radius” — so that future incidents stay more contained.
• n this incident the failure state was held inside the service instances, so remediating on the database side alone was not enough; recovery required cycling the instances. We are putting tooling in place to carry out and accelerate this kind of recovery across all affected systems, so we can react faster in comparable situations — this is already close to going live.
• Our monitoring alerted on the resulting errors only once failures were already widespread, rather than on the building lock contention itself — so customers began to notice before our alerting did. Earlier, convoy-aware alerting on database lock waits is being added so this class of build-up is caught sooner.

The incident ‘Major incident: Major Outage affecting all systems’, which occurred between 2026-09-15 19:10 CEST and 2026-09-15 22:04 CEST, has been resolved.

Our engineering team will review the issue and implement additional measures to prevent similar incidents in the future.

If you continue to experience any problems, please open a ticket with our support team.

We apologize for any inconvenience caused.

Our initiatives are showing clear progress—we are seeing significantly lower error rates and requests are successfully processing. We keep on monitoring.

We identified database/system locks across several clusters and have terminated the affected processes. We are already seeing initial performance recovery. We are systematically reviewing and clearing the remaining clusters one by one. Further updates to follow.

We are still investigating the issue with multiple engineers and try to find the root cause of the incident as fast as possible.

We are currently experiencing a complete service disruption affecting all systems. Access to applications and services is currently unavailable. Our team is working with highest priority to restore services.