Partial Outage across multiple systems

Major incident Web Application EU data center
2026-05-28 11:20 CEST · 1 hour, 5 minutes

Updates

Post-mortem
Incident Summary

On 28 May 2026, between 11:20 and 17:26 CEST, the Magicline platform experienced a partial outage affecting multiple services. The root cause was a software defect introduced during a scheduled release. The issue primarily affected Sales Tool pricing and preview functionality and led to degraded Web Client availability across the platform.

Impact on Customers

During the incident, customers may have experienced:

  • Slow or unresponsive behaviour in the Web Client
  • Intermittent errors when loading pages or submitting requests
  • Temporary unavailability of Sales Tool pricing and preview features
  • Temporary feature restrictions applied to protect overall platform stability

As part of our containment efforts, we temporarily disabled the affected Sales Tool features to protect overall platform stability. This meant some customers were unable to access pricing and preview functionality until full service was restored.

In addition to the prolonged degradation phase, customers experienced three distinct near-total outage windows:

11:33 – 11:44 CEST
13:56 – 13:58 CEST
15:15 – 15:22 CEST

Our Response

Our response followed the status page timeline:

11:20 CEST — Incident declared

Our monitoring systems detected elevated error rates and degraded performance across several services. The incident was declared and investigation began immediately.

11:39 CEST — Root cause identified

The team identified the underlying defect and began working on a fix.

11:46 CEST — Initial containment measures applied

To reduce the impact on the broader platform, the affected Sales Tool features were temporarily restricted.

11:56 CEST — First fix deployed

A fix was deployed and affected services began recovering.

12:16 CEST — Partial recovery confirmed

The majority of services had recovered, though some residual performance issues remained.

13:12 CEST — Secondary issue discovered

A related issue was identified that was contributing to continued Web Client degradation. The team applied additional containment measures.

13:27 CEST — Steady recovery under way

The Web Client showed consistent improvement. The team continued monitoring platform stability.

15:35 CEST — Brief recurrence of degraded performance

The Web Client experienced a temporary performance dip during the recovery process. The team responded with an additional fix.

Resolution

The incident was resolved through a staged approach:

Containment — Temporarily disabled affected features to halt triggering conditions and protect platform-wide availability.
Remediation — Corrected the defect and deployed a stabilised version of the software.
Verification — Conducted extended monitoring to confirm full recovery before closing the incident.

Lessons Learned

Testing coverage — Expand scenarios to cover the specific edge cases that triggered this defect, including load and feature-flag interactions before release.
Recovery procedures — Refine recovery playbooks to avoid steps that can briefly degrade performance during remediation; add hold-points and rollback criteria.
Customer communication — Provide clearer, earlier updates when features are temporarily restricted for containment, including expected impact and next update time.

June 1, 2026 · 18:16 CEST
Monitoring

The incident ‘Partial Outage across multiple systems’, which occurred between 2026-05-28 11:20 CEST and 2026-05-28 17:26 CEST, has been resolved.

Our engineering team will review the issue and implement additional measures to prevent similar incidents in the future.

If you continue to experience any problems, please open a ticket with our support team.

We apologize for any inconvenience caused.

May 28, 2026 · 17:26 CEST
Monitoring

The situation is resolved and the Web Client is fully operational again. We are still monitoring to ensure no further disruptions

May 28, 2026 · 16:17 CEST
Update

The team rolled out another fix and is evaluating the results.

May 28, 2026 · 15:54 CEST
Investigating

Web Client is experiencing degraded performance again. The team is investigating.

May 28, 2026 · 15:35 CEST
Monitoring

We are seeing a steady recovery of the main Magicline Webclient.

To ensure overall platform stability and reduce system stress, we have temporarily disabled certain features within the Salestool.

Specifically, this affects the summaries of picked offers during the sales process.
Our team is actively working on a fix to restore the Salestool to full functionality as quickly as possible. Thank you for your patience.

May 28, 2026 · 13:27 CEST
Update

The team applied another fix and is evaluating results.

May 28, 2026 · 13:17 CEST
Investigating

We discovered another related issue further degrading performance of the whole Web Client. The team is investigating.

May 28, 2026 · 13:12 CEST
Monitoring

The majority of our services have successfully recovered.
However, we are still experiencing a partial outage within the Salestool. This specifically impacts the preview of rate bundles during the initial selection and configuration stage of the sales process. Our team is actively working on a solution to resolve this remaining issue.

May 28, 2026 · 12:16 CEST
Monitoring

The team deployed a fix that resolved the underlying issues. Affected services are recovering and are available again.

May 28, 2026 · 11:56 CEST
Update

The team applied counter measures and is monitoring the situation.

May 28, 2026 · 11:46 CEST
Investigating

Team identified the root cause and is working on a fix.

May 28, 2026 · 11:39 CEST
Update

The team is still investigating

May 28, 2026 · 11:32 CEST
Investigating

We are currently experiencing performance degradation and intermittent availability across several systems. Some features may be slow or unavailable. Our team is investigating the root cause.

May 28, 2026 · 11:20 CEST

← Back