Post-Mortem: Customer Creation Issue on October 15, 2025

Incident Summary

On October 15, 2025, a planned security enhancement to our cloud storage system unintentionally caused a temporary disruption, preventing the creation of new customers for approximately 20 minutes. Our internal monitoring systems detected the issue immediately, and our engineering team resolved it quickly.

Outage Details

Wednesday, October 15, 2025: The issue began at approximately 17:04 CEST following a security update. Our team identified and fully resolved the problem by 17:24 CEST.

What Happened?

As part of our continuous effort to enhance the security and reliability of our platform, we rolled out an update to strengthen the security of our file storage system.

Almost immediately, our automated monitoring systems alerted our on-call engineers to an increase in errors related to the customer creation process. Our team investigated and quickly traced the errors back to the security update. To restore service as fast as possible, we immediately reverted the change, which resolved the issue.

Impact on Users

During this 20-minute window, any attempts to create a new customer in the system would have failed. This primarily affected workflows that involved sending automated emails with attachments upon a new customer’s creation. We sincerely apologize for any inconvenience this may have caused your staff and your new members.

Why Did It Happen?

The root cause of this incident was an incompatibility between our new, stricter security setting and an older component within our customer creation feature.

To improve security, we are moving to a system where all platform files are accessed indirectly through a secure gateway, similar to a Content Delivery Network (CDN). This ensures all traffic is properly monitored and secured. The security rule we enabled was designed to enforce this new, more secure method. However, the part of our software responsible for handling email attachments during customer creation was still using a previous, direct-access method. The new security rule blocked this older method, causing the process to fail.

Our Response and Next Steps

Our first priority was to restore service, which was accomplished by temporarily rolling back the security update.

To permanently fix the issue and move forward with the security enhancement, we have taken the following steps:

1. Permanent Fix: We are updating the affected software component to use the new, more secure method for handling files. This work is already underway.

2. Improved Procedures: We are enhancing our pre-deployment checklists to ensure all parts of our platform are fully compatible with system-wide security updates before they are rolled out.

What We Learned

This incident highlighted the effectiveness of our monitoring systems in detecting issues rapidly. At the same time, it underscored the need for more exhaustive testing to catch dependencies on older configurations before implementing platform-wide security enhancements. We are committed to learning from this event to make our platform more resilient.

We thank you for your understanding and continued partnership.

We have updated our security settings to ensure that customer creation processes run without errors.

We reverted the security measurement for the time being as they turned out to be too strict for a minor case.

All services are working as expected again.

Because of a security measurement setting create new customers was not possible for 20 Minutes. That included also online contracting.